Planning for a Usability Study | Cory Lebson | Skillshare
Play Speed
  • 0.5x
  • 1x (Normal)
  • 1.25x
  • 1.5x
  • 2x
13 Lessons (48m)
    • 1. Intro

    • 2. What is a usability study?

    • 3. The Big Picture

    • 4. Remote or In Person?

    • 5. Moderated or Unmoderated?

    • 6. Finding a Venue

    • 7. Drafting a Screener and Recruiting Participants

    • 8. Developing a Script

    • 9. Setting up equipment

    • 10. Note taking options

    • 11. Observation Room

    • 12. Analysis and Reporting

    • 13. Conclusion


About This Class

What kinds of things should you be thinking about as you plan for a usability study? 

In this course, user researchers Cory Lebson and Amanda Stockwell cover all the basics!

  • What should you be testing and when?
  • Should you do in-person testing or remote research?
  • Is moderated or unmoderated research best?
  • Where can you plan to test and what tools should you be using?
  • Who should your participants be and what tasks should you give them?
  • What other kinds of things should you consider?


1. Intro: Hi, everyone. Thanks so much for joining us today. I'm Amanda Supple, and I run a little strategy. A company called Stockwell Strategy. And I'm Cory Lipson. And I've been doing use a research for decades. He's old school today. We're gonna be talking about usability testing. So when we talk about usability testing, we're gonna talk about what Usability testing is how you could started. How do you find the right people and how you conduct the test, how you conduct the research and how you analyze that data once you get it and create actionable information that could help your team, Yeah, we'll give you an inside kind of back storm peak. Of all the details and logistics that go into creating the really successful usability test were both consultants. So we have a lot of experience, but you don't have to be. You might work internally already in a company you might have lots of experience with the vaccine might be brand new. This course is for anyone who's interested in running their invisibility test, and hopefully this course is for you. 2. What is a usability study?: you might be wondering what usability testing is before we really delve into what usability testing is. We should talk about what usability is. Well, there's an official i eso eso 92 41-11 which has usability, is the extent which a product could be used by specified users to achieve specify goals with three things. Effectiveness, efficiency and satisfaction in a specified context of use and usability. Testing is simply that. Can users use what they are expected to be able to use? And can they accomplish the goals that they're able to to accomplish in a in a efficient time and feeling good about what they did? Yes, you might have heard of lots of other sorts of UX research. I'm trying to figure who users are with the problem space aliens or what stuff you should be building. All of that is awesome, but it's not usability testing. So, really, what usability testing is is the stuff that helps us at the beginning, when we have maybe something very basic to show where something very formative it could be halfway along something that we've partially developed on are ready to to show to some users as is and even at the end are, or at the end, which can also be at the beginning of a new of a redesign on all those times we can figure out. Can users use what they're supposed to use and do the tasks that we want them to? Dio? Yep, So it's not marketing research. Usability testing isn't for assessing problem space. It's really specifically for assessing how well users were able to accomplish the goals that we have for them. So you have. Really? We should now think about now that we know what usability testing is, how we can plan for that usability study. Yeah, is it right for you? We'll find out. 3. The Big Picture: Okay, So usability test is pretty simple in concept, but there's actually a lot that goes into doing one successfully. There's a lot of stuff you need to figure out right at the front and then kind of figure out along the way. So one of the first things you need to figure out is Really What are you gonna be testing? Do you have a fully fleshed out product? Are you gonna be looking at competitors? You have a prototype, and if so, what fidelity is it? And who are you going to be talking to? What? What users are you trying to target? Are they already existing users? If only you have the potential users. Really? Those are the 1st 2 things you need to figure out. What are you testing? And who are you testing? Yeah, and then you got You gotta figure out. Well, are you gonna find users locally where you are gonna go to them? Are you gonna do something remotely? Where you where you're talking with them in real time. But they might be located around the country around the world? Yep. And to that. And you could do something called moderated testing. which is sort of traditional usability testing where someone like one of us is sitting and talking to a person. But you can also do unmotivated studies. Now there's lots of tools where you can set up tasks ahead of time and not even be present . Onda participants can answer a synchronously so we'll dig into the details of that stuff in the middle. But this is all stuff you need to figure out right away. And then pretty soon you're gonna have to figure out where you do testing, particularly if it's local or were somewhere that you go. So are you going to find a venue? Is there a place that you can go to do the testing? Are they gonna come to you wherever you might be? Uh, you've got you've got to know that. Yep. And if you're going to be doing things remotely or on moderated doing things like picking the tool on. So there's lots of sort of online tools that you can use, or even just the logistics of figuring out OK are using video conferencing tool. If you're not gonna be in the same physical location to someone, all those kind of logistics. You need to start getting sorted out. And then and then, of course, beyond the venue you need toe. Figure out who exactly you want, because when you do usability testing, you want either actual users. If if that's the kind of study you wanna dio or you, you might want representive users people who are like actual users but are naive to the product that you have. Yeah, and of course, it can be said that you should be building something that's easy enough for anyone to use, but especially depending on what your testing, they'll be specific contexts that some people might have other people's won't, and you'll really get your best feedback when you talk to either really or representative users. So you've got to figure out who they are and then also where to find them. Yeah. Then, as recruiting is progressing that's already in motion, you'll have time to develop a script, and descriptive will kind of go through what exactly you want to talk with him about what tasks you want them to dio exactly. Yeah, what makes for good task and a great question that doesnt lead participants and what makes for an upset. That usability question and then think through the technical logistics as well is pretty important. So we'll we'll delve into all of that, yeah. 4. Remote or In Person?: So one of the first things you really need to decide with usability testing is will that testing be remote or will be in person? So remote? Usability testing has some great advantages for why you don't have to go anywhere. There's no travel time. There's no travel costs. Simply from your desk. You can do the entire usability study. You could interact with people from all over. They could be local to you, but they could also be other places in the country. They could be other places in the world. It's all good granted. Give sometimes on issues, but but in general, it's really easy to conduct remote usability testing. Um, but the problem might be that your data is a little more shallow. Sometimes you can't rely on the participants having a webcam. You may not see their face, and sometimes their face provides a very rich data set up. Oh, it's wonderful or really get things pretty bad. So So you get you lose a little bit your data. You lose that that in person interaction that that synchronization that happens between a moderator and a participant, and it requires a level of technical sophistication for the participant. While you know, we say Okay, well, of course, when people that can use the web if we're testing something on the Web, we don't necessarily need ones that are super proficient. It can quickly start, Ah, hang out to go to meeting or Web axe or Or What have you so so on? All the more. So if you want them to share their mobile screen, that gets even more complicated. Eso. So being aware of that could be could be a caution against removed usability testing. Yeah, I just want you to think about every time you've tried to start an online meeting, even with the people within your own company, and spend 5 to 10 minutes trying to make sure that everybody can see and hear and all that stuff. So while there's uncertainly, some benefit, especially when you're trying to talk to you Ah, lot of people all over the world and it can be a bit easier to convince people to sign on board for a 30 minute phone conversation. There's also some great benefits of doing moderator for in person testing. One thing that I already mentioned is that as a moderator, you really get to build better report when you're in the same room with someone and you can read not just their face but their body language. If someone is frustrated or confused, you might be able to pick up on that better, Um, and so that's really great. It's great to be in person. However, there are some downsides to, and a huge part of that is travel time, both for you and for the participants. So if you're not talking to people who already live in the city that you live or whatever, either they need to travel or you need to travel, and you need to figure out the logistics of where you're going to host these sessions. So if you have a lab at your office, that's great. But you might need to find 1/3 party location like a A facility. There are plenty around were in a hotel right now because, uh, we'll talk about where that about that second, But you just you do have to kind of consider the presence and concept that, But as you might have guessed, we have a favorite. Yes, for sure, there's something I personally love in person usability testing like we're doing this week , and I just I really appreciate the opportunity whenever possible, to really meet with participants to see participants like we've been doing this morning. In fact, yep. And you cannot really kind of dig deeper. And besides, building reporter in the session, you can chow them beforehand about, you know, have 30 go in with the traffic like we both happened to be traveling right now. So I asked someone earlier for a recommendation for a place to get dinner leader on. It just kind of helps ease the tension a little bit. And you really get that, Richard. Deeper data and ability to read body language and face language. So and, of course, uh, went when we are doing mobile testing, it just life is just so much easier. Yeah. Yeah. And then whether it if something is not a computer, it's just so much because there is so much better. And I'm in it. The user research for the in person interaction, whether it's with the participants or Mike, both our cases for the clients, you know, it's just a lot of fun to do that. Yeah. All right. So you may not be able to choose all the time. You might have to go with remote sessions, but we recommend when you can to do it in person. But we'll give you some tips about how to find and make it the best either way. 5. Moderated or Unmoderated?: okay, So as we mentioned, we recommend moderating kind of being in person when you can. But there's an option also where you can do unmotivated usability testing, which means that you set the tasks beforehand that participants respond on their own time. There's lots of digital tools set up that you can do this on, and it can be really great if you need to get results on data really quickly. And especially if you're looking to get participants from all over the world because people competitive do it at their own time and they can answer at the same time and decent Eastern Tennessee. But, of course, you don't get the ability to dig in and ask them questions or follow up on things. You know, I find sometimes that clients will say, May, uh, we want you to do. I moderated testings that we get large numbers numbers, and there is something to be said that quantifiable data but quantifiable data is great but also could be shallow. You can quantify, but then, if you really want a dig deep, you wonder why What? What were they really thinking? And sometimes it's too late, even if you were to contact a few people later, they may not remember. In general we recommend moderating whenever possible. But if you need to do on moderated sessions, make sure that your extra careful and clear about how you write the tasks ensure that they're not leading. And when in doubt, do a run through with clients or colleagues to make sure that everything is clear. And you're kind of making sure that you are asking the questions in the right way. So you know what? Really? A moderator moderator is up to you. I personally has a researcher prefer that moderated testing and prefer whether that whether it's remote or in person, the real time interaction eyes still gonna be a Richard leaders that, even if it's not necessarily quite, is quantifiable. Yep. 6. Finding a Venue: as we're thinking about moderated usability, testing, riel time usability testing and you're thinking about in person that this moderation will be in person, you may say, Where should I be doing this? Testing. The answer is anywhere you could do, testing anywhere you could do testing in any in any state in the United States, you could do testing in any country. It's all good there somewhere for you and some in some to some degree, it's really gonna depend on your budget. How much money do you have for a space? Yes, but really, the most important thing is that you want to try to find a high concentration of of your users, because either you or they were both of you are gonna have to travel somewhere if you're going to be meeting in person. So hopefully you have a nice big budget on. And if that's the case, what we recommend is going to a research facility. So these facilities are already designed and set up to both hope you take care of some of the logistics around recruiting in the space they might already have, according set up, probably are easy to find or have good signings because they're used to having people come in all the time on bail. Be really attentive to your needs in terms of what the space looks like. There's often a room both for the participant and the moderator and also an observation room. And so serve that traditional glass you see in the focus group commercials. All that stuff is is already in a sort of research facility. You don't have to do it, and often they have extra perks like snacks and drinks. And can that stuff that makes over the comfortable bringing food, which which, while we could go get our own food, it makes clients and make stakeholders pretty happy. To feel like someone is is attentive to their deeds. And then, in some ways, it kind of created are around the research like, Oh, this research study is robust into the robust facility with that one way glass with With this with this great observation opportunity? Yep, and it feels more polishing. It gives a bit of power on to what you do. So if you have a big budget and you're able Teoh, we really recommend that a lot of times those facilities can also help with recruitment, which we'll talk about leader. But they can be really good for making sure that the logistics of the study goes mutely. Yeah, now, if you have this medium sized budget, maybe not quite as big, you may say, Well, where else could you go? Well, we are in a hotel right now can use the really stunning. You'll see our equipment set up a little bit later on, but for now we chose this facility. We got two rooms, one room for for us doing the doing, the moderation and interacting with participants. One room for the Observer's and it's perfectly nice. Now. We don't have one way glass anymore, but we virtual want my glass by streaming the sessions. And still it's up. It's a fairly robust, fairly in person experience. They're still setting up some some coffee and pastries daily for the participants for the stakeholders on, and you're a good portion of the way there. Now, of course, when we're we're here, not all hotels, not all all trying to facilities may have really solid Internet connections, and that's pretty important. If you're testing something on the Internet, you don't want usability study to fail because the hotels connection is bad. Even once in the hotel. Have the Internet connection go out on. That wasn't so good, but basically you just need a place again that's comfortable. Where participants feel good coming and going. Yeah, and to that No, we want to be clear. We didn't aren't in hotel rooms were in conference rooms within a hotel, so it can be slightly uncomfortable to kind of walk someone into over him with a bed. And, you know, so you kind of make sure that it's inappropriate place in some place that has have at least a desk or or a table set up so that it still feels professional and, you know, just kind of really your goal is to make the participant in the stakeholders feel comfortable. Actually, I did. That test wants where we had a Murphy bed. It was a distant did walk in a hotel room and one guy walked by and he said, I'm coming by a little early. I want to see if this was a jet on every where the bed was up. So he felt was legit. Came back later for his schedule okay? Yeah. Okay, well, so all that is good if you have sort of a medium sized budget. If you have less than a medium science budget, you still have some good options because both of those are pay for. So if you work internally for a company of a nice kind of a proposed are nice kind of parallel thing to the two rooms in a hotel are just two conference rooms or two offices within a company. If you're lucky and you have a lab, that's awesome. But you don't need to be that fancy. You could just have kind of two separate rooms within one place. Yeah, even a small office could be enough for just two people. Just just just the moderator and the participant. Yes, and there's also a couple of techniques that you can pull from. You might have heard of guerrilla recruiting. They're currently usability testing, and this is the classic kind of stakeout and intercept people. So whether that's at ah, mall or coffee shop, or I used to work for a big online retailer, so I would go to their retail stores on you kind of intercept people where they actually are There are some things to consider there. You need to make sure that you are at a place where the kind of people you want to target will actually be. So if you're looking to talk to you, say truck drivers, maybe go to a truck stop or a gas station. If you are looking to talk to you lawyers, you might have such good luck at a truck stop. So you need to think about that kind of stuff and you can't be choosy about who the people are, but it's a great way to get quick feedback and totally free space. Yeah, I remember once I was, and I was doing that kind of testing, but in a conference because it was association. So they gave the association Davis Little Vendor Booth and there were, like, high. We're not selling anything. You please participate, and we have some gift cards for you and then hit or miss. We've really got enough people in the end and also, you know, as as consultants, you may say, Well, we don't really have an office that we go to a really robust place. But clients have offices, and there's plenty of renting office places everywhere where you could just rent a facility for a short time in office for a short time, not hotel, where fairly cheaply you almost free. Yeah, you could get a place on. Actually, another good option that I've previously used is a public library that had some meeting rooms that you could rent out for a phrase. So although there's lots of options, no matter what it is, but basically what you need to look for is a place that is fairly convenient for both you and for especially your participants to get to you and on atmosphere that will make your participants and your stakeholders feel comfortable. 7. Drafting a Screener and Recruiting Participants: Okay, so one of the things that we mentioned earlier is that you really want to try your best to get actual or representative users to be participants. Amiri's ability tests so it is great to design something that is easy enough for everybody to use. But that shouldn't be everybody. Andi have to really think about the details of who these people are, what's important to them, how they're using what you're going to be testing, whether there truly the right participants. So, for instance, right now we're doing some usability testing on a really specific piece of suffer that if you are not in a certain profession, we'll make no sense to you, no matter how well it's designed. So it's important to really think through that stuff on. The answer for usability has really, never should be absolutely everybody and many times that's what I hear from client what we want. Everyone where for everyone, you know you're not for everyone. So basically, I guess How do you get these people the way that you should really kind of think through it is through a screener? What's a screener? A screener is a set of rules saying okay, This isn't this is represents criteria that for purchase mints we do. Why? And this represents criteria for magicians we don't want. If a participant has this, they're good. If it ever participant has this attribute, however, we don't want them. Maybe maybe they have a certain knowledge. Maybe it's part of their profession. Maybe it's some background that they have some experience, that they have some other software that they might have used. Now I often hear from clients things like, Well, it really matters. Teoh have gender criteria. It really matters to have raised yesterday. Maybe, but but doesn't really. Or are they saying we need to exactly replicate our demographics? Probably not. It's not about replicating your demographics. It's about finding users that can use the system in from different perspectives and different lenses and kind getting the universe of those users not having every demographic category. In fact, if you construct your screener in such a way that you've got a laundry list of demographics , you make it really hard to do the recruit and find the people that you need the people with the core education, the people with the core knowledge, the core background the poor experiences because that's what matters. That is your topic of the day. Yeah, you really want to pay more attention to behavior and experience and knowledge less than those kind of other things that you like. Where you from? How much money do you make some of that stuff? It really depends on the context of what you're doing. It might be important, but when in doubt aim for a shorter screener a shorter number of things that you wanna pay attention to and look for. Andi also always try to highlight the things that you don't want first, because by virtue of the fact that someone is participating in your screener, they want to be in your study, right? So try to screen people out early on rather than asking a whole litany of questions and then screening them out of the parent and sometimes in your screen. Or you want to construct questions that aren't just saying, Do you do this yes or no? You want to give them a list of five options and say, Which of these do you dio? And maybe only option three is the one you want. But that way you prevent people into saying I want participate. Sounds like so much fun. And then you get participants that aren't giving you the data that you want because they don't understand what you intended for them to understand. Yeah, you also have to be just a little bit aware of the professional usability tester or the serial participant eso. Both quarter and I have had experiences where people have showed up who we've seen before. I we're not quite sure that there really the right fit, but they've said yes, your kind of figured out the right way. So if you write the screen are really effectively, you can make it much harder for people to guess at what you're looking for so that you really, truly get the people who fit kind of criteria that you need. Yeah, for sure on. And you know, of course, that question becomes, how many participants do you really need on? You may hear anything like, 03 participants, five participants, a pretty 100 participants in our each. But you ultimately it's up to you. It's up to you to consider. You know, do you need kind of the big picture things do you need every granular bit of data? Are you trying to balance the The Big Dig ate the big piece of data with getting too much repetition. Now that you've recruited 20 people to come where where those 20 people buy bread participant 10 or even eight, you already know what the issues are. So think about that thinking about how many how many unique tasks you have, you have different user types actually might get different permutations on tasks where you have to work things just a little bit differently because maybe you want more participants than because he divide them up. Yeah, Another thing that consider is, if you're doing sort of a one shot his ability test or doing literate of testing. So if you are intending to do this kind of one set of testing and that may be fixed or change some things and then do another set of testing, you can get away with fewer participants each round because you know that you're going to be kind of validating some of your previous assumptions and continuing to learn along the way. So in general, we also we always recommend to be negative testing because that's just good, good smart thinking. But you can get away with fewer people in each individual. Discreet usability test is a bit cheaper, a little bit less resource is when you have fewer people. As a matter of fact, this week, what we're doing eyes 10 participants and we generally don't necessarily. We don't necessarily expect all tend to show up. In this case, we probably say, Well, at least eight will show up, and that will give us enough information for this generation. Now we're gonna wait two months and then schedule another 10 participants to come back a different set of participants and again, hopefully going to get eight. And then in that way we have our adoration. And we've got two months because that's what the development team feels like they need, which is totally fine. Different development teams work at different speeds that we different complexities of things to be fixed. And while we schedule 10 sessions, so can one hour slots with 15 minute intervals. You could also say when schedule floaters and the floater might be over over three sessions , they get paid a little bit extra thio Thio be over this recessions and probably bring a book or bring their laptop or or their their iPad or whatever and just do their own thing. They might not be used, however. There there should someone not show up, but I would say no. Personally, I prefer to schedule extra sessions and not deal with the complexity of floaters and having spanned over over three sessions. Yeah, as a general rule of thumb, take this of the big green assault is that for every sort of type of user group, per usability test came for about 5 to 6, at least kind of minimum sessions on. And I like to schedule one extra participant for about every five that I hope to include in my study, because even with the best recruiters, in the most reminders, stuff happens and people occasionally can't show up. Yeah, and actually to the best recruiters. They call people the day before they email them the day before and say You're coming right and they wait for a response to, and if they don't get that response, they'll fly, get and say, By the way, I wasn't able to get in contact with so and so just be cautious that you know, I'm a little concerned. Eso so good. Recruiters will kind of follow up multiple times, depending on when that initial scheduling takes place. Cory is talking about using 1/3 party recruiter. If you want some more tips for more detailed information around writing your own screener and doing the recruiting yourself actually have another skill share class. 8. Developing a Script: So when we think about usability, testing were really thinking about task based sessions, and those tasks have activities that users might do in real life. In fact, when they're recruited, they should be recruited to understand what those tasks really are and why you're asking them to dio now. Often you may give him fit data. You're not necessarily say well, this all personal, But at least you want to have them having experience that mirrors real life. And as you give them tasks, you're gonna say, OK, try this. This is what you're gonna dio. Here's our scenario. Go for it and you watch them go for it. And once they're done in one individual task or the whole the whole set of tasks, you can talk with them. You can You can debrief them on a per task basis, or you can debrief him at the end or both. But the critical thing you need to be aware of is that when you develop a script and you list all your task and maybe you have your probing questions, nothing you do early on is gonna buy us what happens later on. Yes, and you should make sure the tasks are related to goals that the team has. So you don't want to ask people to find something just for the sake of finding it. You want to make sure that you're really asking questions and developing tasks and sort of the questions around what your core goals are. So, for example, if you're working for on online retailer on their having trouble with people bowing out before they get Teoh, add something to the card. You want to create goals that are sort of centered around those things to make sure that there are problems with that. So try to think about the context and why you're asking those specific questions on and then you also want to make sure that you're asking them in a way that is clear and concise but not leading. So when in doubt, you wouldn't kind of take out any references to positive or negative emotions and really just ask kind of set the context for Imagine you're trying to do X. Show me how you would do it. Yeah, you definitely wanna want balance What you say. So what do you like or not like about this? What is good or not good about this. And you want a balance. You never you never buy us. I've been Oh, are are observed studies before where someone might say, Oh, what do you like about? That's a big no, no. Yeah, And then the participants. Oh, you want to know what I like? Okay, I like it all. It's wonderful. And but that in fact, you're you're pushing them only to the likes and not the dislikes or vice versa. Yeah, and it's an important thing to kind of know that people will automatically without meaning . Teoh tend to skew more positive when they're talking to you. So it's important to kind of write those questions non biased, but then to really observe what they're doing, right? So they might say that they love something after clicking around for 10 minutes and not being able to find it and getting really frustrated and then finally completing the task much leader. So right, the questions, in as neutral as away as possible, try to order them such that they don't bias the further questions and then also make sure that you set the context so that they're really understand kind of core of what you're trying to do without giving away any hints on. Then always went in doubt, say nothing and just watch. Yeah, I like frame conversation from the very beginning, saying, You know what? I'm not the developer. I didn't If you If you like something awesome, if you don't like something still awesome, we can keep learning. So So you know what? One other thing that I think about usability, testing and writing scripts is are the client teams where Where they come to May and they say, Okay, we're radio transcript. You know what? We have a script already used it for functional testing. And I say OK, so in your script, you basically said OK, the user will do this and then this. And then there's the next path Is this and this and this and that script is different. That's not were not telling the user what to dio. In fact, you kind of tell them the minimum necessary that gives them the real context of what they're doing. And then you set them off on their own. And then they say to you, What do I do next? And that happened today like three times like that. What do I do next? And you say, You know what? I imagine that I wasn't even here and do what you can. And if they get totally stopped, it was okay. We know they can't do that task. We might show them if it's not gonna get bias. There's future task, but but ultimately, we we really need to kind of get that authentic experience to whatever extent possible. 9. Setting up equipment: Okay, so here we are. We're very lucky today to be in a hotel. Any real set up for usability testing, in fact, me, and not cleaned anything. We have not changed a thing. This is exactly where we've been all day doing our usability stuff. So what we have? Well, first of all, we have to participate. Computer, There are studies today is going on a computer. It's a website. So the participant is using this computer. However, you know this this is a laptop. In fact, we didn't want the participants have a laptop experience if they didn't use laptops. So we've set up this setting set up right here for the participant there sitting on this side, they're using real monitor with a keyboard with mouse and really having a full desktop experience. Now, this computer right here This is where the moderators, the moderator I will be sitting here will be talking with the participants the participant has on the moderator role. Hand participant test cards. We don't always use cash cards with this study. It's just a little more complicated. Tests are a little deeper. And so for this case regarding zin task cards, that's especially useful when you have things like lots of different Loggins or passwords, which for this particular spending do have. But we always try to talk not really to kind of set up the context and then give them the test card as a piece of reference rather than just reading it directly. Right. And in fact, you here we have our script that what we're actually using today and this scrapped has has tasks that hasn't intro thanks to really warming warming the situation up. So besides that, we have the loggers laptop right here, the loggers laptop. Let's the longer record in real time sessions. Go on on, of course. You know we talk. We'll talk about logging shortly. But suffice to say I use what I have set up here is more a software where where you can love directly into the videos as they're being recorded very well. This year is a basic logic tech Web camp I like. I like it has to, uh, to my friends here that test good audio quality, get get both participants and the moderator and on then for mobile studies will be doing today. But I brought this anyway I've got my Peabo got document camera. It goes up on then capture device that because my brain just like that and in this way we actually have produced bring your own device without the complexity of saying, OK, let's install this. Let's install that. So yes, and what we're using more today It's not only told that you can use you could really pretty much used any kind of videoconferencing or video sharing suffer, especially if you use a Mac, which a lot of us do more isn't always the best choice, but it's a really nice one because you could take notes right in the video files as we go. But really, as long as you can see what the participant is seen as you're taking notes and as you're moderating while you're going on the other side of the people. But another thing to notice with you notice we're kind of off to the side were not directly in front of the participants on that, so that we can kind of see them with their body language and all clams up in Mallinson around. Just be heads down, taking notes. You want to be conscious of the actual physical space that you're set up so that the participant feels comfortable and I don't feel like they're being grilled or interviewed. You also know that we've got the two of us in here. But that's kind of Max out, right for sure. In fact, sometimes it makes more sense for the longer to be being an observation room or be with a client which, also, or stakeholders. We also have some some political values know what the conversation is going on in the background. So really all depends on what's gonna be best for any given situation. In terms of positioning. Of course, you know if it works better on the side or works better sitting next to the participant. Great. But one thing I like to think of it because I often have a chat going on on the moderator laptop. I wherever just been seeing that laptop, seeing the conversation and getting clues. Perhaps what went right, What went wrong? That's a good point. So you mentioned an observation room, and that's where we're headed next 10. Note taking options: All right, So now that you get everything all set up and you're running the study, one thing we haven't quite talked about yet is why there's two of us here running the usability testing because you only need one person to talk. However, it is super, super helpful to have two people so that one person could be focused on moderating and building that report and kind of falling up with the participant and the other person could be taking notes. It's really important. Yeah, really. One of two things is going. It is gonna happen. If it's only one of you one, you're not gonna get the level of notes. You get some dough. It's although you be collected clapping away, or you'll have to go back and re watch recordings and spend the same time anyway, that a second person would have would have taken except it would take longer because you delay doing the analysis until you get the notes. Yeah, so there are lots of different ways that you can know take. But one thing to kind of pay attention to is that you're not necessarily trying to transcribe every single thing that happened. You're trying to take note of the key things that are related to the tasks in the goals. So were they able to kind of do the tasks successfully? Did they get all the way through, or did they fail miserably? And even if they got all the way through, was there other feedback that they have? Did they have some different expectations? You're kind of trying to take notes of little tidbits of information that will help you lose your analyzed leader. And really the 22 problems I've seen with no taking where it just causes all sorts of trouble. As he tried headlines analyze the data. One is notes that don't actually capture what happened, Thea. Others notes that capture everything that happened, including every word that was said, every action that was taken. They say, Well, I wanna be thorough. The problem of things so thorough is, as you tryingto create a report for a client union just may be overwhelmed with data, so you have the level of depth that gets you the report that you need and that you could synthesize later on. Another thing that's important is especially if you're not in the same room. But even if you are in the same room, it's important to have ah, ongoing channel of communication between the person who's taking notes and the person who was moderating. So we have a Google chat set up when you know I'm no taking and he's moderating or vice versa. But if the person is not physically there on day could be watching separately. You can have either texture, whatever won't be too distracting. But a lot of times the person who is taking notes can observe things, or maybe make suggestions that you won't have the mental space to do or to notice. If you're kind of concentrating on moderating and I find as moderator, I know that I might know there's something that may or may not that may or may not be found you as you're watching it. So I'll sometimes say, Hey, you know what? I just noticed this. This one thing was, it was very minute. It was very quick and just having a little redundancy there. So even even if there is dedicated logging dedicated moderation, you still can't share back and forth in this matter. Fact. Conversely, when we're in the same room and you either way can work. Same room, different room. But but the longer could say, Hey, you know what? You missed this one. This one probe that would have been really good, not our script, But it would have a really good thing for my my logs and you as the loggers thinking about the analysis. So either way you can play. You can be flexible. You can see how what the interaction is like. But you just make sure you capture the right data in the way that will help you to do that to the final report. Do do the analysis to talk with stakeholders about what really happened. Yeah, it can be especially nice, especially when you are kind of wrapping up at the end of something for the person who was quietly taking if you're both in the same room tailpipe and ask some additional questions or if you're in the same room. Also, that note taker can kind of be the problem solver if something goes wrong if you have some sort of connection issues. So when in doubt, both for kind of note taking quality and just for kind of overall smoothness of these village has I always recommend having two people kind of on guard. Yeah, you know what? Sometimes I'll hear from from clients from stakeholders. Oh, the moderation. That's, ah, important thing that experiencing no taking. You know what? I've got an intern who will just take some notes. No, the truth is that they're both equally important, As a matter of fact, is nice to have people that air comfortable doing doing both and then you can switch. You can switch it up a little bit. Make sure that you're taking notes in the in the same format, of course, but basically you switch it up and go back and forth. But both are really important to have experience and to grow the experience and again towards good good reporting and good analysis. 11. Observation Room: Okay, so we saw where where we've been all day, doing the doing, the usability testing, meeting with participants and so on. Now here we are, in the back room where the where the stakeholders or the clients are sitting. And really, this is where they are. We haven't adjusted the thing, and this will really show you what's what's going on, what's happening behind the scenes. So here you'll see a great big monitor. This is where they're watching. Here's a laptop. It's running more A, which is what we told you we were using in the other room. And the the stakeholders are after getting toe watch picture in picture, they actually see a little picture of the participant. They see a big picture of the participants screen now. They don't get that one way glass, but that's okay. We have a pretty authentic, realistic experience with only maybe a three second black and you know what? They'll even so So they were watching here. They're kind of clustered around watching Big Minor. And do you think we actually travelled with the big monitor? Nope. We just said to them, Hey, Cuban, extra big modern lying around will be helpful and they said, Sure, no problem. They brought one over. Now you'll see to Not only do we have attack, we have food. Food off here. We got some cookies and brownies and all sorts of sugary stuff to keep the stakeholders happy and happy. Stakeholders make for a happy study. Well, especially for a long day. You know, it's tough to moderate and all day, but it's also tough to watch all day. And most of the time, our team members are not necessarily Teoh being in the weeds. Feasibility testing like we are so anything that you could do to make the experience comfortable for your team members. Whether that's clients who are paying you or your own team or just kind internal stakeholders, anything you can do to make them happy will encourage them. Toe observer. Participate, which is always good happened. Plentiful coffee. Very bored. Caffeine is good. Yes, again, Happy stakeholders. Happy study. Yes, One other thing to note that in this room we do have lines of communications still with whoever's in the room. If the person who's taking notes and logging is in that same room, that's great where they could be sitting with the rest of the team, which can be especially useful if you want to hear kind of their reactions and the discussion that goes along to see them react to the participants. Reactions can be super useful. It doesn't have to be in another room right next door. People could be anyway, anywhere. As long as you have that screen sharing technology that could be, you know, there, home. They they could be really wherever I personally have logged into a usability study from an airport before. I don't recommend that one necessarily. But it is especially alternatives to have a room, and I kind of Greek participants, especially if you get someone who comes a little. Really, you can do the initial set up stuff with them, like if you've got a consent form for them to sign, or if you've been sent him after his you can use. The stakeholder room is kind of a home base to make everybody feel comfortable, actually, right right here I can't see it so well is the signature form where they sign that they receive their consent and a gift card visa gift card for each of the participants to receive. And overall, I feel like this has been a very comfortable experience being at the hotel. Maybe not quite like that one way glass. Like that market research facility. That is full service. But you could make do that. You could make you pretty well in a place like this. Yeah. Couldn't get by with great visibility. Just pretty much anywhere. Yeah, for sure. 12. Analysis and Reporting: The final step of these ability is to figure out what to do without the data that you've collected and composited to some sort of digestible format for your clients in your stakeholders. So with the data analysis that's gonna look different depending on what your goals are, Some usability tests are really focused on how long it takes for people to do it. So you'll mike analyze things like time to completion or time on task. You might be looking at something as simple as did a pass or fail. Or do they get things right or not, that you also want to look for kind of other clues Kind of the snippets that we talked about for the note taking and look for trends so that you can not just report back what's wrong but also some recommendations about how you fix it or some trends of other things that people might want to look into our kind of anything that you can pull from that will be actionable for the teams that you're working with, and really, the report itself could take any kind of format. Eso, for example, you could have a set of slides, whether power point or in Google Slides? Yeah, you could say, OK, here's here. Some slides here. Some. Here's some images for Newsom Screenshots. Here's we'll call out Here's Ah, little Here's a little inserted video clip into into it where you can see what really happened across three participants who said, remarkably, the exact same thing. The other thing that could be really helpful is if you're recording the participants faces and you capture particularly strong emotion especially well, especially a strong negative emotion that could be really powerful. Teoh sort of convince and help your stakeholders and clients understand what to do and why it matters and how it's impactful. I have kind of a separate story from a little old job where we actually captured someone crying. We like a big fact here, screenshot on. We changed our whole plan for months based on that one image, so it can be that detailed, and it could be great to use emotion, but it can also be a something as simple as an email with simple appoints. I also especially like toe, have debrief meetings where I don't just send off a report or set off some kind of deliverable and call it a day. I like to talk through things. Talk about the experience, kind of say this is what we noticed and give some contacts in some of your feedback. You know, of course you should write, or whatever you put together should be clear enough that people could have it. But it's really more powerful if you could discuss for sure. And sometimes there is a discussion immediately, even before you analyze the data like what just popped out in your minds? Everyone on that Sometimes it's okay. Now that we've had time Thio, Thio go through to be careful. And sometimes I find clients ask for one when they mean the other they say, Oh, let's do a debrief right away and tell me the most sailing critical points that without spam participants synthesized data. No, wait waken talk, but we can talk. Just just give me a couple days toe to analyze, synthesize to really understand what we have on you. Conversely, you know, we talk about these really quick reports. I've also had head clients and and you know, sometimes those clients for official purposes, we want a word document. We want paragraphs well, a zey usability moderator. As a tester, it's a lot of work. It is more work. It takes more time. But sometimes that's what they want and being flexible and saying, Okay, you want you the client of the stakeholder? Want a detailed report? You want the quick PowerPoint with with annotated, annotated images and screenshots, and you want just the bullet points and you can have this bull points so quickly just understanding what your stakeholders want. What your clients want is so important. Yeah, the most important thing is to make sure that you capture the key takeaways and any actionable kind of takeaways or things that your group needs to work on. So it could be here. The things that went well here, the things that didn't go so well. Here's some things we need to test further on. Here are the things that definitely need to change, so, but really, what you want to make sure you do is whatever the goal of your team, your client, your internal team, whoever you're kind of working with, you need to make sure that you meet their goals just like the participants could meet. There's 13. Conclusion: thanks so much for joining us for this overview of usability testing. We hope you learned a lot. Yeah. It was great to be with you today and get to share something with you. Don't forget to fill out the activity sheet and upload it so that everyone can give feedback. Yeah, Feel free to find us on social media again on Twitter and the like on follow us. Good luck with urine tests. Okay. Thank you.