Transcripts
1. Introduction: Hey, I'm Eleanor Chapman, Director of Technical Content and Testing in Mail Chimp. So today, we are going to learn how to test like a user. Sometimes when you're a small business, you can't hire professional outside testers, but that shouldn't stop you from doing testing for yourself. We're going to talk about the different types of testing and demystify some of the jargon in the QA world, we're going to build a framework, and we're going to actually dig in and look through some sites for testing ideas. A lot of people think testing has to take a long time. But I think at the end of those process you'll learn that testing can be something that takes you five minutes or two hours, depending on what you have available, and what you're interested in finding. In the six and a half years I've been at Mail Chimp and in the five years as a tester, I've gotten to participate in multiple site redesigns, big feature launches, and help make sure that our new feature sets work for our users. When you are racing to the finish line and trying to build it quickly, it's easy to overlook small details that can get in the way of your business success. So this class is set up to empower you to get a little bit of time and make sure that you have the opportunity to check those details, and then take a step back and analyze your product to make sure that it's actually suiting your needs. Let's roll up our sleeves and start digging into testing.
2. What is QA Testing : So, let's get started with what QA is. QA stands for Quality Assurance which originated in manufacturing as a way to determine defects in the production cycle. When applied to software, it basically means that we're checking to make sure things work. QA matters because it's what allows you to sleep at night, you're checking to make sure things are working, and that your business is running smoothly. Testing gives you confidence in all the projects that you're undertaking. Testing also allows you to evaluate changes that you're making objectively over time, and it's an easy way to helping you implement your ideas. Getting started is really daunting, there's a lot of jargon, and there's a lot of process that can be instituted in QA. There's also a lot of really big businesses that have really effective QA strategies, that just don't apply for the little guy who's doing a lot of jobs over a very short amount of time. MailChimp is pretty similar to a startup. So, when I got started, we had to wing it and make it up as we went along, I did some digging, but it was hard to find a human in the academic world of some testing. So, we have created a framework that really relies on the mantra of test like a user, which is really easy to apply it to anything that you want to get started with. The two biggest challenges are time and knowing where to start. We can't add more hours in a day, but we can make it easier to use the spare 10-15 minutes or an hour that you have in between other tasks, by timeboxing to make testing an accessible activity. This may seem obvious, but if you're not sure where to start, start with what matters to you. It's easy to overlook the things that are the most critical to your business. If you're a e-commerce store, you should probably start with your checkout, getting the money is important to your success. If you're a website, you probably want to make sure that your homepages are loading or that users can navigate what you think your most important content is. If you're feeling overwhelmed at the thought of getting started with testing, I have three pieces of advice. Start small, focus on the big picture for you and your business, and asking questions is a totally valid form of testing. Start small, you don't have to find everything every time you sit down to your computer to task. When you start small it makes it a little bit easier to make incremental improvements over time, which will help your business be more successful because you aren't changing a ton of things after every testing session. When you're just getting started, it's really exciting and you want to dig into everything, but try and focus on your biggest goals as a business, that makes it much more manageable, it gives you an easy starting point, and you're actually making an impact on the things that matter most to you, for your company. Asking questions is a totally valid form of testing. Sometimes when you're not sure where to start, you can start with a question. What matters most? What do you care about? What is your gut telling you that you need to investigate? What do you think is successful about your website or store that you want to improve over time? So, now that you're familiar with some of the ideas behind testing and have a place to get started, we're going to unpack how do you actually use these tools in the real world.
3. Step Zero Exercise: Test Like a User: Rather than continuing in the lecture style on all the different kinds of testing, we're actually going to flip the script a little bit and do some testing of our own, right now. Go ahead and open up your website or a site that you visit frequently and we're going to start exploring some testing. I'm going to use Tattly which is a temporary tattoo design site. For kids, but really for adults. My daughter loves it, and that's a site that I visit regularly to shop for birthday gifts and just poke around to see what's newest in the design world. Don't overthink this. If you need to, go ahead, grab a drink, take a deep breath. We are going to get started by just using the website. At MailChimp, we have a core philosophy of test like a user not like a tester. Go in, look around, try and add some stuff to your cart. Since I'm using an e-commerce site, that makes sense for me. If you're navigating a website, start reading, click some links and maybe ask yourself a question and figure out if you can find the answer. When you're digging into a site, it's nice to ask yourself questions in your head or out loud as you choose while you're navigating around. Some questions you can ask yourself is, is this the content that I wanted to load on the website? Are there any broken images? Am I able to click a button and actually add an item to my cart? You're going to want to think about whether or not the website is working as you expect it. Maybe it's working as it was built but it doesn't make sense in the long run. Ideally, after you've been testing your site, you're going to want to make some changes, then that means you're going to need to communicate with a designer or a developer or just make a note for yourself later. A lot of big companies use ticketing systems like Trello or a Pivotal Tracker, and an enterprise area is Jira. I recommend Trello or Pivotal as a quick free option. Personally, I use Jing to take screenshots even when I'm not at work and record bugs for other companies, and I have also really enjoyed using Cloudly. On my desktop, I have a folder where I store screenshots which is how I usually like to take notes so I can remember the URL of the site I was on, and actually make a note of what I was thinking about when I film the test. So, I'm going to go ahead and get started with testing Tattly. I've already got the site loaded up and I'm really looking to see what Tina has put on the site lately. I notice at the top there is a carousel of images. So, right now I'm making sure that I can actually read every single image because they have text. I want to make sure that the carousel is running at a really awesome pace so that people can really drink in the designs and read the texts and enjoy all the thought that went into putting this carousel together. Every site has a bunch of links so, I always like to click the first one I hit and see where it takes me. Looks like if I click the Tattly logo, I go right back to the home page that I was on. So, I'm scrolling to the bottom now to check out where I can find all the tattoos. Seen the area where there's designs. I see things we love, and I'm not sure if that's the employee's recommendation or the most shopped tattoos. That might be worth noting or it might be something that we just decide we want to celebrate. Now, I'm looking at the left side navigation just to make sure it makes sense to me as somebody who actively purchases from Tattly frequently. I see a lot of design options which is really helpful. There's also sets. So, I'm going to click in there. I really like getting a bunch of things in one deal so that really appeals to me as a shopper. It looks like the pages taking a little while to load, that might be something that I would note for a developer later to make sure that the pages are loading quickly. There's a lot of images, Tattly has really design heavy because it needs to be. So, we want to make sure that page alone is something that we're thinking about when we're designing the site. So, on Tattly now, I've got a big image at the top that looks really good and then, I'm scrolling through and I see that I believe that these are different packs of Tattoos which look really awesome. If I hover over the image, it changes to show me all the designs in the back of the package. I'll go ahead and click this sold-out set and I'm going to see if I can actually buy it. Looks like Tattly site does a good job preventing me from adding a sold-out set to my cart which is a big win. That prevents users from getting confused and wondering why something that they thought they purchased maybe it hasn't been delivered if they didn't notice that the pack was taken out of their shopping cart at some point. So, I have a habit of scrolling to the bottom of the page because in my history, I kind of know that users have a tendency to scroll all around and it looks like there's not much in the footer for me to navigate if I'm shopping. So, that's good to know. So, there's more from this particular designer that I've been looking at. Now, I am clicking through the recommended similar designs and it seems like that page took another second to load. But once I'm on the page, I can see all the photography and that is loading very quickly. The carousel is really easy to use and it's really awesome that the photography reflects how people who are purchasing tattoos can lay out their designs on their body. I'm going to go ahead and add something to the cart to see how that whole process works because, at the end of this, Tattly is an e-commerce site. They're going to want visitors to actually convert and make a purchase. All right. So, I can see the item that I purchased. There's a good note that it's a set of two. I can see how much it costs. I can also see the quantity and I'm going to go ahead and add 10 and see what happens. It looks like when I typed 10 into the quantity, there's not a refresh button. So, it looks like I've only purchased one. Never mind, I found the update button. It just wasn't in the same slot where I expected it to be. So, when you're testing, you'll see things that ping your radar and maybe don't feel quite right. That's probably more UX decisions and those kinds of test can be subjective. You'll want to get with a designer or take a minute if you are your own designer. Think about it and then, address it at a later point in time when you've thought through what you really expected the experience to be, and maybe look at your analytics to see if what you thought is actually what you should do. If you are finding bugs that are error messages or things not loading, that's going to be a deeper technical issue that you're going to want to get with your developer or look through with your website provider to see how you can rectify the problem. The technical bugs probably deserve immediate attention because that's something that's getting in the way of your user. Right now, if I were to continue testing, I probably keep clicking around designs and adding things to my cart and then follow through on the checkout process. So, it has to be said that Tattly has a really great site which means that we can have some persnickety reports that is usually the sign of a really well-designed website. In our test today, I've found that pages loaded a little slowly when they were image heavy which is not unusual. Whether you know it or not, you have just successfully performed manual testing and our next lesson, we're going to dig a little deeper, talk about the different ways that you can approach tests. We'll cover manual testing and some deeper technical issues.
4. Types of Testing: So, in our last video, we tested like a user, rather than testing like a tester. But it would be unfair if I didn't teach you some of the terminology that's really common in the QA world. We're going to discuss a few different types of testing, and demystify some of the components that go into computer-driven tests, and expanding on a manual testing frameworks. In the QA community, you'll hear a lot about manual testing versus automated testing. Manual testing is what we did in our last video, where a person sits at their computer and starts investigating a website, a store, or a product for utility. It requires human analysis, a person actually interacting with the website. Automated testing focuses on what's under the hood to make sure that it works. It's all code driven, and mostly the domain of developers. Within the manual community, there's a few highlight reels that we can talk about, in terms of testing and mindsets. There's exploratory testing, smoke testing, and user acceptance testing. We're also going to talk about performance testing, and I want to make a note that that is a type of automated test, that checks to see whether or not your site can handle a certain amount of load, or the number of visitors that are coming to your website. The first type of testing we're going to talk about today, is exploratory testing. It's the Indiana Jones style of testing, loading up your page and digging in to see what you can find. Exploratory testing is the fancy terminology for what we did in the previous exercise. You probably have an idea of what you're most interested in testing, but, you don't necessarily have directions for how to get there. So, for example, I went to blabla.com which is an Atlanta-based children's toy manufacturer, and I decided to check out some new items that they've posted, specifically, their new very cute cactus pillow. I decided that I would like to add more than one, so I changed the quantity to four. Then I added it to my cart, using their Add to Cart button, I saw a really helpful load on the button that told me that, blabla kids is adding the item to my cart, and then they gave me another notification that said that the pillows were added. I noticed if I stayed on the page for another couple of seconds, that that button reloaded to say "Add to Cart" again. So, now I'm not really sure if the items were successfully added or not. This is a great example of exploratory testing, because, I went to this site to potentially buy pillows, and found a important bug that might be an area where users are getting confused. Certainly wasn't expected, and I certainly didn't plan it. Another common form of testing is referred to as smoke testing. Basically, it means where there's smoke, there's fire. It's all about finding areas in your app that are incredibly important and are no-fail zones. A really good example of this, is your login page, if you're an application, making sure that users can access their information and you use your tools. Another example might be when you're testing, if you're an e-commerce store, focus on your checkout process, versus looking for 404 pages. If users can't get through the checkout, they can't buy your goods, which is a big problem for business. Last, we have user acceptance testing, which is a common form of testing in many businesses. You get an actual user to accept your new build. And they take a look at what you haven't quite launched into production, and give you feedback as to whether or not it works. Outside perspective is useful, because, somebody who doesn't know what you're trying to accomplish is always going to see something different than what you think is important. Sometimes having a second set of eyes will show you that you've built a confusing workflow, and when you've been staring at something for five hours to try and build something new, you lose sight of what you've actually accomplished. So, grabbing a friend and having them do a gut check, is a really useful for just making sure that things are going smoothly. So, I mentioned that performance testing is an automated process. However, there are a few tools that are available in your browser that will help you approximate performance testing, or at least get an idea of how quickly your site loads, and how much volume it can handle. Every browser include something called Dev tools. I'm going to show you Chrome, because that's my preferred browser. So, in Chrome, I click the upper-most right hand three dot menu, go to More Tools, and then you'll see developer tools as the last option in the drop-down. Go ahead and click that, and you'll see a whole bunch of code pop up into the right hand side of your screen, or at the bottom. I personally prefer mine on the right, so that's why you see mine load it that way, but that's not how Chrome comes necessarily out of the box. But no matter where it loads, it's the same information. You can go to the Console tab, and start navigating the site that you're investigating, and you could see errors in the console, you can see if something isn't working when you start clicking buttons, you can also see what code is actually loading, which can be really useful if you're building your site yourself. Under Network, you can see how long each action of a page takes to load. If you've been actively marketing your site, it's probably good to do some performance testing just so you can make sure that your new visitors actually get to see the content that you've been working so hard on. Another quick way to do performance testing, is quite literally to press something on your page, like a link or an image 100 times, just to see what happens. So, I just caused an error on candyjar.com by clicking the scoop button as many times as I can think of, until I saw the red error load in my console. Now, I'll be honest, I'm not entirely sure what that error means, but I might want to reach out to the site developers just to see what they think. This is another good time to use your screenshotting tools, because it's very helpful in bug reporting. If candyjar.com received a lot of traffic, or they had an inexperienced web user, clicking the same button over and over, there might be some problems, because an error isn't loading onto the page, and the user won't necessarily know why there candy is not going in the jar. Sometimes it's hard to figure out which testing method will work best for you. Really, you have to take a look at the resources you have. If you are building your own site and using something like WordPress or Squarespace, you are probably going to want to focus on manual testing, because that's the domain that you have the most control over. If it's a custom site and you have an in-house developer depending on how small your businesses is, you can probably ask them to do some simple performance testing or automated tests to make sure that things are working just fine. I'm not going to lie, there's a lot of terminology that goes into testing, but, this should be enough for you to have a reasonable conversation with the tester, and maybe with your friends to show them some tracks.
5. Creating a Testing “Framework”: We talked a lot about the different kinds of testing, and what really makes sense for your business. So, in this video, we're going to talk a little bit about one specific tool that you can use no matter what type of testing that you choose or what kind of business you're working for. We're going to build a framework together. A framework is really a template that will help you record information and make it a little bit easier when you decide that you have an extra 10 minutes to experiment and test. Having a framework in your tool set is a very useful way to make testing accessible and easy whenever you're ready to do it. Don't be intimidated by the idea of building a framework. Really, a framework can be something as simple as a doc that consistently records information you want over time. I've called on our testing plans frameworks, because frameworks by nature are more expendable than templates. When you think of a framework, it tends to be something that you can pull apart and use just pieces of as you need. But, if you have a doc that's already outlining everything, you know the information that you want to record before you even sit down to your desk. So, this is what a framework looks like. It's pretty much a list of the things that you want to know at the end of testing. I started with a header, that tells me what I'm actually testing at this moment and time, and I'd probably copied this doc and name it after what my first header is, just so I remember what I've investigated. The next line in my framework is the date. When you're making a lot of changes to a website, sometime the date becomes really important. You might have found a bug two months ago that you're just now looking at, but it turns out it's irrelevant, because you've made five changes since then. So, the date captures the timestamp of when you found what you found. The next line of my framework is who tested. Now, I work on a team of testers. There's about eight of us. So, knowing who performed the test is really important. It can be helpful to know who did the testing, because you can make some assumptions about what matters to them as a user or as somebody that you've brought on to maybe do a little consulting work. After that, I really want to know what exactly I tested? So, I have a line in here that says, which feature I looked at? If you're doing exploratory testing, you could make a bulleted list that runs through every page you actually visited on this site. You can make a quick note, and sometimes I've even taken a screenshot with my phone of my personal notes, and then thrown them in a test plan just so I have it reported. So, the next line of my framework is actually listing the feature that I'm testing. You basically are telling yourself what you did, and how you did it. This applies to all types of testing. If you've been doing user acceptance testing, you can make a bulleted list of all the things that they looked at in their process. In some cases, when I've used handwritten notes for testing, I've actually taken my phone, snapped a picture, and loaded that image under this particular section. The next line is probably the most important. Your priorities, what actually matters in this series of testing? Try and remember that you should stick to what matters most to your business over hunting down nitty-gritty bugs all over your site. I've got a line in my framework called considerations or gotchas. Usually, we use this for things that are dependent on each other. I like to make a note of things like my store doesn't take Amex, so I shouldn't test with that. What I should check to make sure that the error message for Amex works. There's also sometimes known bugs that you're when you're testing a part of your app, you want to make sure you've captured in your test plan just so you don't have to think too hard about it later. The last line of my framework is called show-stoppers. You should use the showstopper line in your framework if you're testing things that aren't in production. How are you going to define whether or not something is ready to go to the public. The show-stoppers or things that you absolutely want nobody to ever see, and you should hold back launch until those are all cleared. In testing, you're probably going to find a lot of minor things that maybe are high priority. This show-stoppers part of the framework, helps you stay on track, and determine when things are ready to launch because you know they're good enough. The last section you'll see in our framework doc is a note section. It's basically where you brain dump everything you're thinking about while you're testing. It also helps me stay on track. If I know that I've prioritized my checkout process, but then I think of another tests, maybe around how things get in my cart. Then I can put something in that note section to remind me to go back to the actual process of putting items in my cart and stay focused on actually checking out. You should feel free to make this framework your own. You're going to learn a lot over time about what testing matters most to your business. So feel free to add and delete lines, or take notes as you see fit. There's no one right way to make a framework. Do what works for you. In the class resources, you'll see my framework that you should feel free to take and customize for your own business.
6. Conclusion: We've covered a lot in class today and I appreciate you attending. We built some frameworks, we did some testing and we investigated ways that you can apply QIT your role. I hope you found this useful and I hope that you go out, grab a friend, and get started testing today.
Alianor Chapman, Director of Technical Content & Testing, MailChimp
