The Complete Conversion Optimization course | Ruben De Boer | Skillshare

The Complete Conversion Optimization course

Ruben De Boer, Head of CRO, author, speaker, teacher

Play Speed
  • 0.5x
  • 1x (Normal)
  • 1.25x
  • 1.5x
  • 2x
56 Lessons (4h 22m)
    • 1. Course goals and benefits

    • 2. Why conversion optimization

    • 3. What is A/B testing

    • 4. Two tips to start your CRO success

    • 5. Course structure

    • 6. Know your company goals and KPIs

    • 7. Optimization research sources

    • 8. Expert review theory I

    • 9. Expert review theory II

    • 10. Expert review

    • 11. Document findings

    • 12. Web analytics analyses

    • 13. Web analytics - Google Analytics terms

    • 14. Web analytics - Make sure everything is measured correctly

    • 15. Web analytics - Dropoff points, bugs and errors

    • 16. Web analytics - Document findings

    • 17. Web analytics - Advanced segmentation

    • 18. Web analytics - Google Analytics events

    • 19. Web interaction analyses

    • 20. Customer feedback analyses

    • 21. Usability test

    • 22. Competitor analyses

    • 23. Science and psychology basics

    • 24. Science and psychology advanced

    • 25. Page speed and mobile friendliness

    • 26. Bringing it all together

    • 27. Setting up hypotheses

    • 28. Test plan documentation

    • 29. PIE, ICE and PXL

    • 30. A more complete framework

    • 31. Setup & AA test

    • 32. Binomial and non-binomial KPIs

    • 33. Can you test here

    • 34. CRO for low traffic websites

    • 35. Setting up the test

    • 36. Setting up the test advanced

    • 37. Quality assurance

    • 38. Test documentation

    • 39. Multiple tests at the same time

    • 40. Multiple tests at the same time advanced

    • 41. Setting the test live

    • 42. Keeping track of the test

    • 43. When to start analyzing

    • 44. Frequentist vs Bayesian statistic

    • 45. Analyzing test results

    • 46. Segmenting test results

    • 47. Reporting on test results

    • 48. Draw learnings from the test

    • 49. Example report

    • 50. Finalize

    • 51. The success formula

    • 52. Get people on board

    • 53. Overcoming the confirmation bias

    • 54. Build a knowledge database

    • 55. The ethical side of our work

    • 56. Thank you


About This Class

Do you want to increase your conversion rates? Get more A/B test winners and become a Conversion Rate Optimization expert? This brand new course is a complete blueprint for every step of the CRO process.

My name is Ruben, and I will be your instructor during this very complete course. My journey in Conversion Optimization started back in 2009 and today I am a Conversion Optimization manager and consultant. The methods in this course have helped me to realize over 50% of A/B test winners for several websites and it can do the same for you.

In this course, we'll be going through the CRO process together, step by step. We'll be doing optimization research, build your own test plan, setup and run A/B tests and much much more. The course is very practical and engaging and full of useful information and examples so you can be successful. It will be a lot of fun and hopefully very useful!

Key learnings include:

  • Conducting proper optimization research based on data, users and own expertise

  • Generating valuable insights and prioritize them to increase the percentage of a/b tests winners

  • Set up, track and analyze successful A/B tests (the right way)

  • Get more insights on your website visitors based on research and test learnings

  • Realize continuous optimization based on research and test learnings

The course is designed for Conversion specialists who want to excel in their line of work. It is also designed for marketers, growth hackers and designers who want to learn Conversion optimization. And for entrepreneurs who want to keep increasing the conversion rates of their website.

No prior knowledge is needed. You do need access to Google Analytics and preferably a testing tool like VWO, Optimizely or Google Optimize (any tool is fine).

Ready to increase your conversions rates? Then wait no longer! Let's get started straight away. :-)



1. Course goals and benefits: welcome to the complete conversion optimization courts. If you want to become an expert at conversion optimization, get more B test winners and keep increasing the conversion rates of your website. Then this will be the right course for you. My name is Ruben and I'll be leaving you food. Its course. My Career Inc Overstuffed Imitation started 10 years ago, and today I'm the head off conversion optimization and user experience team off one of the largest publishing companies in northern Europe. The methods I'll be teaching in this course has helped me to get over 50% of a beat death tweeners for several websites. The course will cover every step off the conversion optimization process. By the end of the course, you will be able to get a great insights based on your users, your data and your own expertise. He will also be able to set hypothesis and prioritize them for more success, and you will be able to set up a B test the right way and analyze the results from those tests to maximize your number of learning. The course will not go into food that statistics. It will also not cover every part of the human brain. Instead, Brookover. Everything. What you need to know in a very practical way. Together we will go through the process step by step, build your own testing then and setting you up for success, of course, is designed for converts, especially who want to excel at their line of work. It is also designed for market. There's designers and grow peckers who want to learn from first optimization, and it is designed for entrepreneurs who want to keep increasing to conversion rates on their websites. My goal with this course is to help as many people as possible in our great line of work. If you have any questions during this course or after that, please feel free to contact me. I'll be happy to help me out. 2. Why conversion optimization: what is conversion optimization and why is it important if you ask a lot of people, they'll tell you? Conversion optimization is a systematic approach to increase the number of website Fister's taking the desired action. So, for instance, if you have e commerce websites, so website setting items A of 100 Fister's on your website and one person ends up buying something, you want to increase that number to two people buying something or prefer, but even more. I think this is a great description off conversion optimization, but I believe that if you do the right way, it is so much more than just this. Through conversion rate optimization, you will increase the conversion rates off your website. But you also get a lot of knowledge on your website. Fister's behavior knowledge. They don't even know themselves, and with this knowledge, you can improve your websites. But you can also shared with the business and hope business could benefit from your knowledge. So, for me, conversion rate optimization, it's a systematic approach to give your website. Fister's the best possible experience on your website. While you learn from their behavior, you can share those learnings with the rest of the business, making it future proof well, increasing your conversion rates 3. What is A/B testing: a big and important part of conversion. Optimization is a B testing A B testing is splitting website Fister's over different variations off the same page and analyzing the impact on your KP ice. So, for instance, for the websites book a bus which is the website will use in this course, we could run a test with increasing the button size so 50% off the website. Fister's will still see the original website, but the other 50% will see an increased button. In your data, you can see which variation resulted in the highest number of conversions. Now a B testing is useful to analyze what work doing website. So to improve your website and to learn from your fister's behaviors and motivations. Now s you see in experiment not everything should be tested but you will cover this later on in the course 4. Two tips to start your CRO success: before we really dive into the course and all the theories and examples, I like to give you two tips to really kick start your success and conversion rate optimization and the first ISS get your research rights. Optimization research is perhaps the most important part in conversion rate optimization. There are many people in our line of work who, based on a B, testing ideas on their own expertise and on best breakfast this And as you will learn in this course, there's nothing wrong with that. Doing an analysis off a website based on best practices and your own expertise gives you great first impression off websites. But really, insights come from your users, your data and from science. And the best and most successful test ideas come from combining those insights. So when you get your research rights based on your users, your data and science, you will have a much higher chance to find your A B test winners. You will learn a lot more about your website Fister's behavior, and he will really start to increase the conversion rates off their websites. And it's exactly what will cover in this course. Now. The second tip is to get your data and statistics rights when you have your great test ideas based on your research, it would be ashamed if you get your data and statistics wrong. When you get your data and statistics rights, you would roll the right conclusions and learnings from your A B tests and she will keep building on your success. You will learn over this and much, much more during this course. 5. Course structure: the whole course will be structured around his model. We'll start off by covering all forms of research and gathering insights. Next, you'll create hypothesis and prioritized him after debt. You'll set up a B tests, and I like the results of their tests. Throw learnings from results and reports on your findings. We will end, of course, by seeing how you can become even more successful in our line of work. During the course, we will be looking at a real website. By the end of the course, you will have your own solid testing plan setting you up for success. Furthermore, this course will provide you with the lecture slides, screen recordings, useful free tools, Google analytics reports and in depth articles. In case you want to know more about a certain topic, the best way to follow this course is from beginning to end and put everything into practice straight away. Now again, if you have any questions, please don't have state to contact me 6. Know your company goals and KPIs: before you start with anything related to conversion optimization, it is very important to know the right goals for the company and for your website, and this might sound very logical and obvious, but it's very important to the line with the rest of the company. For instance, if you have e commerce websites and perhaps you have one, he might optimize for increasing number of transactions, so higher percentage website fishers will enter buying something. But if the rest of the company is optimizing for a higher average order value, you're optimizing for the wrong thing. Ah, higher number of transactions and the higher average order value could result in very different optimization practices. Besides knowing the main goes off your website, it's also very important to know the micro goals or micro conversions. A micro conversion can be someone signing up for your newsletter or someone darling in the white paper. These micro conversions could be a very important step towards the main conversion. Besides in commerce websites, he will also find a lot of leads generation websites, mainly in a business to business sector. Well, their main goal is to get high quality leads, but on some landing pages. It could be important to just get the address off potential prospects for affiliate websites. Their main goal is to have their Fisher's click through to other websites ankle for death. And then there are publishing companies are publishing websites, and these are a whole different story. They make their money through better fuse or fruit, having people read sponsored articles so their main goal could be to have their website fish there, read as many articles as possible or return as often as possible back to the website. Because if they returned more to the website, they'll read my articles, Seymour banners and Seymour sponsored articles. In this course, we will look at the Book of bus that you and book a bus. It's a company that helps you find the right coach or minivan for your group travel, and it's gonna be anything. This can be a company event, a school trip or Bester party with a big group of friends. The team of book a bus will help you find the right coach and save money with that, and their website go is quite obvious. They won their website fister to go fruit, a funnel and ask for a quote. So in summary, before you start with anything related to conversion rate optimization, always check their main goals and key metrics. 7. Optimization research sources: Now that you know the girls off your company, any website, it's time to start doing our research and to be successful in conversion. Optimization research is of great importance. Good and proper. Research and findings from the research results and valuable insights and customers needs motivations and behaviors. It would therefore result in better input for a testing plan, better hypothesis and much more successful A B tests. There are many, many forms off research conversion specialists can use, and basically you can defied them over this matrix. If positive, inquisitive research. And you have saying and doing India Section who cover all these forms of research and when you're done with your research and you got your important findings and insights, it's wise to make a nice report out of this and shared around with older stakeholders in the organisation interested in conversion optimization and with all those findings and insights, it's also very important to document them somewhere. You're covered this as well. Later on in X section 8. Expert review theory I: it's time to start our research our big converse optimization research and we start with the research. It's wise to start with an expert reef you in an expert review, you will look at your own website from your own expertise from your own experience as a conversion specialist. And this is not the best form off research simply because you will look at the website from your perspective, which could be completely different from the perspective off your real website fister but an expert review. Let's give you nice feeling for the websites. You get to learn the website. You might notice I'm low hanging fruits or some potential room for improvement on certain pages. Certain areas of the website and in our expert review will look at book of bus again and we will be using this matrix. A link to this matrix is available in the resource is off this lecture and with this matrix we will look at five important areas and we will start off with clarity. Clarity basically answers the question if the content and offer displayed on your website are as clear as possible to the website visitor, so this every page have a clear value proposition. Is there a clear official Herrick E on every page? Art officials supporting the content and supporting the value proposition like images and videos? And does every page have a clear next step? So first up is the value proposition, and I see a lot of websites doing this wrong or not showing a value proposition at O good and clear value proposition. Met Justin needs and motivations off your website. Fister ends different shapes. You from your competitors. A good vie proposition basically tells your website fister why they should do business with you. It tells him how you are going to improve their life, a clear and good fisher hierarchy minutes. It's easy for your website faced ER to see what belongs to what and what is important If you check out this light, it's very easy to see what belongs to what and what is important. And this might look quite easy and obvious because these are just four balls. But the content off your website should be exactly the same. It should be easy to see what belongs to what and what is important. Clear visuals on your website like videos and images shoot at to the content adds to the very proposition. It should strengthen the value proposition and it's very important to align those images and videos with your website fister. For instance, if you sell to teenagers, don't show images off adults clear. Officials on your website make the value proposition stronger, more clear and better for your website visitor, and the last one for clarity is clear. Next steps every page should have a clear next step for your website. Fister. If there's no clear next step, your visitors won't convert before we continue with the expert review. Let's quickly check out some websites with good, very propositions and let's start with Spotify on their home page spoke to five states Music for everyone. Millions of songs, no credit card needed get spot to fight free. Now this basically answers how Spotify is going to improve the life off the website Fister . It's also aligns with the needs, and it's probably unique compared to competitors as it states millions off songs notice. There's also a clear official hierarchy and the clear next step. This is apparently the most important parts off this front page in effect when I scroll down. There's only the food or there's nothing else. This is what you should look at. Now let's look at and bounce and banks states convert more elites and seals from your ex spent. Take your ad budget further and grow your business with inbounds landing pages, website pop ups and sticky bars. This is also a very clear value proposition for the target audience. Notice that this is very much tater to the target audience. When the mother reached this, she will probably not understand single words, but its target audience will find this appealing and straightaway know what unbalanced is all about. Now the next one is uber, and this one I like a lot. Uber states moved away. You want Dr when you want, find opportunities around you and check your phone, get where headed. There's lots of focus here as well. It's very clear that this is the most important part, and this is where it should look at notice. Also, the official supporting the value proposition, the blue sky and the hand out of the window gives you a very relaxed and certain feeling, which matches the value proposition off. Uber booking dot com starts with fine deals for any season, from cozy country homes to funky city apartments. I noticed at booking dozen states Look for rooms to stay. No, it says. Find deals. This is very appealing to the target audience and notice. There's also a clear next step here. And last but not least, my own website conversion idea stuff come website is still under construction, but the value proposition is already set up so everyone knows what conversion ideas is about and what it is it states become a conversion optimization expert master Every step off the serial process with practical, engaging and high quality courses. Learn online psychology with a complete online psychology book. This will be released later. Oh, for affordable business to consumer prices and a clear next step neuron More an official supporting the value proposition. Now remember that the value proposition should be reflected over your whole websites and business, including return policies, guarantees fast shipping your officials, your videos, your customer support. Your value proposition should be in the DNA of your organization and at every stage, every step and every page off your websites. I hope you find these examples useful. Now let's continue with the expert review matrix 9. Expert review theory II: the second thing we will be looking at using Expert review Matrix is relevancy. Relevancy answers the question If every page on website meets the visitors expectation, if every page, because assistance with the traffic source your Fisher came from. So, for instance, did a fish to come from Facebook through Google or didn't go directly to your website? And is the message on the landing page consistent with the message off the traffic source? And it's it's matched the intention or for website fister. So, for instance, if a user come from Facebook from ads displaying a free white paper to download on the landing page, you're lending Page should have this white paper, and it's free down those fairly clearly stated on top of the page, so it meets the fister's expectation. Third is friction. What is causing doubts? What is causing doubts and hesitations on your website for your website fishers to take the next desire action? What mentioned difficult for them but foreign fields could be too long and asked for unnecessary stuff. For instance, when Fisher wants to sign up for a newsletter and website also asked for a name and phone number. Don't ask it unless you really have to. It could also be hard to fill out the form. For instance, you have Teoh zip code in specific kind of way, and you get this error message all the time because there's a space orders, no space or there is no capital letters without berths. And ours are also a big form of frictional your websites that could be really big conversion killers. A typical buck could be a non existing page. Also, Colt a 404 page. Another example is the filters CDs on many websites that filters are not functioning properly. So, for instance, when I look for blue items, I get red items as well, or I don't get any items displayed. And, of course, a search for insanity. If the search from saying doesn't work, if it does not show the right results, it is clear buck and a big conversion kit on the last in the speeds. The website speed is very important these days. We are very impatience being humans are very impatient, so if your website is faster, it's much easier for the user to go for your funnel and eventually comforts before one will be looking at its destruction. Is there anything directing the visitor's attention away from the most desired action? Is there anything unnecessary? Distracting the visitor? Everything on your website page on every website page should be relevant. It should act in a motivation off your website. Fisher. If there's something irrelevant, like an image or content, simply remove it if it does not act to the motivation. If it does not strengthen the value proposition, remove it. It is only distracting the website visitor, and the last one will be looking at its motivation part of motivators or your websites persuasive at the right time and believable. We'll go into more depth on this in this research section, especially in the psychology lectures. But with motivators, you can think off social proof and payment, Aikens. But most importantly, do you have the right motivator at the right time? And it's unbelievable, with benefits, matching the needs and motivations off your customer. Now in summary, with this matrix and in our expert review, we will look at clarity, relevancy, friction, destruction and motivation. Now let's look at a book of best website and do it expert review for them using the Matrix 10. Expert review: In this new and updated video, we will conduct a quick expert review for book of us. If you want the full review, please check out the lecture resource is where you can find a. Pdf fell with the full analysis as well as a link to the expert re few matrix. Now let's have a look at the home page off book of us and find possible improvements. We can find the first improvement over here. The next Continue does not tell the visitor what the next step is. So this could be a Clare to issue. Possibly a better text for in this button could be this text over here. So get a free quote. Another Clare to issue could happen with the locations dropped down over here in the form, it seems like you can feel out any location. But in this drop down menu, we see major European cities. So how do these relate? Are these the only destinations the bus can go to? Are these the only locations you can use to book a bus service? Or are these just some examples? This could confuse the website fister, and therefore it could be declared t issue. A friction issue happens here in the foreign fields. When we click these arrows to drop down, fields does not open. The drop down field is displayed by clicking here. A motivation issue could arise as there is a lech off trust indicators When we scroll down , we do find a lot off good reviews over here. But book a bus could make them a lot more trustworthy by adding the surname and the picture off the client. Also overall review score, which is displayed at the bottom off the page. And it's actually very high for Book of Us Could be edits to the reviews over here. Now let's fill out the form. I'll go from my hometown of Amsterdam for a nice trip to Brussels on the eighth of June. I'll return on the same day. I will leave early in the morning and return after dinner and perhaps some drinks with 23 friends. This form seems to be working perfectly fine. Apart from these errors on this page, this button request quotes is by far the most important. But there seems to be quite a loved destruction. There's quite a lot of text above the falls on the bus and is dot over here have exactly the same color as this button, so this could potentially be a distraction issue. Furthermore, potential customers could miss a lot of information right here. In this case, the bus is like a product to the potential customer, so this website should contain product images and product details. Book a bus could at images off the bus both from the Insight and the outside, and they could give more details about the bus like that's the best. Contain a toilet. It's their music on board and does it contain free WiFi? So this is very likely a clarity issue for a lot of Web site visitors. A friction issue happens here in the foreign fields. The quotes off the buses can be sent to the email off the potential customers. Therefore, a potential customer might not want to leave their phone number on his website yet, So these are some nice insights for book a bus. Based on our expert review, I entered the findings and a few others here in expert review. Matrix and I created a report with all the findings we got for book of us now for your own websites, you want to check every page on every device and find possible issues. A link to the expert review matrix and this report is available in a lecture. Resource is 11. Document findings: we now have our first research insights. An expert re few matrix was a useful tool to help you with the expert reef you. But we will do a lot more research and get a lot more valuable insights and all these research findings and research insights you want to document them somewhere in one place. Now, Excel is a good place to start off with. If you're just starting with zero or only Ciro, specialist in the company, Excel is a good place to start. The same goes for most project management tools. They're good place to start. But if you grow, if you do converse in conversation for longer and your team grows, you want a proper to. To document your findings, see what's going on. Discover bottlenecks and see the progress off your team. You want to see if your run more tests if you have more winners. If you have more valuable insights, you want to document them in a proper tool. Now feed of you is a good tool for days by core. It's an A B testing tool, but in 2017 they rolled out there zero platform, which basically helps you document everything that's going on, but my favorite so ever is effective. Experiments feed LBO is nice, but effective experiments is more customizable to your needs. And in the beginning, this can be a little bit off a hassle, because you have to customize some templates to tweak it to your needs. But eventually you get used to it and a very useful tool for everything related to documentation from me. Effective experiments is a tool which I used in my work. My whole team is an effective experiments. Steak oversaw ineffective experiments and is the court and Arciero meetings ineffective experiments its east to see what is going on? Who does what the progress off the team are protesting more Are every more test winners? How many researchers did we do? And most importantly, you can build a big and very useful knowledge database now one of the clients in this course at a very clever idea to build a trailer board based on what he learned in this course and based on effective experiments. And it's a very good idea. This will definitely help you keep track of what is going on in the process off your team. Unfortunately, It is kind of hard to build a knowledge database with Freddo and with another's database. It's gonna be very useful later on. Wayne. Do more Ciro and want to be more successful. Now let's have a look. A documenting our valuable insights as mentioned. If you do not have any tools to document your findings and your progress, start off with Excel because you have to document your findings somewhere. So Excel is better than nothing at all. It's an excel Far could look something like this, and I included all the insights from our expert review already. So we have our research source, which is expert reef you. In this case, if the research findings we have to device, we did the expert reef. You just for desktop, of course, for your own website, you want to do it for mobile and maybe tablet as well. If the based group said the navigation, the front page and a quote page on better something should be tested or fixed trade away, we'll come to that later on this course and might also want to name people who are responsible and the status. So whenever you do a new research you can just adds to this file. So just include here, for instance, Google analytics and keep working from there. So excels Good place to start to document your findings, but it's mentioned my favorite to is effective experiments on here. We have expected experiments were in the book of Bus Project and straightaway. See your dashboard with well total ideas told experiments. Total success rate, which means a number off winning tests you've got and test spreads. Which means on how many patients website are you testing on your told experiments. Progress. Very useful to do list recent activities. This is a good first dashboards. The project stats here. It is very insightful to see your progress as a team you see for each month the experiment , the number of experiments you lunched. You see the number of experiments completed. The number off ideas created eso ideas come from research insights and a number of researchers created. Of course, those left experiment outcomes, which is how many how many tests results in the winner the page tested on and for parables test it not a fair, useful thing is experiment bottleneck. You can see at which stage off the testing things get stuck and you've got some other stuff which is useful to see. But what's very useful is this part over here, the workflow you basically see here, the whole see euro progress as the same as this course is built up. So we're for research from research and get your ideas and our processes. From there on you go to experiments or a B tests and your reports. So let's start with research. I already included all the expert re few observations. So your expert review 17 observations, observations and see them all here. These are all the observations we got from our expert review. Um besides the fact that you can document them easily here, you can also add tax, and it's very easy to find back research insights later on. So this is effective experiments. You'll be seeing more of it in the next videos 12. Web analytics analyses: next up in our big conversion optimization research its Web analytics and we will be using Google analytics for this. And who will cover three things. We'll start off like checking if everything is being measured correctly. Next, we'll discover books and errors using the data in Google Analytics and Third, we will have a look at drop off points. Drop off points are sexists or pages or your website where your website fish their leaves, the websites or doesn't continue in the funnel towards the conversion. 13. Web analytics - Google Analytics terms: before we dive into Google analytics, it's important you're familiar with some of the terms using that tool. Now, if you are already an experienced user in Google analytics, you can either skeptics lecture or set it to double speeds. But if you're somewhat new and this could be in useful starts, but you will see lots in Google analytics and hear a lot throughout, this course is a term users. A user is someone fisting your websites. You can be a user. I could be user. Everyone can be one user when they visit your website, but one user can visit your website multiple times, and these are called sessions. So you as one user, you can visit the website five times, meaning one user, resulting in five sessions. So free users could make up to 20 sessions, for instance, and their space shoes based fuse is basically a number of pages. Fish that on your website very easy. That bounce is when the user enters your website, there's nothing and leave again. So user comes to your websites, sees one page, doesn't click or in fact, anywhere you could Stroh, but he only sees debt page and then leaves again. So the bounds rate is the percentage off sessions which results in a bounce. The exit rates it's kind of similar to the bunch rates. Exit rate is the percentage of sessions that leave their website from a certain page. So within a session and user confess it's multiple pages before coming to the page where he or she leaves the website so it accelerates. Is the percentage of sessions leaving your website from a certain page said. This is on page level Goals and Google analytics are very important for conversion optimization. A goal in Google analytics is an event over user taking the desire action. So this could be anything like a purchase, a sign up for newsletter or darling that the white paper and a desire action you want to use to take can be go and Google Analytics, a funnel or a cell phone? Oh, and Google analytics is every step a user has to take in order to convert. So for any commerce website, a funnel might start with the category page. Next, the user has to go to the Progress page, the guard batch, the checkouts, the payment page and finally the usual convert. The whole process is called a funnel. Now, these are some of the terms you will see in Google Analytics. And here in the next lectures. If you have any questions on this, please feel free to contact me. 14. Web analytics - Make sure everything is measured correctly: before you start to do anything in Google analytics, it is very important to check if everything is being measured correctly, because if things are not measured correctly, your data is flawed and useless. I wouldn't go to the happen panel off your Google Analytics accounts. You might want to have clicked around all these things. All these links, all these settings. It will give you nice feeling of what's possible, and you might want to just some stuff which is applicable for your website on your business . But it's a conversion. Specialists are definitely some things you want to keep in mind that I want to check if it's set up correctly. If you go to the right property and the right few, which you can do here, start by checking out a few settings. What you always wants is to exclude all hits from known bus and spiders. We don't want to have any bots and spiders in our website data. These you're not humans. Thes will not convert, so it's just messing up your data. So exclude these, which also always want is to switch on the site. Search trekking now book a bus is very small websites and it does not have a site search functionality, so it makes sense. It's switched to up. But most websites do have a search functionality on their website and want to switch this on. You want to track that? This gives you a lot of valuable insights. Will see how many people use the site search, which search terms they're looking for. And if they will convert after using the search and definitely switched his on. Next thing you want is filters. I see that book. A bus does not have any filter set up right now, but you always want to exclude traffic from her own office and terrific from agencies who work on your website simply because these are not your potential customers. These are not your typical website visitors. These are your colleagues, and it's very unlikely chemical verts, and they don't behave like a typical fisher on your website, so you don't want these in your data. So always filter out your own office and that of agencies working on your websites. The next thing you want to check is if e commerce is set up and in this case, book a bus, it's unfortunately not set up, but you want to switches to on and set up e commerce with e commerce set up, you get more insights in your seals, your transactions, the order value. You get a lot more valuable insights for your analysis, so definitely switch this to one now. Mistake, which I often see on equal Merce websites, is that their payment service provider is listed as a traffic source. So this means in Google analytics, you can see where people came from before they lend it on your website so that they come through Google or Facebook or did a lens that wrecking your websites or through a link on another website. And you can get a lot of information on the behavior off these people from such traffic sources. So how many pages did they see how many how many ministers people stay on your website again from Google and of course, the conversion rates for traffic source. But what happens often on equal Merce websites is that the payment, so before the conversion, takes place in an external website. So the fish, they're based for a night in baseball product on external websites and then comes back to your thank you page, and it is when a conversion is being measured after payment service for fighter, it's not excluded as a traffic source. Your data will say that from the traffic sources, the payments served for fighter gets all the conversions on. That makes no sense because people come from Google. People come from Facebook people and directly. And yes, they go to the external website to pay. But you don't want to see it is in your data, obviously, so always exclude your payment service provider here. The last thing you want to check are the goals. Goals can help you get very good insights in whatever is happening on your website. What you want is to set a goal for every single step in your phone. This could help you discover Bucks and Evers. You can find drop off points and you can analyze a B test with this so SEC book. A bus has set up their goals correctly, but it's a bit of a different website. So let's assume we have for each commerce website I want to start a new goal and every step of funnel shoot vehicle. So let's start with the list page, for instance, or the category page every like to call it. Give the destination continue this nation equals Dude. This is however you set up your your your real structure on your website. So, for instance, could be gets agree a safe tickle after did the category, of course, after product based product page and will be equals two brought whatever depends on your websites and then we have the cards check out Step one checkups to check out, step to and the thank you page for all these steps. You want to make a goal and later on you will see why. In summary, you want to make sure that everything is being measured correctly, so you will obtain reliable and trustworthy data. It is therefore, very important to check if you have excluded all hits from known boats and spiders. Also, you want to enable your site search trekking. Next, you want to check your filters and excludes traffic from your own office and from the Office of Agencies working on your websites. He also wants if e commerce enabled and you want to exclude your payment service provider from the referral lists. And the last thing we discussed is that you have to set up a goal for every step in your funnel 15. Web analytics - Dropoff points, bugs and errors: if you have set up your goals correctly like I just showed you. You can build your own custom reports with your whole seals for no one there. And as you will see, this is very important for your Web analytics research for our Google Analytics research. I off course cannot show you real data off. Book of US data on Google Analytics is very sensitive data. And you don't want competitors to know your reps a data asshole. So therefore, I can show you this because of competitors. Get this data. Well, I'm sure the owners book a bus will not like that. So I'm using an old, profound old who analytics profile from their previous websites to show you the steps you have to take to do a proper Google Analytics research for your website. If you follow these steps along, I'm sure you have much more data in your Google analytics. And if I have to show your data now, I will select one day a long time ago just to show you how it looks and how it works. So the story off our Google Analytics research to find drop off points in the funnel and ducks and hours. We're gonna set up a custom reports, and as a conversion specialist, you will spend a lot of time in Google Analytics in the custom report section, which you can find here. So here we go there, ready to custom reports, but that's set up. Are we click nucleus reports. And and if you've never seen this before, it might look a bit difficult. But actually, it's quite easy. In this section, you can create your own Google Analytics reports using all the data older metrics older. The mention older mentions Google analytics has available, so let's see how this works. What he basically wants is to show our final step by step in disgusting report. So let's start by adding a metric, Let's start by getting users next. We want to see how many sessions those users have. Very good, for instance, select the bounce rates. There we go. Now we want to set up our funnel and want to start at the front page. How many users actually saw the front page and the first for him to get a quote. They want to see how many people made it to the quote page and at how many people ask for a quote, and this is something we've set up in our bulls in a previous lecture. So in this case, the Book of Bus, we have to look for show form. That's how cool. How did name goal number one and want to set go one conversion rate? There we go. The next step is the Quote beach with just being defined a step one. So the goal is named Step one and want the conversion rates off this and the final step is the quote request or in a success page. Boat requests. Let's see, go to conversion rates and if your life will also do the completions, so that's an actual number off quote request. So these are older metrics, but we need dimensions as well, and we want to set up a drill down off these dimensions. So here's how. Let's start with device device category so we can see a difference off these metrics between different devices that this desktop mobile and tablet now we can create a drill down because mobile might be interesting. Bud's. I also want to see the browser and next the browser first. This will come in handy when we look for mainly books and errors. So here's one, but you can have different tax too Disgusting report, which will do? Yeah. So this name listening this, uh, browser that's not attempt. So just like the same data, there we go. But this time you want to see, for instance, device category and then user type and user type is if the user new to the website arrested returning user that where you can see differences between those groups. So let's go this tap user type. All right, let's edit final reports well again. Include the same data, and this time you want to see again Device. I usually start with device because behavior on different devices is give me a very different amongst your users. So that's over the vice category and want to see source. Medium source. Medium is the traffic source. So where did people come from before they landed on your website that they come through Google that they come from a Facebook campaign? Or did they directly lands on your website is very useful to see where valuable traffic is coming from and difference in conversion rates amongst those so Let's call this the source Medium reports. All right, now we have our custom report with everything. We've been young, click safe. And now you see your custom report here just to show you what we did, What it just made. How works I selected one day a long time ago on the old based on old website of book a bus just to show you outwards. So we are in the browser except which would just make. And we see our device category. We see the user See the sessions, which everything we did. If you have a big e commerce website off course, you want this to be the funnel over e commerce website. So, for instance, category page, product page card page, checkouts, etcetera, etcetera. So you will see the results for desktop. So we see a very low bounce rate. Which goods indicates a buck in itself, but it skip it for now, almost 100% seized 24 Well, this logical because it's on the front base, 30% makes the first step and 15% finally converts. And you could see the same for mobile and templates. Now, if you click on desktop we get the drill down, so we should get older browsers. So here we go. You can see the difference was between the browser and what we, for instance, see. And of course, data is not a lot. We do see that Firefox has a very low conversion rate compared to the other browsers, so they could mean there is a buck on the desktop. On Firefox, we click on fire folks, you get a different browser versions. And, of course, data is not a lot. But we see that 53 and everything else, basically, as zero conversions this because there's a bounce rate of 100%. So maybe the website is not displaying well on Firefox number 50 version number 50 free. So that could clearly indicate the buck. And this is how you see books and error. So let's I just had back. So besides a possible buck on the desktop and Firefox and, of course, your jacket for, oh, devices and old browsers, you can also see the drop off points, for instance, Here you can see that mobile. A lot of persons drop off from the front page. Quite loved. See the front got a lot of fish or see the front page of a book of us, but only 1/5 continues to the quote page. So if you start testing on mobile, your focus should be on the front base, because this is a big drop off. Of course, if you have your whole seals final set up, you can see exactly where the biggest drop of point is and what should be. The focus off your testing an extra browser. Weaken OSI to use their type. So for desktop quickly contest up and we get new and returning visitors. You can see what happens here and can also see this for source and medium so you can see what it's for but traffic and what it's not in this case, being effort. This is advertising through being gets a lot lower conversion rates than advertising fruit . Google beside discussion report and has mentioned you will spend most of time in Custom Reports section. You can also check out pages with high exits. You can check for bounces and can check for search, so let's do that quickly. If you go to behavior site content on landing pages, you can see all the pages. People land on you see a number of sessions and you see the bounce rates. There are two things to note. If a bounce rate is too low, this could indicate a book. Something is telling Google analytics that no one bounces on your website, and that's very unrealistic. So if you have a high traffic page that set it up using the filters include sessions. Let's discuss it created an 40 ends well at bounce rates left in 2% it applied. And if you already so they're a lot off pages with Sarah percent pounds on this on this day on this old book of bus website. So something on the Web site is dead and Google analytics that every user is interacting with the page every user is not is doing something on the page or checking more other pages on the website and as of course, unrealistic because a lot of Fister's will come to your website. I think now this is not for me. I leave the website straight away. It's impossible that no visitor bounces at all, so there's definitely a buck going on here, of course, besides, too low bonds, which is an error. We also want to high pounds. Um, most likely will not anything here. And of course, these metrics depends on your website and your data, but baited with high bounce rates could be a focus for your A B testing and for your conversion optimization practices. Exactly the same goes for exit pages. If a fish has a lot of page views, a little exits, you know something is wrong there and he shoots. Focus your testing on that. You should focus on improving that particular page, the last thing I want to show you. And, of course, there's a lot to show in Google analytics. Um, it's the uh site search section and Google Book. A bus does not have a site search because it's a fair, small website but hasn't mentioned you can find a lot of valuable information here. For instance, people. A lot of people search for a certain item on your websites. You might want to display it more prominently for interest, put it on the home page, or, if people look for search for a particular item, my website and don't have it, you could think about adding it to your website because apparently people expect it on your website and want to buy that or download it whatever they're looking for, so most of focus will be on the custom report. But also don't forget to check out the exit rates, the bounce rates and the site search. So these are just some of the areas in Google Analytics, which you can check out to discover books and hours, drop off points and to get test ideas for your A B testing. But of course, that's a lot more in Google analytics. I invited to click around and discover all the basic functionalities if you haven't done so now. For the sake of this research, and for the sake of this course, let's assume that book a bus has no conversions on mobile browsers. Safari first in 10.0. And if we see a high drop off from the front page, meaning a lot of users leave the website from the front page without asking for a quote, 16. Web analytics - Document findings: Here's the Excel spreadsheet again. Um so research sources Google, Google and Olympics research Finding Mobile Safari at 10.0 and has a low conversion rates error could be in book. Good. Be book. You want to discuss this with your development team? It was on mobile, and the page is uncertain, so this is X elegant. I also have effective experiments open in a research step. We add a new research, and we named Google Analytics Research have to give to summary and G. A gun is ducks errors and drop offs. Oh, the drug offs type is lunatics to date 25th of August and also to completed today. Otherwise, discourse video will be very long and connect some tax if you like. So let's add a new research and let's at the observation and conversion rate very low on Mobile safari 10.0 and good beat. But that's a text. So it's a mobile and it's about and shift observation. Now we have one observation in the Google Analytics research. Let's just assume for the sake of our research that book a bus, um, we see that on the front page. Not a lot off people on fill out the first form. They don't feel out there. They're they don't ask for a quote. So I dropped off on the front page, began its for a bit safer. Old devices and its front beige but prefer baby do this in effective experiments and observation. So again, high drop off on front page at X on this, both mobile Best up and tablets on the front page, of course, from page there we go. Safe observation. So these are just a few examples, Um, Google and let it is much bigger. And there's so much information in this, but you'll need a whole new course to go through Google Analytics. But I hope you understand that using the goals and portraying your funnel into custom report gives you valuable insights on where but possible bugs and errors are and where the big drop off points are for your website. Of course, when you do this research this Google Analytics research for your website, you'll probably get much, much more valuable insight 17. Web analytics - Advanced segmentation: in order to get more insights from the data and learn about your website Fister's. You can use segmentation with a segmentation. You create different groups of website Fister's and you could end a life their behavior on your websites to create a segment you go to Google Analytics in. Click on here in this area and click new segment on order. Set up a new segment. You could go to conditions and in this case, let's see the differences between new visitors and returning visitors. So, in this case, will go to user type. Contains new visitor and let's give the name new visitors. It's safe now. You can go can compare new visitors to returning visitors again. Great new seconds. Let's call this returning visitors traded condition, user type, some things, returning visitors returning visitor and we had safe. Now, with these two segments selected, we can go through Google Analytics and see the differences. For instance, in pages these users have been visiting now, other useful segments are converters versus enough confers, so people who converted already set it up. Here is transactions. It's larger than zero, and people did not convert. His transactions is equal to zero. Now with these two groups, you can see what people did. Who converted first is what people did. Only website who did not comfort. Another interesting one is people who spend a lot on your website and people who spent a little. For instance, if the average order value on your website is 100 euro, I would very much like to see what people do on the website if they spent three times as much. So to revenue per session is larger than 300 euros. And of course, I would like to see what people do if they spend a lot less. For instance, people that spend less than 30 years. What is the difference between those groups? What different pages do they visit? What areas of the website do they see? How long did he stay on the website? What else did they do? And the same goes. For instance, for sites, surges use whose search on the Web site are looking for something specific. It's very interesting to see how they behave and where they used to search, for instance. So in this case, I added a segment site search and sessions with search is larger than zero. And of course, if you run a big company, a multinational company, you can check out regions that comfort very well in this case. For instance, Germany vs regions who don't go for it as much. When you have your segments set up, you can combine these with the custom reports just made as displayed here on the slights. By doing so, you will get her extra valuable insights. So, in summary, you segmentation to find new and interesting insights. 25 we just discussed. Our new first is returning fist. There's uses who convert versus users who don't convert users who spent a lot first users who spent a little on your websites. Users who used to site search and use you don't and high comforting regions versus low converting regions. But I'm sure you can come up with many more segments relevance for your business. So do check these outs because with these segments you get a lot of valuable extra insights 18. Web analytics - Google Analytics events: events are user interactions with your websites, which you can set up and measure yourself so by the fools. Google Analytics gets a lot of information in, for instance, the device or users on the location of the user. The pages Someone visited the time on site Simon page exit rates and much, much more. But still, you might want to get even more and extra information, and that's why defense come in handy now. Defense could be set up through a line of code on your website, and probably a developer can help you with that. Or you can set up through a tech manager like Google Tech manager. Perhaps you kanoute yourself or your weapon honest, and you can name the events as you like, but I always three things to name. No, they are optional if events category, event action and events label. So, for instance, he wants to have information. Google analytics. If a user downloads documents on your website, for instance, PdF or the White Paper pdf so you could set it up with events category being conversions, the event action being downloads and even label being the name off the document. Another example could be a newsletter. Sign up. This comes in handy. Wenders. No confirmation or thank you. Page that case. Google Analytics won't measure the conversion, so you can do it through an event in this case event category could be Converse, since again, event action Sign Up and the defense label, newsletter and way of affiliate website. You want to measure external links so you could say defense. Get very click out event action the name of the button, for instance. But an overview page or but in product page and the label is the euro off the website, external websites, other useful events you can set up our, for instance, a click and a navigation when someone plays video or error messages in forms which are being displayed to users. Because if you get the information on the error messages informs, you know where your users get stuck and get improved. It and these are just a few examples. But you can measure everything with events, so events, if you're white, range of possibilities and a lot of useful insights in Google analytics, And when you combine these defense with segmentation, you can see the differences between user groups and this will again needs a lot of valuable insights, a lot of learnings about your website visitors and a lot of opportunities to improve your website. 19. Web interaction analyses: when you are doing your Web interaction analysis, he will be looking at click maps, scroll maps and recordings with Glick maps. Officially see where your Web site fish for clicks on a certain page. Now with scroll maps, you see how far your website Fisher scrolls down a certain base in the form of a heat map and with recordings, you actually see their website fish there, navigate over your website. You basically have a screen recording off your website visitor, and it's gonna be a lot of fun and you'll get lots of insights. It's fun to see your website fister navigate over websites. You really see them interacting. You see them clicking, typing, scrolling. You just don't see their weapon. But I don't think you really need that information. And even though it's not a fun, it's also a lot of work. But you get gets valuable insight from recordings. So my advice is not to look every recording from beginning to end. This will take up a lot of time. Instead, check back the recordings off pages where you see a high drop off and see what happens there. See how where users interact with that page. Now most ALS have besides this free. Also the option to get a mouse movement map, and now it's moving. Map is basically heat map of where your website fishers move their mouth over a certain page on your website. Well, I highly advise against using these, and that's because there's never been a correlation found between where people move their mouth and where they actually look. So in other words, a mouse movement met doesn't tell you anything. Now a good two for your Web interaction analysis is Hajar Honduras fairy user friendly and affordable, so I can definitely recommend using this tool. Now let's have a look at book a bus while using Hajar like the Google Analytics data. This can be sensitive data, so I cannot show you everything. But I can definitely show you some stuff which will result in very valuable and interesting insights. So here we are in Hajjar and we are looking at heat maps off the Book of Bus website. So by ending, hit me if you can really see where people click on the page. Um, what we see is that a lot of people click on the form so that's that's a good thing. What all sees a lot of people click on these arrows. And in the expert review, we saw these arrows he's dropped on. Arabs don't work, so it's very important for them to fix that. So, beside the clicks, we can also see, like a mentions, the mouse movement. But as mentioned, don't use this data. This doesn't correlate at all with use Baker. Which do you want to see is the scroll that, Of course, 100% of visitors who come to this page. See this part of the page that makes sense. If you scroll down, you see people dropping off or clicking somewhere up here on this is very, very, very important for a book. A bus. They're gonna be happy with this finding because only 30% off visitors see these very important benefits over here. This is really the value proposition, as mentioned before off Bukoba. So this is definitely an inside you want to record. So I still have my Excel fall open. But I'll stop inserting insights here. I think you get the point. I'll use effective experiments for this ago to research and a new research name. Would your ah use your interaction? Now this is Put your missus just type that's named heat map. And again it's today. Just get some quick insights from this. So an observation. Only 30% off visitors see the important benefits and tech desktop heat map, and it's on front page and you can at whatever take you think is useful. So first observation. But that means obviously a 30% is here if you scroll all the way down the page. Yeah, barely. Anyone sees this really, really nice and high customer re few number. So let's add that as inside as well. Things only 16 percent C Custom RV. Few score, messed up front page. Save observation. There we go to very nice observations. We can see here another thing we can do. Of course we can check out everything looks on mobile, but I want to show us well, it's the recordings. I want you all the recordings and they have to go. That's, uh, that's a lot of work, but what I will show is at least one recording. So you see how it looks like Let's find, find an English one. There we go I think is yet New Zealand's. So we saw in Google analytics. They love people drop off on the front page, so I would advise to just look at this front page in your according otherwise, you'll spend a lot of time on the checking recordings. You can really see how people interact with your website and how to behave, which might lead to valuable insights as well. 20. Customer feedback analyses: customer feedback. It's a very valuable source for your research findings and there are many forms of customer feedback you can use. Let's start with surveys, pulls and feet back toe. All three of these are accessible through again Hajar. The two recovered in a previous lecture. Sarah face our questionnaires, which you can have your website fished or fill out in such a survey. You couldn't ask anything you need to know. You can ask questions related to the intentions, the motivations of the website fister, or how the website Fister thinks about your website. You can ask everything you need to know bulls I find very useful and I use them a lot with Hajar. A bowl is basically a mini surf eight where you ask your website fister one or two questions through a small pop up, which shows up at the bottom off the screen. Such a bump up such a bull Good, for instance, show when someone enters your websites in such a bull, you can ask your website Fister's why they fished your website today, or what they're looking for today on your website or what their first impression is another useful pull. Its exit. Intense ball. This is a poll which will bump up when a fister is about to leave your websites. You could, for instance, have an exit in temple in your cart. So when you're fister is about to leave your websites so it moves with the mouse upwards, a small pool will pop up and could ask a question like What is preventing them from finalising the purchase with you today, or what is holding them back from final ice in it? Purchase through an accident temple. You got a lot of useful information on why Fister's drop off. Another useful location for a pool is your thank you page. So you're thank you pages where Fister's come when they have converted. For instance, they bought something on your website or they signed in for our newsletter. When you show pool there, you could ask your Fisher what made them decide to shop with you today, for instance, or why they decided to sign up for your newsletter today by using pools. You would get a lot of valuable insights when you have a pool running on your Web site, aim for 100 to 200 useful responses when you get all your responses, group them fills them, so you get your valuable research findings. Now Hajar already shows you a lot off example questions, which you can ask in your pools. A feedback toe can also be set up through Hajar, but you see bailouts also a very popular and goods toe to use feedback tools with and you might have seen feedback tools before on websites you visited. It's always a small button on the side of a website, or I can in the bottom right, and when clicking on such a button or in Aiken, you can leave your feet back about a website. And a feedback to was fair. Useful because your website Fisher's really give you good feedback, which can help you to find bucks and errors, friction or destruction. Visitors will really leave valuable insights. Valuable feedback fruit, a feedback toe, another valuable research methods to get their customer feedback, our car shorting and treat this thing with guards or thing if cards with information displayed on them, which is available on your website. And if respondents sort those cards to get him so. But having respondents sordo scars with information, they're basically making your main navigation or how they think your main navigation should be structured. You can do a car short both online and offline, but aimed for at least 30 participants. But online you can do much, much more. The car certain will give you very valuable information on how to structure website how destructor navigation. So your visitor can easily find on your website what they're looking for and converts. A tree test is the same as the card sort, but the other way around. In this case, you already sort of the carts based on your current navigation or basically improvement. You want to make in your navigation and you ask your respondents to find something to find information in those cards. So by asking something like, if you're looking for item X or information white, where which is search for that in which group off carts Now you can see how easy it is for a website fister to find something on your websites. A great tool for doing your card sorting and treat testing research is optimal Workshop Ultimate Workshop Both has the card sorting and a tree testing and many MAWR research methods online The main advantage for doing it online is that it's easy to analyze so you can get a lot more respondents A to like Upton Workshop as all the data, all the statistics and just analysis off your card. Sorting your tree, testing a big advantage unit offline a static and have discussions with your respondents or have group discussions with their respondents. Other great sources for customer feedback are jet locks, group discussions, your social media channels, where people leave comments and questions and refuse off your products and websites on review websites or on your own website. Reviews will give you a lot of valuable information for your testing thing, but very, very valuable. Recent shores for customer feedback is your customer service. People after customer service are experts at your target audience. They're experts at your website visitor. These are the people that talk to your customers every single day. They have many phone calls. Many emails filled with questions, concerns, complaints and praises from your target audience and holds a ton off information and I condemn, definitely recommend you to sit one day with their customer service team and see what kind of questions and concerns come in and don't do this once. Do this every now and then, like every month or every quarter. You'll get so much insights on your website. Visitors for book a bus, I noted. Customer service gets a lot of questions one day or two days before the bus actually arrives. It's always a secretary or the one who booked the bus who ask questions related to if everything is set for tomorrow, if they're assured of bus will arrive on time if everything is going fine. So this is definitely a clear and valuable insight for our testing plan, because apparently the one who booked or the one who does checking needs certainty he or she does not want to lose face in front of the whole group they're responsible for. So, in short, customer feedback, it's a very valuable research source to get a great insights on. There are many customer feedback sources, so let's quickly check out how to set up a pull in Hajjar for book a bus. So in our analytics data, we noticed that quite a lot of people drop off at the front page, so be nice to show a pull to users who are about to leave the page with the question why they're leaving. So we're Hajar. We got the Bulls. It's very easy. The name of the bulls, Whatever you like, you're your website. Fister won't see his name and targeting a suit on desktop. If you want to show a pool and when someone's about to leave your website, that's only possible for best up, because what it does on what your notices when Mouse moved is going up. And that's when it is play the pool. And, of course, on phones, there's no much movement. Specified a page on the home page, the question that has made a quick question. Is there anything preventing you from asking a free quote with us today? And people can leave the answer here. This is a long text answer there more lots of more possibilities, of course, a thank you message when people left their response. The parents can change the background color. We can say if you want to pull on the bottom right of bottom left and, of course, to behavior. And in this case, I want to show the Paul launched a ball when users about abandoned the bitch on a desktop device on Gun Fister's to see the bulls often until they submitted response. That's whatever you like it on the show points, and that's it. That is how you set up bullets very easy. If you aim for 100 to 200 useful responses, you get some great insights. So we actually did this pool for a book, a bus and already started set up a research in effect experiment customer feedback. I included feedback or the information we got from customer service. So one or two days before the trip, customer service gets a lot of questions on. If everything is arranged and going well, the best will arrive on time and such. And in the pool and ex intense pull on the front page. Some fissures on noticed or most of them who responded to the poll said they were not certain about the details off the trip. Yet. When coming to book of us, that's a valuable inside. Of course, if you analyze all the full answer should get a lot of Moor insights, which you can plays here and use in our testament. So excited, full. It's also very easy to set up surveys on next without saving. Now it was very easy to set up surveys with Hajjar and you can ask website Fister's to fill out your survey. 21. Usability test: during a usability test, you will go through your website with an actual person, and it's gonna be someone from your target audience or an extra customer and to give this person assignments or several assignments, and you will see how this participants performs them on your Web sites. You will see this person go food of flow for your funnel, what this person does, how he behaves and where you get stuck. You will get a lot of information. What this person likes, what he doesn't like about your website to do. Such invincibility test. You could have a fancy left where you have a desktop or mobile, depending on what tester running. You could even have an eye tracker and the area where stakeholders can work to use baby desk being performed. And this is the best and ideal set up because not just you as an interviewer will learn from usability test. Also, your stakeholders can watch the use basic tests being performed and see the participant going over dare website. If you don't have a nice set up like this, that's okay. Start by fighting someone or several people from your target audience and perform usability test simply behind a laptop or mobile with you. But this set up already. You get a lot of valuable insights, which you can use for your testing to do a usability test. It's actually pretty easy to be a good moderator, requires some practice, but you can learn fast if you keep this in mind what you see here on the slide. First of all, it's very important to look at what the participant dust not so much at what he or she is saying off course what a participant is saying. It's important but has been mentioned, saying and doing can be completely different things. A participant could perform a certain action but not be sure why he or she did that. So therefore, it's important to look at what the participant does and not so much what he or she is saying. The second trip is to not make the assignments to specific make them personal. For instance, Imagine you were in the fashion industry, so you're sunning close online. Now you could ask the participant to find him or herself a nice shirt, but you can also make it much more personal. You could start off by asking the British mints What party is coming up and a participant might have a wedding coming up. Thank you. Respond like Okay, so a wedding is coming up. Can you find yourself a nice suit or dress or order close you would like to wear during the wedding, and this will make it much more personal and much more realistic. Now you will actually see your participants go through your website and find him or herself a nice shirt or suit for their party. You could make it even more realistic. You could give them a fracture, which they will actually have to spend. That will make it very real and very personal. You will really see the behavior, how people perform like they would do at home. You will see them, check the prices, check the quality, check the stuff the shoes made off, or see if the colors match. So make it as personal and Azriel as possible. 1/3 tip is to shut up. Really? Keep your mouth shut. I see a lot of moderators happily chatting, asking lots of questions, bombarding the participant with questions. But it's all about the participants behavior. It's not about your talking and all your questions. You really want to see the participant navigate over website and think a little bit out loud. But half the participants do the talking of the participants. Ask questions. It's not about you. It is about the participant. A good proportion for talking would be 10%. You do the talking and 90% the participant does the talking. Because if you talk a lot and as a lot of questions, you might steer the participant to a particular solution or a particular part off your website, and you never want to influence the participants during a usability test, you want to keep it as natural as possible and have to participant go over your websites like they are at home like you are not there. So don't steal your participants into particular solution. And don't judge really tell your participants they can do nothing wrong. If something goes wrong, it's your website that's wrong, not the participant. The participants can never be wrong, and the last thing is just to be relaxed. You're just having a nice conversation. Don't be all tense. Ask one question at a time. Re lecture minds relax your body language. Mind your breathing and be as clear as possible to the participant. Just having a fun conversation. Besides the tips, there are some questions you want to avoid when doing the use beta test. Don't ask questions like Do you like this or what? Do you think about something? Or would you use the website more often? If we edit distant website off course, As mentioned, it's important what the participants says, but the participant does not always know why he likes something. Why does something? And if he would return, we're off to through the website if something is at it, so try to avoid questions like these. Now there are some typical situations which you will come across in every usability test, and the 1st 1 you often see is that the participant asks how something works. What if this happens? Don't explain how something works. Have them figure it out. That's the whole point off usability test. So instead of explaining how it works, ask the participant. What would you do at home? Or what do you think you have to do? What do you think? How it works now? Another situation you might come across quite often is when a protest mint looks confused and it's what happened quite often. But simply just ask the participant if this is what they expected to happen, and you will get your answer and you will know why he or she is stuck. And the last one is the petition thinks he or she did something wrong and you have to liberate here again. The participants can do nothing wrong. It's your websites that's wrong. The participant is always right Now. How many people that you need in usability test? Because in the few can take up to 45 to 60 minutes need time to prepare, so it's quite a lot of work. And if you haven't heard this yet, the answer might come as a surprise because she only needs six people in a usability test. When you interview six people, you will find all the major issues and improvements which are needed on your website. If you do more than six, let's just say 10. She'll just be hearing the same thing. People do really tend to get stuck at the same point on your website. Don't doing his bit test once do you split test fixed issues and few months later, do use B test again. If you do use pretty test every six months every three months, fixed issues your participants will keep on finding new issues. You keep on getting ideas for your testing plan and improve your website. So in summary, a usability test will help you get a lot of valuable insights for your perverse optimization. Princess, don't do you speed test once but repeated every six months or every three months to get Maura and more valuable insights, you will keep on finding out what issues your website has and where participants get stuck so you can improve your conversion rates. 22. Competitor analyses: the website off your competitors can be another youthful source in your conversion optimization practices when you want to check out your competitors. Websites don't just analyze the websites off the big competitors in your markets which you know by heart. Also check related websites which you can find on Google for book a bus, for instance, a potential clients could search for coach hire for group travel on Google, the websites that show up here he wants and the life does as well. You could, for instance, and the lives that's up free results in Google because apparently those website Fister's haven't over. Let it needs with your website, Fister's said. These websites as well good results in very useful insights. When analyzing a competitors website, check the basics, See what value proposition is being mentioned. What are the benefits that are displayed on the website? What functionalities air there and also check out there refuse and their social channels for complains, praises and questions because this can also be very useful. If there's a lot of complaints on a certain topic for your big competitors, you might want to mention on your websites that you do this right And if there are lot of praises on your competitors refuse and social channels, that's apparently important to the customer in this markets. So you might want to mention that on your website as well. So your competitors website is another useful source for information. But please keep in mind that even your biggest competitors might not have a conversion specialist like you do on board. They might not do a B testing or do research like you do your research. So do check their competitors websites. But keep in mind that it might not be perfect, so always check it out. But when you find use for insights, put them on your testing plan. Whatever you find, always test it to see what works on your website and what doesn't. So let's shortly happen. Look at two competitors off book a bus. I know one competitor by heart that's flicks bus flicks. Books is a very large bus company here in Europe who also rents out private buses for group travel. So when we land on the website, um, we kind of see similar things with book a bus, you have two big hero image with the bus, which mixes with the Fishers intentions. We get a nice line here, which is again, like Book of Us, not a riel value proposition. What is interesting is that they emphasise a private bus with a driver. Ever see your foreign fuels, which looks like the same as book of us? It's just horizontal cuts. Amuse Piece on here did mention thus with a driver again, so he mentioned both here and here. Let's see if they do it anywhere else on the page. Yet here on also own private bus, which I mentioned here as well. And so the dry first only being mentioned hair on book a bus. And I don't think you see anywhere if the bus is private, so maybe flexible things there must have. Fisher's worried that other groups willing to bust as well so book a bus could definitely tests putting more emphasis on the driver and the fact that your private bus So what else we noticed very quickly. It's the pictures with the social proof it flicks. Bets can have those pictures off their clients. Bukoba should have the same makes it much more trustworthy. All right, let's let's ask for a quote I did it already. Sort of. Patients have to load. We can find it here, and what we notice is it's a much cleaner page. If you compare Teoh book a bus, there's much more going under is much more noise. So Book of us could definitely test to reduce some noisier and the lead some information to make it more clean. What's also interesting is that this itinerary and book a bus and this itinerary are exactly the same, and you can see the price difference. So, yes, book a bus does prove itself to be much, much cheaper. There's another company. I found this one through Google B. C s Charter Services. On what I straightaway see, there is no value proposition. There's an automatic rotating slider, which is just distracting. I cannot see a price quote. I guess I straight up straight, we have to send them something, including my email and my phone number. Well, I'm not ready to do that. Um, my stock photo. Never use stock photos. Use photos of her own staff. We controlled by this photo at some stock for the website. Doesn't look trustworthy. Adul um what else? This could be interesting, so they show how much better it is for the nature to truffle by tour bus. Oh, that's good. That's, uh, Book of us could test with that as well. And I see big client logos here, so off some very famous clients. So apparently this is a pretty good company, even though the website doesn't make a very good first impression of Mutis basically clients very good company. So with these insights, I place him again ineffective experiments, and we could use that for our testing plan. 23. Science and psychology basics: next up is perhaps one of my favorite parts in Converse optimization, which is science and psychology. First of all, because I'm very interested in you would behave here, I find it fascinating how people think how they behave and what their motivations are. But another reason why this is my favorite part is because it can be very helpful to be successful at conversion optimization, and I still see a lot of websites doing this wrong. Off course. Psychology is very famous online and in conversion optimization, but it mostly sticks to the basics. You see some social proof here with refusing ratings, and you get some scar steed there with, for instance, last tickets available last rooms available. But it can go into March more depth, and we'll see that in this lecture and in the next one. Let's start with the basics and we will start with the six principles off influenced by Children, and this is a good place to start off with. This theory, which is explained in his book Influence, covers all the basics you need for online psychology. The 1st 1 is reciprocity and reciprocity in means we have the tendency as human beings to return a favor. We want to repay our debt, and we don't like to feel inducted. So how come use online? Well, give something away. I'll fell you for free and it's gonna be information like a great advice or a great white paper. Your website Fisher can download or awesome email campaign or apart off your product, which give away for free by giving something away for free off value. Your website fister will feel somewhat in dept. It and therefore much more likely to comfort and pay for something later on. And the 2nd 1 is commitment and consistency. We, as human beings have the need and a tendency to be consistent. When we start something, we want to finish it an online. You will see this a lot in check out processes. You'll see all the steps a fish there has to take in order to complete the check out process. And when a rep said Fisher completes a step, most often, you see a check mark appear. So by displaying the steps and using the check mark, we have to feeling we started something and were more likely to finish it and therefore confers the third principle is very famous. Online social proof social proof is our tendency to do what others do, especially when we feel uncertain, and you will see this everywhere online. For instance, with reviews and ratings, a lot of websites used the principle off social proof as mentioned. Try to make it as real as possible. The first name a surname, a picture and a good customer story are much more trustworthy than just a single first name and one sentence off review. The next principle is authority, and this is our tendency to obey figures, authority and this principle. This tendency has made a huge impact on the history of mankind. Lots and lots of people have done great things by following great leaders like Martin Luther King, for instance. But in history, we also did a lot off bad things by following bad leaders like, for instance, what happened in Europe in the forties off the last century. So we have the tendency to bay figures off authority. How can be this online? Well, if your company is mentioned or displayed in the media, for instance, a newspaper or television, or when it wins reward or your website. Wins and rewards do show these icons on your website. If people see this Aikens, they will see us and authority and therefore more likely to be a U and more likely to convert. The fifth principle is liking when we like someone. When we like people, we trust them or we believe them more online. You can create liking by displaying pictures off your staff and off your team. You could even add a personal story off a staff member. Prefer Billy related to your product but showing pictures off your staff. People will see the real face behind the website, and this can create liking, helping people to convert more. And the last principal. The sixth principle is scarcity, and this one is also a very famous online. When something is cars, we evaluate more and therefore we wanted more and you probably notice from the last two tickets available. Last five hotel rooms available a temporary discount low on stock. You'll find it everywhere online. When something is scars, be valued more and therefore we wanted more so a website fister. It's more likely to convert, so these are the basics. These are the six principles off Children E. And you'll see this a lot online. Well, there's much more to this. I will check it out in the next lecture. Let's have a quick look at the book of bus websites and let's start with Reciprocity. Book of US goods could give something away for free, off high value in the hotel pool. We saw that a lot of people leave this page because they're not certain yet about all the details off their trip. So what book a bus could try or the people from book a bus? When people are about to leave the page, show a pop up with a dom illegible document with the 10 best tips to prepare for group travel. This will trigger reciprocity, and this will increase the chance of people returning to the website. Now if you go to commitment and consistency, let's check out a quote bitch because this would be perfect here, it says, How does it work? You have Step one Step two, Step three. But users, Your website. Fisher's already came to this page from here, so they're ready. They already did something so it could show up here like this side upon your trip or plan your trip with a check mark and death will fuel for a website fister as if they've taken the first step already, which triggers commitments and consistency. Next one is social proof and book. A bus has quite a lot of social proof already, and we mentioned they could make it more real with surnames and pictures. So no new insights there. Their authority book a bus does not have any media mentions or awards yet on their websites . If they do, they should definitely place it here, but also could signal both authority and social proof is if they have big clients over here , that would signal authority as well, then liking. I haven't seen any pictures off the staff online, so they could definitely include some pictures off the staff to show they're a real company with with friendly, happy people and work with passion on book of us Every single day. They do show relevant pictures, and I think they also show they understand the the website Fister. So that's also part of lighting. And, of course, the last in a scar. STI scars the It's very famous online, but I think scar City will be very hard for a book of us unless they have certain deals. But I don't think they have that to these. There's a new insights. I include them in effective experiments so we can use them for a testing plan later on. 24. Science and psychology advanced: with every assumption you make and every a B test hypothesis you might have. That's a huge chance it has already being researched by someone else, and you are very likely to find its research papers online. For instance, on a website like Google Scholar, Google Scholar is like Google, but it will search scientific research, and it's a big data base with a lot off knowledge, and these researchers have been done either online or offline. But you will get a lot off information on your target audience. Just go to Google Schooler or another research related website and learn more on your website. Fisher. Learn more your target audience. It is very, very insightful on Google Scholar. You could try search terms like reasons to buy your product or habits in online hotel room booking or, in the case of book, a bus habits and online bust reservation for group travel, you can search for motivations to buy, or it could search for Baver models. Or why does someone buy something, or why buy something expensive online? There is so much information you can find when you can only find the abstract over research paper, which you want to read from beginning to end have to pay a large sum of money to get the whole PdF. There are two solutions you can try. The 1st 1 is to go to Google and simply type in PDF and the title off the research paper you want to read. That's a big chance you'll find the website with the pdf freely accessible. A second solution is it's it I got from a student in this course. He provided me with the website the Hess. Almost every research article freely accessible. You can find a link to this website in a resource is off this lecture for book a bus. I also did this research. I checked out Google Scholar and got two very important, very interesting insights. The first has to do with accountability, responsibility and group thinking. And we saw this in a customer feedback analysis as well. We sold that the customer service off book a bus got a lot of questions one or two days before the bus actually arrives. If everything is okay, if everything is fine, if the bus will arrive on time and I got this from scientific research as well people needs to feel certain. They need to feel sure the best one time, especially when they book for a whole group and it don't want to lose face. The second insight I got was also very interesting. Apparently when people have to travel quite far, especially with the group, they again want to avoid uncertainty and start searching for information about the destination they're going to. But they do not do this on websites from the destination. So, for instance, when someone with a group is traveling to Barcelona, he or she will not look for information on websites from Barcelona. Apparently, according to science, this person is more likely to search for information on Barcelona on the website off the travel agency. So this is again another great insight for book of us. So with these two insights, which have apparently to do with avoiding a certainty, checking everything is arranged properly, not losing face weaken, definitely at these two insights to our research plan, and now we know a lot more about our target audience. So, in short use scientific research wifely, it can really help you to learn a lot from your target audience and get great a B testing ideas 25. Page speed and mobile friendliness: when you want to increase your conversion rates, it's also important to check out your page beat or pay stoning times. Enter mobile friendliness. Let's start with the page meet. It is getting more and more important because we as humans get more and more impatient. Just think about your own behavior when you want to watch a video online. When the video takes a long time to load, you're very likely to close to video and never watch it again. What could also happen is that one the videos loading or even when you're watching the video and you get a notification on your phone or your laptop, you check that notification and totally forget about the video and the same holds for websites. When your webpage takes a long time to load your physical, your website Fisher will get impatient. He will leave the website and therefore this person will not convert, and we see exactly the same happening in this graph as well. This graph comes from a study done by orange family with lots of websites, and what we see is that if the page load time increases so it takes longer for Web page to load the conversion rates drop besides defect that we're getting more impatient. We're also spending much more time on our mobile phones, and the conversion specialists in most markets really need a mobile first mindsets because the mobile friendliness off your website, its crew show for your conversion rates. Now, if you want to analyze your page speed and mobile friendliness, there are several free online tools online, which you can use. Let's see how we can do a page speed analysis for Book of Us, and for this we will check out to free tools. And we started with bass. Beat insights by Google simply by entering a webpage euro in this box and hit. Analyze Google, start analyzing and get a great looking report with details and scores. Now this might take a while, but here we go, and what we see is for mobile Bukoba scores 54 on a scale of 0 to 154 is average now for desktop book a bus very well. With a score of 99 I have to go back to Mobile. We can scroll that because this apparently needs improvement and Google will give more details on different scores on tips and tricks on how to improve the page. Pete, and this can get very technical. Ah, and might be very difficult to understand for you. But if you see that the patriot of your website needs some improvements, you can run this analysis and out of this analysis. So oldest information you can send it to your development department or I t department and a person there should be able to understand these details, and this person should be able to fix the issues. Therefore, improve the page speeds and therefore improve your conversion. Not a great free tool is webpage test and Web based S is even more details again can simply inputs the Europe Off weapons. And it will start analysis. And this will take a little bit longer than the Patriot insides tool from Google but will come up with a lot of details. It's sort of with Bates ever book us. This looks pretty good. Three. Ace. This leads need to live in more improvements, but I learned that it looks quite good if you scroll down, we get numbers, we get wonderful grafts. We get charts. There's a lot of information here. And if you click on a wonderful graph, for instance, you get even more details now again, this might look difficult and hard to understand. But if you send this to your development department, you, the person there, should be able to understand this and should be able to fix the issues and therefore again increase your conversion rates. Now let's check out the mobile friendliness. You again check out to tools, and the 1st 1 is My heartedness is by Google. Step my sides dots with google dot com and again here you can simply input the euro off webpage, and Google will come up with the loaning time on three G. And here it says, it's good. So here we sold based speeds of mobile. It's not that good, but here says it's good that therefore it's always useful to run multiple analysis with multiple tools. And it even says an estimation fister lost due to loading time, which pretty nice anger and get your free report with more details here now the second tool for the mobile friendliness. And it's really a mobile friendly test again by Google so you can insert your euro here starts analyzing. And according to Google, this book of bus page, it's mobile friendly. I get some more explanation here. So more details here again, if you see problems Senator R T Department Development Department and the person there should be able to fix the problems. So these are four great tools to analyse the page. Pete on the mobile friendliness. If you see problems here, you can have it fixed by a development department, and that should be able to increase your conversion rates because you usually get a much better experience on your website. 26. Bringing it all together: We now have a large list off research findings and observations, and it's time to bring everything together. We did a research for a book of bus, but it was a rather quick research for this course. If you do research for your own website, you want to spend many hours, even days, into your research because the more researcher do and a better research you do, the more input you get and the more successful your A B tests will be. But we did a rather nice research for a book of us, and now we have to determine if we're going to test something or fix something straight away. So implemented on the website straight away, an introduction off this course. We checked out the pyramid for Converse optimization and we will use it here again in order to determine what should be tested and what should be fixed when a potential improvement for your website. It's related to the definition off your value proposition. You always want to test it simply because we're not certain which value proposition which definition will work the best. We're also not certain which definition will persuade your website fister most in order to convert. When something has to do with technology or functionality, you want to fix that straight away. Technology is everything related to bugs and errors, and bucks and errors are major conversion killers on your website. You want to fix those as soon as possible, especially when a lot of Fister's see those books and errors. Research observations related to functionality can, for instance, be filters, not displaying the right results or your search not displaying the right results related to the search term. You want to fix functionality issues trade away as well. Usability. It's a bit more complicated. Usability is to ease your user experiences when going for your websites When going for your fund. When a usability issue, it's very clear you want to fix that straight away. For instance, when five out of six participants in usability test gets stuck with the same thing on your website, you want to fix that. But if usability issue is less clear, it's less certain you want to test that he wants to test. If usability change, you're making sourced issue and results in higher conversions. At the top off the pyramid is persuasion and persuasion is something we always want to test simply because we don't know. We're not certain what will persuade the website fish their best in order to convert so always tested perspiration. Now let's have a look at all the findings and ideas we got for book a bus. Here we are ineffective experiments again and you see all the research types I did for book a bus. And even though it was pretty fast research, we got quite a lot off observations we can work with at the next step to bring anything to get her is take all the insights and findings and create ideas to improve our websites and this week and find here. So from older research observations, I created five ideas to test on the Book of Bus Website and, of course, for a website. You will think of much more ideas based on your research insights. But for the sake of this course, I'll keep it to five. If you have a big e commerce website, you will likely end up with 30 ideas more based on your research insights. So the five ideas I got our creative value proposition containing safety and security own private bus with DR for affordable and hassle free. And we got this insight from heat maps from customer feedback research from competitors, from literature from your Ristic analysis, which is basically another name for an expert review. Other ideas. I got our show, more benefits and original arguments why people should do business with book a bus. We got this from customer feedback analysis competitors again, heuristic analysis and literature. 1/3 of their make social proved more real by mentioning surname and including pictures off the clients. We saw this with flicks Bus. So the computer analysis again we got this from literature and again we saw in the heuristic analysis fourth idea move over re few rating up from the food er to the top off the page because book a bus had a really nice review score from the clients resume heat maps that barely anyone sees this nice number and again in literature and the heuristic analysis, we saw that it is very important for book of us and the last one show people have taken the first step already on the quote page. We got this from literature psychology, which we covered, and I saw flex bus does this as well, so we also in the competitive analysis. So these are just five years, which will work with. But as mentions, you will probably frink off a lot more ideas for your website. Now. These are five ideas which you want to test on our website to see if it increases our conversion rates or not. But of course, we also have insights, research insights, which you want to have implemented straightaway, which wants a fixed straight away and a great example. Giving idea was the insight that this drop down arrow it's not applicable. It doesn't open to drop down menu, and we saw it in the Heuristic Analysis and Expert Review. But we also in Horcher on the heat map that a lot of people click here, so this is typically something which you want to have fixed as soon as possible. Other typical things You want to fix it straight away. Our older bucks and errors you can find on the Web sites. And of course, the website speeds When your website a slow there's no need for testing it with users when you make it faster, because making it faster will always result in the higher amount of conversions, so website speeds and bucks and ours are things that we want to fit straight away, and there's no need to testing it. 27. Setting up hypotheses: for every finding you got, which should be fixed. Trade away sent it to the I T. Or Development back lock. You might want to help the developers out a little bit by giving some sort of privatization . For instance, fixed the book for our first, which is close to the final version. So close to the sill, for instance, or fix the one first, which costs us the highest drop off but out of in debt. Send it to the I. D in the veteran back lock and leave it there for all the other findings you got for the rest of the findings. You want to test that you want to create hypothesis and run a B tests to see what works and what doesn't to start off. We now have a long list off ideas and research findings, and first thing you want to do is cluster the ones that belonged together and tell something about behavior off your potential customer. By doing so, you are creating behavioral hypothesis and behavioral offices are very important, very useful. It can really help you to be very successful at conversion optimization, and it can help you to learn and get our valuable insights about your customers, behavior, learnings and knowledge, which you can share with the whole organization. An example for book a bus We got a lot of research findings indicating that people need a sense of certainty and safety when they want to book online. For that group travel. And from all these findings, we can create a solid behavioral hypothesis. And that is that people need certainty and safety when they book a bus for that group. Travel. And with this pain for hypothesis, this is a nice bay for hypothesis. Were for finding which you contest, which is related to the behavior off the potential customer. Another one for book a Bus is that people need rational arguments. Reginal benefits before they book a bus for that group travel. And that's because they have to tell others about their booking inmate, definite seller manager or the group. Why they did business with book a bus. And from these insights, we can again create a nice and solid behavior. A prosthesis sending us something about the behaviour off the potential customer. When you have your behavior hypothesis, you can start verifying them with a B tests and these are just two bay for hypothesis, which being made for book a bus. But when you do the research for your own website, which will spend hours, days or even weeks on, you should aim for 5 to 10 gave ripe offices, and once you got them, you can start a B testing all the idea she got from your research. And with those ideas, and with a B tests, you can verify or rejects your behavior, right. But this is, for instance, if we run in a B test for book a bus and it ends up in a winner, it could verify behavior. Apophis is like, for instance, that the people need feeding off safety and certainty when they want to book a bus for group travel. And when you find tonight Beatus winner, which verifies today for a hypothesis, you learn something about your customer. You learn something about the customers, behaviour and motivations and needs. Now, if you find seven a big test winners verifying one behavioral hypothesis, you can be very certain you learn something valuable about your customers. So through bay, through hypothesis and a B testing, verifying or rejecting those Baver high. But this is You are doing research when your customers behavior fruit A B testing. And this is not just beneficial for your websites and for increasing your conversion rates . There's also very beneficial for your marketing department, both online and offline. It's also very beneficial for product innovation, and it's also very beneficial for the future off the company. So when you have for Baby Roboticists, which is verified seven times at least seven times, share its share your knowledge with the company. Now you might wonder why you need at least seven A B test winners to verify behavioral hypothesis is, and it is because the reason for a B test winner could be different that you actually think so you might find yourself and they beat this winner. So maybe tous generating higher uplift or your website. But the reason why this test wins is different than you think. So imagine for book a bus we have behaved right prosthesis that social proof in the early stages off the customer journey is very important. Now we might run in a B test where removed the social proof up on the front bench, and this results in a winner. Now it's very tempting to indeed think that the behavior a prosthesis is correct. That social proof in the early stages of customer journey is important. But maybe it was not the social proof that cost a beat us to win. That costs a beat us to generate a higher uplift high conversion rate on your websites. Maybe because she moved to social proof up on the front page, he moved another piece of content more downwards on the page. And perhaps that piece of content could have hurts the conversions. So you still find yourself in a beat as winner by moving the social proof score up on the front page. But it might not be cost by the social proof. It might be caused by the fact that another piece of content moved down the page, which heard of conversions. There could always be a different reason. Do you think? Why a B test one or not? And therefore you need seven A B tests. You need seven A B test winners to verify the behavior right buffet is, and that way it can be absolutely sure that the behavioral hypothesis X correct, and you guys shared with organization. So with these behavior hypothesis and a B tests verifying these Baver up offices, you don't only increase the conversion rates and improve your website. You get are valuable learnings. You are doing research on your customers Behavior and findings from this research can be shared with the whole organization. Now, for every test for every a B test you want to do on your website, you want to create test hypothesis and test hypothesis are set up as displayed on this life . So changing something into something else for a particular audience results in an increase or decrease for a certain K p I. Because off this and this reason, so for book a bus, we have the idea to change the very proposition instead of having one sentence. We want one strong sentence with free benefits related to the feeling off safety uncertainty for the website Fister when they want to book a bus or coach for the group travel. So the test hypothesis could look something like this, changing the vote proposition into value proposition with free bullet points related to the feeling of certainty and safety for all website Fister's on mobile desktop and tablet results and more quotes being asked because the website Fister will fume or certain and safe with their decision. Now this is a very solid test hypothesis. Now this might feel a lot off extra work. Sending up all these test hypothesis is for every idea you have and every desk you would like to do. But believe me, these death hypothesis is really helps you to focus your testing. With these hypothesis, you know you're doing the right thing and can generate earnings from oil your tests. Now let's do one more. Let's get the idea of moving the overall social reefer. Score up from the food er all the way to the top of the page above the fold, just below the call to action button. As this will likely help Fister's trust Book of Bus More Now the test hypothesis might look something like this, changing the position off the social proof score to the top of the page above the fold for all website. Fister's mobile desktop and tablet results in an increase in number of quotes being asked because the website fish there trusts Book of Bus more now this is another solid test high practices, which really helps you to focus your testing so in summary, make high but Mrs Start by clustering Richards findings, which belonged together and create behavioral hypothesis and then for every idea and every test you want to run, create test hypothesis. Doing things this way will not only help you succeed to increase your conversion rates and improve your website by doing it like this, you'll actually do research on your customer behavior. You'll get valuable insights and learnings and can share this knowledge with the rest of the organization. 28. Test plan documentation: now off course, we want to document our hypothesis and ineffective experiments. I have already moved, discard related to the value proposition from ID's toe hypothesis. When I click on it, you see that prosthesis here and I should do exactly the same for the review rating test of moving the reviews up. So I moved from my D's toe hypothesis. Click on it and I can include the hypothesis right here. Keep the check mark, and there it is. Now you probably notice a big message here related to prioritization, and we will have to look at that in the next section now. Ford of Behavioral Hypothesis. I include that in the research section. You'll see it right here. Here's the baby hypothesis. And every time I do in a B test and it verifies the behavioral hypothesis, I'll click on add observation and includes the AP test here. But we haven't tested anything yet. This can be believe it. All right, this is effective experiments. Of course, you can also do than excel, and you'll find a link to this document at the end of this section, the area of the best hypothesis you have to device the page and characterization comes in an exception. The hypothesis we have to brave. But is this here? Number of times has been proven in which page it has been proven. Thank you for watching the video and thank you for making it this far. In the course. You are about halfway now. Now, remember, if you have any questions, remarks or concerns, please feel free to contact me. I'll be happy to help you out. Now. If you like the course a wants, help me. I'll be very grateful if you can leave you short riff you hair on skill share. This will help me reach more people, and it will help others who are considering to take this course. 29. PIE, ICE and PXL: you now have a long list off hypothesis based on your research findings. Next up, it's key to know which hypothesis you have to test first on your website. In other words, we have to prioritize our hypothesis, and there are different frameworks to realize this. A very famous and often used framework is the pie score and pie stands for potential importance and east. But then, Joyce, how much improvement he expects. The change will make importance. Stand for how much traffic will see this change and how valuable is to traffic. For instance, there's a traffic come from expensive online marketing campaign, or is it a very important? Let it bake your testing on and is, How easy is it to implement the change? Now you give all these free a score between one and 10 you take the average, and that's the by score for your hypothesis. Obviously, the high boxes with the highest by score should be tested first. Ice is another famous framework, and it is quite similar to the pie sport. Ice stands for impacts, confidence and ease. Impact is what will be the expected impact if this works, and this, of course, is related to the number of visitors that see the change and the research sources you used to get to this hypothesis. Confidence is, how confident are you that this work and is the East off implementation? So is he Is it to make this change now? Buy an ice are nice and easy frameworks to use, but it has its downsides. There is one huge limitation and its effect that these frameworks are based on opinion. It's not based on facts is based on opinion, because if you look at pie potential, how do you know how much improvement you realize with their test? How sure are you? And the same goes for impact in the ice? How do you know how much impact this will make? Well, if you do love a B testing and get a lot of experience, you might get close to gift the right scores in these frameworks. But still, it's not fact based. And if you lot of a B testing, you'll see a lot off surprising results from her tests. So I like frameworks, which are much more based on facts, and a big sell framework is a great example. The exam framework is from Pepe, who runs the website. Conversion excelled of Come any made a framework based on facts. You can see this framework here on this life and ask your questions such s. Is that change noticeable within five seconds? Yes or no? It's a change above the fools yes or no, but also that hypothesis come from user testing or from data analysis and this framework. It's much more effect based and you will get a much better prioritization off my butt assists and therefore you will generate more successful tests and be more successful in your A B testing. But also, when we look at the B X out framework, we can still see two major things missing. First of all, it does not ask the question if the hypothesis is based on scientific research and this is very important because, as we saw, we can get so much insights from scientific research and the second thing that's missing. It's the question if the hypothesis or the idea behind the hypothesis, if that has been tested successfully on this website already. So with that in mind, I created my own framework, which is quite similar to the P excel framework, but with the necessary changes. You will see this framework in the next lecture, and of course it is up to you which framework you want to use. 30. A more complete framework: Let's covered a framework which I like to use. It is based on the frameworks we've discussed in the previous lecture and is based on my expertise and books and course. I've read and done It's fact based and hopefully covers everything for successful privatization and therefore successful a B testing. It starts off with the question. Has this idea being proven before on your websites? Every time a similar idea results in a successful A B test, it gets one point. So when they score gets very high. And apparently you've learned something valuable about your website, Fister's behavior or motivations. The next question is the change above the fault? Yes, one point No, Sarah points. And the same goes for the next one. It's a change noticeable within five seconds. If it is noticeable within five seconds. And apparently you're looking at a big change. So yes, gets one point and no get Sarah points. Next question. Did you get the hypothesis from usability testing yes or no? Did you get it from customer feedback? Web analytic analysis went into X analysis? Or did you get this idea and hypothesis through science for Oh, yes, get one point and no results in zero points. Now the next question is about the traffic. It's a traffic on the page or pages she wanted. Test on it, said Low one point medium two points or high three points and the percentage of river you going through page or page if you want to test on, they say between zero and 20% 1 point 21% and 60% 2 points I went in 61% or higher. It gets three points at last. Questions about the keys. How easy is it to implement this change and old frameworks take the implementation sign into account here. So how much Cody's needed, How hard is it to create? But I also like to include political discussions. Okay, so you might have a beatus, which you want to set up, and some steak orders or a lot of stakeholders don't agree with that because it doesn't match the Brent style guide order. Something else going on. Maybe they have no faith in the variations in your test, and you need a little meetings to convince these people. And these are also working hours for you. So how easy is it to implement the change, both technically and politically. Is it less than four hours? It gets three points. Is it 4 to 8 hours? It gets two points, and if it's less than two days, one point and over two days, zero points. Now let's look at the hypothesis for Book a bus and let's get the idea of changing the value proposition, as we did not do any a B testing yet on Book a bus that you get Sarah points for time being proven. But it changed is above the fort, and it is noticeable within five seconds. So each one point we did not do any usability testing for book a bus, so their points here, but in the customer feedback realized that the customers off book a bus needs certainty and safety when they book a bus for the group trifle online. So one point here in Google analytics, we saw little high drop off from the front page, so it's important test here gets one point, and in hip lifts and wept into X analysis. You saw that people barely scroll down and they missed the real benefits and via Proposition Off book of us So one point here in scientific research, we sold that people needs certainty and safety when they book a bus or coach for the group travel online. So one point here and traffic the traffic is high, so three points, usually for most websites. When you test on the front page, the traffic is low. But for book a bus, almost every fister sees the front page, and we're testing on old devices. So mobile, desktop and tablet. So therefore, the traffic is very high. Three points. The same goes for the percentage of revenue on most websites. The centers of revenue going through the front base is quite low, but with book a bus, Fister's have to see the front page in order to convert and you're testing old devices. So three points here and the East while chasing the value proposition is quite easy. We can do this within four hours, so three points here, making a total off 15. Let's do one more. Let's look at the idea of moving the overall social score up from the food or to the top of the page. It has not been proven yet on book a bus that au so their points here. But we're moving something above the fold and it is notable within five seconds. So for each one point we did not see the usability testing and we did not get this idea from customer feedback. No one literally asked if they can see the overall social score. But we did see a wet and latex analysis that testing on the front page is important. And in weapons, the accident assists you sold it. Barely. Anyone sees this nice score. So for both one point in science, we saw that people need certainty and safety when they book a bus for the group. Travel online and Social Proof can definitely help with that. So one point here as well, and that's in the value proposition test. The traffic is high three points and remember you is high because we're testing on orifices and old Fister's have to see the front page in order to convert so both three points. Moving to social score up to above fault is also pretty easy, and I don't think we need any political discussions on this so we could do this within four hours, which results in three points, making the total off 14. Now, these are the period ization scores for to test hypothesis. But of course, for your own testing plan, you want to set of the prioritization for every test hypothesis you have. Now, let's see how we can document this. And you've seen the Excel file in the presentation. It was pretty straightforward with the scores here and total score over here. But now you will see the true power off effective experiments and s defect. It's fair, customizable, because I can include my own prioritization framework in these templates over here. So I already set the Paradise ation score for the tested with moving the review rating up and you see the score off 14 and I'll show you how I did this with the value proposition test. So click on very proposition here still die processes. And this is what I created in the settings. So did the score of zero. It is a buffer fools. So, yes, it will give one point over here. It is noticeable within five seconds. We did not get it for usability test. They didn't get it from customer feedback. The song Web analytics that the front page has a high drop off. We saw that people miss the benefits in the hit neck. So weap interaction? Yes. Did you get it? Fruit? A scientific articles. Traffic is high. Revenue is 100% and we can set it up in less than four hours. So very effort per station. Score off 15. Now, if you have a lot of hypothesis which I assume you will have when you do your research for your website, you can sort them by purity score, and you can see which tests you have to do first. So in this case, the value proposition test. 31. Setup & AA test: it's time to start setting up our first tests, and for this we need a whole different skill set. Compared to the research face. When setting up and running tests, you need to know a little bit about statistics. I prefer believe you know, some website coat and we need to be able to monitor and analyze our test results. Fortunately, their many useful free tools you can use for this and we will cover all of them in this section. When you start a B testing on your website for the first time, you always have to start by checking the set up and running on a test. When you check the set up, you have to check one important thing. You have to check if the snippets off the testing tool. It's hard coded in the head sack off your websites. I melt implemented through a tech manager like Google Tech manager. If the snippets over testing toe is implemented food tech manager, this will cause flickering flickering means that Fister will see the website and then the website will load again with the changes off your variation. This will wake the website a lot slower and really influence the behavior off your website . Fister. Something you don't want now how do you make sure the snip it off the testing toe this place in the head Sack Ocean website? Well, the easiest way is to ask a developer, but if you want to check it out yourself, go to your Web sites. Opened up a chores off any particular Beijing your websites and find the coat and see if it's placed in a headache. Estes plate on this light. If it cannot find a goat, it's either implemented fruit tech manager order coat is not installed at all. Next, you need to run in a test and a test. It's like an A B test, but without changes between the two variations. So you have to variations but no changes between them. Therefore, a The reason why you want to do in a test is to check if the set up works. If data is coming in and if users are equally defied it over to variations. So in summary, if this is the first time you're in a B test on your website, start by checking the set up and then run in a atheist to make sure your data is correct. Air test results are trustworthy. If you don't know how to run test yet. Don't worry. We will cover every step together in this section. 32. Binomial and non-binomial KPIs: it's time to start with our first test. We have a long list off hypothesis and the hypothesis will be starting. Test first is, of course, the one with the higher spirit ization. Now, before we start with this set up off our test, it's very important to choose to go or gay P. I have to test and basically there two kinds of KP ice. Yes, buy no yo or binary KP eyes and knowing binomial KP ice. And the 1st 1 is easy. A binomial K p i is gay. P. I did result in either a yes or a no. So someone converted or he didn't comfort a website fist are left with email address to sign up for your newsletter order website fish and did not someone bought a jackets or did not buy the jacket. It already results in a yes, someone converted or know someone did not comfort a number only okapi. I makes things a lot more difficult. A number on the Okapi. I It's not a simple yes or no. It is a zero to whatever I number, you can think off an example from known by only okapi. I could for instance, be based shoes. Someone can see one bakery website, 10 pages. Someone can see 100 pagers or even 1000 pages if they feel like and working with number number KP ice makes things a lot more difficult and a lot more advanced. You really need in advanced and difficult test set up to get the right day dyin. You also need Maura offense and very difficult statistics to use, so it's much harder to analyse your tests if you're working with nobody only okapi ice. For instance, If you work on the website off a publishing company, I advise you to make the number one with JP I bio meo again. So how can we do this? Well, let's say, for instance, take shoes are your main KP I. So you want as many people as possible on your website to see as many pages as possible. And this is of course, a nun binomial K p I. But you can make this binomial by stating that your KP I is that your website fister. So at least three pages in the session now, if you analyze your test results, you can state that fister so at least feed pages, which is yes or he or she didn't see three pages, which is a note, and this makes it binomial. The same goes for the number of sessions on your websites. The number of sessions is, of course, a number only Okapi I. But you can set a target a k p I. So someone had at least two sessions on your website or someone returns to your website. This will make it again. Yes, someone had at least two sessions on your website or know someone did not have at least two sessions on your website. So by making a number no me okapi, I binomial, you make it a lot easier for yourself. You don't need to difficult and it fast set up and I don't need to difficult statistics. This will make it love, easier view to analyze your tests and look easier to get her valuable insights. Now let's look at book of us the hypothesis with the highest prioritization. What's hypothesis where we change the value proposition in order for people to feel more certain and safe when they do business with book of us. So this will be the test will be working on first and a golf. This test is of course, quite easy. It's a binomial gaping I because the go is that either a website fister ask for a quote order, website fish there does not. 33. Can you test here: the next step a student German if we can actually run this test or not, and this old depends on a number of Web site visitors and a number off conversions going for your test. A rule from in a B testing is that you need at least 1000 Fischler's and Freedom it 50 conversions for each variation your testing, but I barely see these numbers work if you want to make it statistically fillets with 1000 visitors and free on 50 conversions perforation, you need to find a big uplift. You need to find a big A B test winner. The change in your variation shoot result in a huge uplift in your conversion rates. For instance, if you work with 1000 conversions for each test variation your testing. You need to find an uplift off 10% on the KP I to make it statistically fed it. So you need, for instance, 10% more seals. If you're running an ICO Merce website, that's of course, huge. That's a big uplift. Just imagine the revenue off your website over Southern increasing by 10%. I've seen it happen, but it's not very likely it's a very big increase. But if you work with 10 thousands conversion sparked test variation, you could actually start detecting an uplift off 1% on the KP I. And it's always to do with statistics. The more visitors and, um, or conversions going for your test variations. We'll help you detect smaller changes, which are significantly fell It. Besides this, it is also very important to half your tests run for food, business cycles or full weeks. You want to run your test for one full week up to maximum off four full weeks mess because if you only test in weekend, your behavior or for customer could be very different from the other days of the week. So the change you're seeing, because after a B test, good work for the weekends, but you will not know the impact for, for instance, to Monday into Tuesday. He also don't want to have your tests running longer than four weeks, and that's because people can delete their cookies. If you run the test for longer than four weeks, you can be sure that a lot of people the lead to their cookies and when they return to the website, they could see a different variation than it sold the first time. This, of course, the future data and the lucha results and makes it less trustworthy. In order to calculate the number of visitors and duration your tests should run, we will use a useful Frito on a big test guy dot com. But before we start filling out our numbers and we will do that for the book of bus websites, it's important you are familiar with some of the terms mentioned in this tool. Let's start with one sided versus two sides. One side. It means that the change in a conversion rate cost by the variation. It's either bigger or smaller than the conversion rate and control. So, for instance, when you look at transactions for e commerce websites, you want to the variation toe have more transactions, and when you're looking at bounce rate, you want to see a lower bounce rate, so less bounces. So this is one sided testing. It's ITER. Bigger border change is smaller, and this is what you will usually use in a B testing. Now, if for some reason you want to see if the change is either bigger or smaller, so it could be both in death case you will use to side of testing. But in 99.9% of the beatus, you will use one side. It's now the power. The second term, it's The likelihood is that your experiment will detect effect. Wender isn't effect to be detected, and this might sound confusing. Technical. It is important. But this is all you need to know, because in a B testing, we always had the power to 80% and the power depends on your simple sized, effective to test and the confidence level. But in a B testing, we set the power to 80% and the governor's level is the percentage of times your experiment shows the same results when you repeat the experiment. So the percentage of times your results hold when you repeat a B test. If you said this to 90% it means that in 90% off the cases, your results will hold. If repeats AP Test. At this stage, when you calculated test aeration a sample size, you don't want to put the conference level lower than 90%. If you do so, it could fair homie that did change you find in your A B test is caused by chance and not caused by the change you made in the variation. All right, so let's see our desperation and test size or minimum sample size for our value proposition test. To fill out this numbers, you have to go to Google Analytics, and the only thing you'll need is at this segment. Set up a new segment, and you can do this from basically any screen, any section in Google legs. So the test is on the home page, so you could two conditions. Make sure your date is set for one week because the numbers here ask for unique Fister's on your desk page per week. So, in this case, the home page, if defined in Google analytics with flesh loads on the right side and you can see here. Of course, I can't see that show the data, but you can see here the number off users that happen on the home page in the last week. So let's assume this is 15,000. But then there's one thing to note because we want to do segments and we want to segment at least for desktop. So we need sufficient data to see not only the whole test but also what happens to visitors on desktops. So we'll add an extra filter, which is to fry scategory containing desktop. Again. I'm gonna show date up. Let's say number of users is 10,000 so we felt 10,000 here. All right. Next you have to know the conversion rate and for that so conversion rate for people who visited the home page. So with that, I'll leave. The condition contains that stop and go to sequence because first people have to see the homepage go to page contains slash and next we want ah, transactions per session to be larger than zero. And again, you need the e commerce set up for this. Otherwise, you can also search for Paige. Here we go again. And then, for instance, your thank you page that worked as well. In this case, let's keep it to transactions Verse Ishan larger than zero, so you will get a number here. You still have desktop visitors, and the conversions can a terrible versions. But let's say it's 2100 which is a conversion rates off 21% version rates, 21% expected Improvements over control. So this is the effect which you think you're variation will have to change the change of uplift. So which how much do we think the conversion will increase? Well, the food is 15% but that is a lot of that is very high and this number is subjective. So I need to do a lot of testing on the website to see your average result. But for now, let's stay on the safe side and say like 5% increase in conversions, which would be quite nice. Maximum number of weeks for a B testing. This is kind of optional, but I want it to be maximum two weeks now, said the hypothesis to one side. It's the power to 80% and the required confidence level on set it to 90% do not access. We need a minimum sample size off almost 7000 visitors. Burke test variation. So with our set up, that would require minimum desperation off 1.38 weeks and as mentioned, we want to test full business cycles so we round that up to two weeks off testing and that's it. We have to test the test. Aeration is two weeks on the minimum sample Size Bir variation is 6906. If you have more traffic, don't be modest here. You want at least this number in your variations Because if you go, if you want to do segmentation even deeper segmentation you need more data. So please don't be modest here. This is the absolute minimum. Simple size preferably. You want this a lot higher? 34. CRO for low traffic websites: in the case that you do not have enough traffic and conversions for statistically, fellas, a B testing you can and still should do conversion optimization. When you have a low traffic website, there are three things that you can do, and the first thing is that you still have all your valuable research insights, and what you can do is implement the ideas on your website, which are based on several research. Resource is so implement ideas with the highest priority score straight away. Imagine that you have 10 ideas based on several research. Resource is by implementing those 10 ideas on your website is trade away. You're very likely to see an increase in your conversion rates. You just don't know which ideas resulted in the uplift. But if implement five of those ideas right now and the other five a month after that and keep track off your conversion rates, you're likely to see an uplift. If you did your research right, the second thing you can do is still run a B tests, but with a lower significance level. This means that you need less traffic and less conversions to find yourself when they beat this winter. This way, your A B tests will not be statistically fell it But you will get an indication of what works on your website and what will not work. We will cover statistical significance later on in this course, and the third thing you can do is to trek micro conversions. So, for instance, when you are running on a B test on a product detail page often ICO Merce website Instead of measuring for the number of transactions, you could measure the number of visitors who reach your cart page. So the next step in your funnel as you will have many more CART visits compared to transactions. He needs a lot less data to make her a beatus. Statistically fell it. So remember, if you have a low traffic website, you can and still shoot do conversion optimization. 35. Setting up the test: All right, let's start building our A B test and this course we will use feed. They'll be over this, But there are many, many tools out there in the market. Besides feeding LBO. Often, I see it's a very famous and often used one. You also conferred a be tasty Google optimize and probably a lot more. In my experience, these tools do pretty much the same things. They're quite the same, apart from the price you pay for it. But they like to stress one thing. And at this that these tools are testing tools. You should use them to set up tests and have them spit visitors over different variations. I never used these tools to analyze my data and analyze the results off the tests. For data analysis, I use Google Analytics with Google Analytics. You can segment much more, and the data is March more trustworthy because the core off Google Analytics is gathering data and being able to analyze the data. Whereas the core of Fido Vo optimize Lee and any other testing tool for the matter is to set up tests and split Fister's over this variations so always set up your a B tests in your testing tool and do the data analysis in Google analytics. Here's how we're gonna do it for book of us. Let's set up our first test for book a bus, don't you? We find ourselves in the feet LBO to feed of your websites in the big test section. Let's trade a new tests which we will set on the front page of a book a bust that you And here we are in the fetal video editor. So for our test, we want to change the text over here and make it a proper value proposition. Now, there are two different ways to set up tests. First of all, you can use editor to change the chances text mover, Reese ice, um, and love other things. Or you could use the coat, which is JavaScript and C is this. Now we're very easy tests and small changes. You could use the editor. You could use change through this section. However, it's much safer to do this fruit coat in the editor through the chapel stripped or a C S s coat. And that's because when you change something in the editor, Fidel, the oval, right the goat itself. And that does not always work for old browsers or old devices. And if you it again, right, goat or you have a developer who can help you out The best ways to always make changes through the JavaScript and CSS possibilities in Philadelphia, but also in other testing tools. Now, already made the goat for this change will be quick goat to copy it from here. Copy this and JavaScript and got this in the C s s it done. And there it is, your private bust with Dr for affordable and hassle free. We work together with 800 plus certified best companies to get the best best for your group . Our customer service team will make sure everything goes smoothly, say, 15% to 40% by looking through a book, a bus. Now that is a value proposition. So we have that set up. Next thing you want to do is making extra variation. And this variation will be a copy off the control. And I'll explain why we want to copy control because he wants our users to have the same experience in the control and in the variation. Now, when the feed LBO coat loads on the website. People in control might have a faster experienced in people in a variation because people control nothing extra needs to be loaded. Therefore, you don't want any fish. They're going over the control. Which video bill or any other testing tool suggests you want to create a copy off your control and that Fister's split 50 50 over dese two variations. So when it takes a long time too low to feed over your coat, the people in control left the same experience as two people in the variation. Now, another reason for cup control is that you might want to add some extra coats to it. For instance, if you want to send extra information to Google Analytics, we will cover this in the next lecture. 36. Setting up the test advanced: as mentioned. Feed W O and other testing tools are perfect for setting up a B test and splitting users over the variations on the test. But these tools should not be used for data analysis. We have got Google Analytics for debt. Google Analytics collects much more data. You can segment your data, resulting in more valuable insights from your test results. So if you want to set this up, you want to place a little well, some lines off Java scripts at the end off your drive JavaScript coat. Don't place it in front. The reason for that is if your code somehow breaks and the variation is not noted, so the user just sees no changes. You don't want to signal Google Analytics. This user is in the test and see the variation. No, this will happen when you place the event here. If you place the event to Google and its to signal Google analytics at the end and your code somehow breaks, this line of JavaScript will never be read, and Google analytics will not be signals users in the test. So it's at this goat. I got it here already, so we'll place it right here. Now, I'm not gonna explain everything, but this is an interval to check it. Google Analytics is being loaded. And if it's being noted and events will be sent to Google analytics, if you check your reports and Google Analytics, you see events, category, defense, action, anything label. So let's say event Gutteridge A B test. Excellent. Is, um, whatever you like, maybe the I. D. Or 97 whatever. And the label is Variation one. Of course. You want to copy this and added to copy control as well. But then we named this copy control or just control. Now it's also possible in the settings off Adobe oh and other testing tools to link your tested Google analytics using a custom dimension. But I advise against doing so for a couple of reasons. First of all, a customer mentioned is automatically signaling Google Analytics straightaway if a user is in the variation or in a control. But again, when the coat breaks, you don't want Google Analytics to collect any data on this. You don't want this user to be in the data and to be in your desk data. So customer mentioned is out of medically in the beginning, off the coat. And it will tell Google Google Analytics this users in the test and see variation, even though your line of code breaks and years you wouldn't see a change at all. Not at a disadvantage of customer mentioned is that custom dimensions can change name in Google analytics, so you could change names or your colleague and change names. But if you want to check back on a test like you did a couple of months ago, it's harder to find it. If the name off the custom mentioned has been changed on the last reason is that the customer mentioned comes from cookies, so people will keep the cookie even though the test has stopped already, so data will keep coming in. It will keep on saying like, Hey, this user is interrogation one even after we stopped the test. So it's much more convenient to set up a new friends at the end of the codes, which you can easily and always recognize in Google analytics. There's one other nice trick you can do because you cannot just link your test to Google Analytics. You can also do it to torture and I'll show you how. Let's say he wants at a heat map to this test. Go to Hajar during the heat map. Let's give the name best I d whatever. Seven, Um, variation. Miss it, even for a variation. But you can also do it for copy control next on a number of pages. And here's the thing you want to do when you want to trigger a Java strip snippet. So let's say a B test variation. I d. Seven. No, now you can place a line of code again en sorry in your JavaScript again placed of buff to Google Analytics defense, I've got the line of code here again. I will share it with you and end of this section. So here we are at this here says, Basically, this checks before jars. There is this whole jar trigger and they just give the name off your job Script trigger, which in this case is a bit dissed. Variation. I D. Seven. All right. Next and you create the heat map and you can create a new heat map. Arrested quickly. Great new hit map for just I d seventh control. Next. Next, just with your a be best control 97. You can name whatever you like and copy this line of goat. Put it in copy control above the Google Analytics C. Thanks. And we changed it. Trigger name, too. Could be this. And there we go. So this is how we set defense for Google analytics. And this is how we trigger Hajar. Now, I want to show you one left. Think related to the defense. If you want something to be measured in Google Analytics like a click. But it's not in your Google and legs right now. You can also add it to the code off your baby test. So, for instance, we want Google one date on Google analytics own users who click this button, for instance, And then we can see the difference between the two variations with variation and control. Now, this is a living more technical that we could place it here. So this is the javascript goat that functions for all the variations and the control. Whatever you have, you have to go to ready. Here. Um, here we go. You hear what it does? It says when this and this is the Berlin Selector, is being clicked. Sent in events to Google Analytics stating a B test. It's again, whatever you like. Action, click Quote Britain Front page. Whatever you like. Of course, you can also add it in the variation in complete control. But if this this code is for both of variation in the copy control, so we can just place in the campaign JavaScript and CSS. Now, if for some reasons thes codes don't work, that might be because off a particular setting or how you use your tools or your segment of yours. It's good, for instance, be that you need to prefix here. If for some reason this doesn't work, ask your developer. Yeah, for him or her, it's very easy to fix it in summary When setting up your a beatus. Always try to make the changes for the JavaScript and CSS coat, so you ensure that your test is working fine on all the different browsers your users are using. And when you set up your A B test, always make a copy off the control, so your users have the same experience in the variation and control. When it comes to page loading times and with a copy control. You can have extra lines of code for data tracking now, at the end of the coat of both of variation and a copy of To Control at some lines of code to send defense to Google Analytics to signal which variation your fist there is seen and you can name this offense s your life. For instance, a B Test 87 Variation one or zero a B Test 87 Variation one however you like, and, however, it's easier for you to find your data back in Google analytics. If you prefer, you can add heat maps and pulls from Hajar to your control and variations. A consent. Extra events to Google analytics now in the screen recordings. I used to click on the orange button as an example to show you how this line of code works . This could be useful as users goods entered a quote page for different pages on the website , but it's mentions you can measure a lot off things with defense and can use that your A B test as well. Now, if you have any questions about this, please feel free to contact me 37. Quality assurance: we have finished setting up our A B test, but before he put the A B test life meaning before we allow visitors to enter the test and see the variation or control, you want to make absolutely sure that everything is working perfectly fine. And this is called the quality Assurance. We want to make sure that everything is working fine, because we don't want to be responsible for breaking certain pages on the website because of our A B test. And we don't want to be responsible, for instance, for the check out not working a full week and lung because our A B test Boston working well and broke a check out. You want to make sure our beatus works fine on different devices, different browsers and different screen sizes. And of course, we also want to make sure that data is coming incorrectly because otherwise our testing is useless. So let's cover the quality insurance here. We find ourselves in settings off feed LBO and what what want to do with this quality assurance is to check if the variation is working correctly on all devices, So desktop tablet mobile want to see if it works in different browsers and want to see if it works in different screen sizes. And, of course, we want to see if day that is being sent to Google Analytics. It's a good way to start on in any testing to urine feed. LBO disclosed this. It's the preview moat you can if you start with grieving mode and just check around as the first quick and easy check. But what I want to show you is how to actually see the test when it is life, because that's the most real situation. You can be a and to do so, we have to go to your ills and unable to campaign for a specific visitor group. Now, if you don't be oh, and May and mostly other testing tools I know optimizes the same options you can send in a lead test life, for instance, you're based on a cookie, so on a cookie value. So let's say the cookie name. We can choose whatever, let's say Tokyo and value off the kookiest test. Now, if you save this, only people with this cookie and we'll make sure it's only us. We'll see the death being life. So how do we do this? Be hurrying book of us in your browser type in Jaffa Scripts, documents, dots Cookie equals feet. O B O equals death. This video equals test is the same as this. Enter and cookies being set. Now to make sure we see the variation we go to utter, we make sure that the percentage of traffic we want, including campaign it's 100% because we want to see variation. And we want to customize traffic allocations for each variation. And this gay she want to set it to 100% 40 variation. So now 100% off the visitors off the websites who have this cookie see the dumping and from those Fister's 100% will see the variation. So if you save this and set this life, we will see the text because no one else will have this cookie now Probably easier way to accomplish this is not by setting cookie, because if you want to test on different defies in different browsers, you will keep to have have to set cookies. Your browser's another easy ways to select I P address. So I already have my address here. Many websites. You can just google it to check your I. P address is my uterus. I want to come in here. It's safe, I said. At this life, keep in mind that with I p address everyone on that I p address will see the text variation . So these might also be your colleagues who might be surprised. A certainly. See it changing the website. So be careful with that. But it isn't easy and quick way to set a test life just for you. Now let's start in good well grown. I already have to test right here. So we see the value proposition received a change with variation. We could make the screen size smaller. Everything seems to work perfectly fine, make a bigger again. And of course, we want to see if data is being sent to Google Analytics. You want to see if Google Analytics knows that we're in a test and we see the variation and for debt is a really convenience Google grown extension Quote the event trekking trigger. Now, if correct, the event should have been sent to Google Analytics stating that we are in the variation and considers by clicking on this icon and there we have it. You're if the event category A B test, the Action 87 and a label variation one now very important, of course, is to see if the foremost they're working fine. So it just feels at some quick information. I'll go from again from my City of Amsterdam because you remember, in the test set up an extra string of goat T sending events to Google Analytics with a click on this button. Of course, it's not really relevant for this test, but it's to show you that you can send extra information. And of course, we want this form still to work, even though we make changes in the code over here. So we get the growth and their way. If it's, this forum still works. Now let's see if Event is sent to Google Analytics. And there we go live to Calgary, a B test action click and quote, but in front, so everything seems to be working fine. In Google, Chrome Variation works all the functionalities on the website work. It worked in different screen sizes and day that's being sent to Google Analytics. Now, of course, you also want to check this and other browsers. So I already set it up in Firefox. And here we have again the value proposition we have toe before and see if we do smaller. And it works perfectly fine here as well, of course, want to fit out this again and of course, will reject the same for Internet Explorer. And can we felt proposition? Something goes wrong here, but I don't think it's anything to do with the tests. Make it smaller and bigger. And it seems to be working perfectly flying the last thing you want to do. Of course, it's also check if data is being sent from the control to Google legs. Of course, we use the same steps after control opened here already. I don't have to bother you changing all the settings in Fidelio. And again we can checked defense trekking tricker. Here we go a bit pissed 87 label control. So everything seems to be working fine for both the variation and the copy control. Now, of course, you want Teoh do the same steps for your mobile phone, so check the I P address off your mobile phone and check with both an android phone and IOS From, of course, this chrome extension event. Tricking Trekker does not work on your mobile phone, so you'll have to check that on your best, and that's it. This is how you make sure everything works perfectly fine in the test before you said it life and again. If you have any questions related to anything in this lecture or the lecture before, please feel free to contact me. So in summary, do a quality assurance to make sure that everything is working perfectly fine and a data is coming incorrectly said each variation life to your I P address or to cookie that only you f and check your variations on each defied your testing on, so invest up tablet and mobile when you're testing a mobile. Also check it out on both Android and IOS devices and see your variations into the different browsers and on different screen sizes to see if everything works perfectly fine . 38. Test documentation: what we haven't covered yet in this section is how to document every step. Ineffective experiments now get it Doesn't really matter which to use if use excel or project management tool doesn't matter that it is important to document every single step. So you and your colleague always know what's going on. So let's do this ineffective experiments. We decided to start with the hypothesis with the highest privatization which makes sense. And for Booker Busters waas the value proposition test. So we moved is to testing, and we are going to create our testing plan. Now it is possible to establish a connection between effective experiments with either feed LBO or after might flee. And this will help with data coming in automatically. But in this case, I'll show you how to set it up to create your testing plan without the established connection with VW. So again, oldies Fuchs here ineffective experiments are customizable, so you can remove or add whatever field you will like. But this issue used these fields are used for our testing plan. Have you already have to experiment? Name? We got this idea for use of testing that sort enough for your statement. Heat maps, customer feedback. We got it from competitors, literature, whatever on, Let's just just for the records experiment i d. I used to stay the idea in Fidelio. In this case, this was I D number seven. We can add a description if you like, and we already made the hypothesis. You can take an experiment lead so you can also sign a colleague two days from this case. It's me, you could say to start dates. So that's saying we start at 21st September and we calculated that we want to have this test run for two weeks, so we'll stop this test on the fifth off. Mike Taylor. Really? The estimated sample size, which was 7000 sisters at least, and 100% of traffic will be in this test. They were destined desktop, tablet and mobile. And that's Curiel Is book bust up to you, etcetera, etcetera, fast forwards. When we fill out all the forms and fields and the other screen sets, we see this view. If all the information we need the goals to statistics which are coming in which you have to fill in ourselves later on and we have toe screenshots. There we go, the control original in this case and variation one. Now back to ideas hypothesis. You'll see here that this just set to plant good experiments and see older steps here. Draft Bland's design and development You a pending approval approved life, etcetera. If this are too many columns, this is too much information you can again customized. It's so he might do something like selective protesting desk creation. Death, life. This cage refused default, so we planted tests already did that we designed the test infidel b o the development. We also did the Q A in the last lecture. Of course, you want to notify the stakeholders on that. We want to run this test so it's pending approval, and that's assume we got approval. So it's approved, and next step is to set it life off course. When you work with the bigger Ciro team and work on lots of tests in every column, there might be something going on, and this way it's very convenient and easy to see what is going on. An experiment stage off your conversion practices 39. Multiple tests at the same time: I often get the question if you should run multiple tests at the same time. And my answer is, it depends off course. You want to run as many tests as possible because running more tests means to increase the chance of fighting in a B test winner, thus increasing the gents of increasing the conversion rates off your websites. But the tests you're running should not influence each other. Otherwise it will dilute the data, now the easiest way to run multiple tests. At the same time. It's to test mobile and best off separately, and a really advise you to do so because a user might show very different behaviour on desktop compared to a user who is on mobile so you can run both for tests at the same time . If you do best up and mobile separately and for the rest, for all the other tests, be smart. If tests can influence each other, they might want to wait with that second test. But if more tests at the same time, it's possible and have sufficient visitors ankle versus to do so. I do advise you to death as much as possible to find as many as possible test winners 40. Multiple tests at the same time advanced: a more advanced method to run multiple tests at the same time is to have to test run mutual Excuse it. Mutual exclusive means that if you have more than one test running at the same time, a user can enter only one of those tests. And when users in the test, his user won't enter another test, which is running so an example a website Fister, who is part of test number one, will not be part of test number two or free. And like flies this way, the test will not influence each other. Making your data much more reliable now to set up off mutual exclusive tests in the testing tools can still be a bit difficult. It's quite complicated the first time you tried this, but there scoot documentation online, you confined to help you out. For instance, Feed of you has documents how to set up mutual exclusive campaigns and also optimized me as good documentation. Here's the documents Neutral exclusive experiments in optimizing. Now, if you re food is documents, it might look quite difficult. But after you've done it once after, you have set up a mutual exclusive test the first time the next time it's just repeating the steps, and actually it's quite easy. The documentation is actually quite good, so if you follow these documents step by step, you should be able to run mutual. Exclusive A B tests if off course, you have enough Fister's ankle versions to do so. Now I know feed over euro is planning to make the set up off Mewtwo exclusive tests a lot easier. But if you have sufficient visitors and sufficient conversions and you want to run multiple tests at the same time, take a look at these documents and give her the shot run mutually exclusive tests on your website. 41. Setting the test live: Joseph Escapee, I You have set up your A B test. You make sure that everything is being measured A make sure that everything is working correctly. Now it is time to set it life, Head fish. There's an trigger A B tests. Document it and monitor your test. Let's set our value proposition test life. And to do so, we first have to go back to the settings in Fidelio Member for the quality assurance we set the best life to our idea. Arrest cookie. So we want to disable that feed over us. Many second galleries, um, based on traffic on defies you can create your own, but for this test, we want to have everyone see the variation or C control, so you can disable this option and now O visitors will by default see the test safety's now what we also did for the quality Assurance waas the traffic allocation. We still want 100% for fissures in the test. If you have lots of traffic on your website, you can reduce this number based on your minimum simple sites. But we have to reset this part. Of course. Remember, we don't want any traffic to go over control, but you want 50% of traffic to go of Variation one and 50% off all traffic Girl over the copy off the control Terry go 50% on variation, 50% over the couple control. And now we can close this. That's APIs. This should be fine before you said it. Best life always go to summary and check everything one more time we can ignore Dismissed For now, if video code is installed correctly, it will tell you here you have to variations Progression one and copy control. It's a 50 50 split. We have a goal tracking on successful H, but as mentioned, we want to track the goals in Google Analytics. But Fido Video wants us to set up a goal. It's it's mandatory and we see the 1% off traffic is included. You see the company name, and that's it's everything set up correctly. We know everything's working fine so we can start the test. And of course we want to document it as well. So back to effective experiments. We can set this test to life and there we go. The first desperate book of us is up and running 42. Keeping track of the test: with your tests now, being life, you can see the data coming in every hour or even every minute if you like. And my advice, A subject. The test every single day, starting two hours after you launch the test. And this is not just for fun. It is, of course, a lot of fun to see the data coming in and see how your test air test variation is doing. But it is also very important for the quality assurance. Every day you want to check the preview mode you're testing to like Www Oh, and see the data coming in in Google analytics. And this is very important because I've seen it happen quite a couple of times when there has been idea release. So something on your website changes. So the code of your website changes a B test brakes. It stops working, and by checking your test every single day, starting two hours after lunch, you can see how your test is doing. If everything is still working and you're sure that everything is running smoothly and get the correct interest worthy data into Google analytics to see if your test is still working fine, you could start by checking out the preview moat off your testing tool. But what I would advise is to set up a new custom report in Google and Alex and for discussed a report. You want to use general Metrics to see if everything is working. Fine. So let's see what we got. So is it a title for discussion report to be a beat death check. And we got some general metrics which are users Recessions. The bumps rate, number of page views, the pages, procession effort, session duration. So how long is usually test on the website and the final version, which in this case is a quote request. If you want to see this for the device, get curry, the browser and a browser version. So this is the data in general. But now we want to see the data for the users in both of control and variation. Now, remember, by with setting up the test, we set up a piece of code to send data to Google analytics, whether users in the control or the variation. And we did this through an event and we will use this data right here. So you create a new segment by clicking here and let's name the segments a big test Ivy seven Control with three conditions on events. Action. Waas. Find the seven and event label WAAS control. So now we'll include all decisions, which, at the event, action off 87 defend Label off control. And we had safe and I will do exactly the same. 40 Variation. So name with whatever your life. A B test idee. Seven variation. Pretty condition, even action being again Ivy seven and even label. It's variation. What, And there we go. You will end up with reports looking like this. If you see somewhere data, for instance, in for some defies or some browser or some browser version, something is wrong and should check it out. This report is very useful and as mentions check. It's two hours after launching the test and every single day until you finish test. It's a lot of fun to see how your test is doing, and it will make sure your test is working fine. 43. When to start analyzing: when you reach the end of your desperation and you have to need a number of Fister's in your test, it's time to stop the test and start analyzing well before you stop the test. It is very wise and importance to double check if you have to need it. Number of visitors in your test based on calculations you make before launching a test. And it's because there could be many reasons why the number off website faced or fluctuates . It could, for instance, be the case that your testing toe is not sending every fister into the test. But it could also be the case that a big marketing campaign stops while you were running the test, and this costs a drop in the number of Fister's on your websites. So when you reached the ends off her desperation, always double check if you have to. Sufficient number of Fister's in your test, and if so, it is time to stop the test. Never stop a test sooner than debts. You can only stop their A B test sooner if the results are very, very poor, or when it causes a buck or never, but never stop a test because you think it's statistically significant and the variation is buried and control for the ordinary rounds. Always let the test run, consistent with the calculations you make before and only then when you have reached the end of her desperation and you have sufficient number of Fister's in your test done, you can stop the test, and then it's time to start analyzing and see how your test and a test variation before stopping and tests in Fidelio or any other testing tool is, of course, a piece of cake. You go to the A B test and you hit the pause or stop button, whatever so you are using. Of course, I also want to document defect that you stops this deft Sobek. Ineffective experiments in experiments Few. We moved this test from being life to complete it. This is completed pending reports. Now, If you were open, you can state when it when it that stops, and you can at the data in here as mentioned when you establish a connection between Tokyo and effective experiments, data will show automatically. But still I would advise you to check this because you want to get your data from Google analytics. Enough from Fetal Rio and it's it. We stop the test. Now it's time to start analyzing it. 44. Frequentist vs Bayesian statistic: Before we start digging into our test data, we have to determine which method of statistics we're going to use to analyze our results and their two methods you can use. The 1st 1 is too frequent ist method. This method looks at statistical significance. The second method is to be Asian method, and this looks at probability. So the chance that a variation will outperform will beat control, and more and more conversion specialists start using deviation method. And it's also my preferred method. When I do my Sierra work and I'll show you why. When you have your test data, you can block that in this matrix, you have what's being measured in the test and what will happen in reality. Ideally, you want to find a test winner. So the data off your test states that variation will be to control. And the best situation is what happened in reality as well. And this is the best quadrant to be in, because this means you have a successful test and you will make the right decision. Now it could happen, and it will happen a lot that your test measurements states that variation will not outperform control so variation is not better than control, and if this happens, hopefully this will happen in reality as well. And this quadrants you will not have a test winner, but you will still make the right decision in the two other quadrants. You will make the wrong decision, and this is called a force positive or falls negative. Now let's look at the frequent ist method. This method looks exit discourse, significance, and it's perfect for science and medicine studies, and especially with medicine studies, you don't want to leave anything up to chance. The confidence level, when research is related to medicines can go as high as 99.9999%. But with conversion optimization and with a B testing, we are not in the business of medicine. We are in the business off making money and learning about our customers behavior. So how high do we have to set the confidence level then? Well, if you said the confidence level for a B testing too high, you will decrease the number of winners and you will end up with a lot of force negatives. And it's a shame because that means you're missing out on money, not aviation method looks at probability or chance. So the chance that the variation will be to control and with using deviation methods, you leave a bit more up to chance, and you're likely to declare more test winners. But because she do so, he ends up with less force negatives. On the contrary, you might have more false positives. But if the test measurement states that the variation weed control, this might not happen in reality. But the chances are very, very, very small that the variation will actually performed worse than control, especially if he did. The tests are right, so with the right restoration and the right number of visitors, so to risk worth taking when you used to be Asian methods, you will declare more test winners. You have less force negatives, probably more false positives. But by doing so, you're not missing out on money. Another reason for me. Why a used to be Asian method is defect. That's much easier to explain to stakeholders. Statistical significance is still a bit hard to explain and quite hard to understand. Not a lot of people fully get, but statistical significance is aviation is a lot easier to understand. It's simply the chance that variation will beat control between zero and 100%. It's much easier to understand for the stakeholders as well. Now what is an acceptable probability when you used to be Asian methods? Well, that is up to the business and the business case, and it all depends on how much risk you want to take for the website I work for. I usually use a probability off at least 75% so the chance that the variation will beat control should at least be 75%. If that happens and the business cases positive, I declare a test winner. Now the next lectures, you will see how this works for book a bus. 45. Analyzing test results: So we finished a test and I want to start analyzing it. And in the light of test results, we again will set up a custom report in Google analytics. One want to see is the event action event label a number of users. So in this case, we will see the number of users who are in either control or variation. We only want to see the event action and even label when event category contains a B test when it's exactly a B test. So where there is this report, click safe Now again, you get general data and we have to add extra segments. So we created new segments on Let's Name with a big Test ivy seven visitors and you will see why in this segment you'll set up the conditions exactly the same as she did in Attesting to We Go to Destin Toe. See the euro includes book a bus, don't you? Slash, says the home page in Google Analytics. This is the page, exactly. Matches this left, and if you set up a test for certain defies categories, you want to edit here as well. So device get it for me, could be best up. But in this case, we said with just for everyone visiting the home page were both mobile best up intended. So he hits safe. Next. We want to make a new segment and want to see all the fissures in the test who conferred it . So this name it's a B test. Ivy. Seven conversions. You can leave device, get a green here, but in this case, it's for all the users, so we don't have it. They will set up a sequence first, and we'll need to test has to see the homepage or debates where the test is running in this case, the home base and next you want them to convert in this case, go to completions greater than zero. So the first segment you want to see every fizzes, er, who has been in a test. And in this segment we want to see every fister who has been in a test and convert it so it's safer as well. First, you want to check if there's no sampling. If Google analytics gets love data and it starts simply and that makes your data knots or less trustworthy, so make sure there's a green tick mark here, and it says this report is based on 1% of decisions. So if every set of fine you should get a report like this, first of all, you want to check the differences between the users between the number of users and if the differences between the number of users in the control and variation is too big. If it's larger than 2.5% something is wrong. There's some our somewhere with the set up. A might want to check it out with a developer in your team. But if the distribution between a number of users in the variation control it's almost the same, you can start analyzing and for debt. We will use the the Haitian calculator, also by a B test guys. So we saw in the data that users and a sort of control for 15,000 with 3000 conversions and if variation user be users, be there also 15,000 with 3154 verses. Now, before we hit maker population, we want to set up the business case as well, because that's very important. So this desperation days was 14%. We indeed included 100% off the traffic efforts. Order valued. Well, let's stage. That's a visitor who comforts is worth 30 euros to book a bus. And when we implemented tests, you want to make at least an additional 10,000 euros extra because we have to use developers to implemented test. So we calculate this business case and now started running a simulation where X checks the chance of being sort of variation outperforming a control. And in this case, the jets will be out. Forming a is 98 0.4%. You see a lot of data here. So the variations and up 5% and a gents off at least earning 10,000 euro extra. It's 97.5%. So these are pretty good numbers. This is You get this from your test and should be proud because this is clearly a test winner. I just broke down. We see the business case and we see that estimated total contribution is one on 17,000 over one on 17,000 euros extra. So this is a great results, and this is definitely a big test winning. You definitely want to implement the variation, but to get more learnings. You want to segment your data as well? Maybe there are differences between desktop and mobile or differences between the traffic sources, and we will see that in the next lecture. 46. Segmenting test results: If you have sufficient best data, you can use segments to generate extra learning about your customer behavior and to to deeper analysis on how your baby desk performs. There are many segments you can use. I always advice to second for the five skate to curry because the behaviour off a website fister on best up is very different from the hate behavior off a website, fixed or mobile phone. But there are a lot of other segments you can use. For instance, the mobile phone Brent, the traffic source. A new first returning fish there. The location, time of day, day of week browser. It is whatever your business needs or whatever it's relevant for your business to generate extra learnings about your customers behavior. When you use segmentation, it is very important to make sure you have to sufficient number of visitors and a sufficient number of conversions for each variation to analyze your test results. So the segment our results we can use our custom report again. So here are making Google Analytics in the custom report for her a B test analysis an additional at the overrule tap, which I made in order Sigmund's you get simply at extra field to the mentions off reports. So this report make sure reflect flat table in the type of report. And that's whatever segment you need. So, in this case, to the five Scategory, so you can see if there changes between Destin Mobile on tablets. I selected the phone Brent, so you can see if there are differences between Samsung, Apple, iPhone and Sony and any other mobile Brennan, for that matter. We comtech differences between browsers, and you can check differences between the traffic source. So let's check out. If I still saved is your board again, you go to device tap. You should get a A report looking like this so you can see the differences between the desktop users in both variation and control and the differences between the mobile users in variation and control. And you can use this numbers again to fill out an aviation calculator and see if the variation is better than control and you can see the business case. So let's do this for desktop. So for desktop, we see we have 10,000 users in control with 2100 conversions and for variation if 10 thousands with this many conversions. Desperation was 14 days. Average order value. Like you said 30 Euro Spiric aversion. In this case, you want to make at least 5000 euro when we want to implement the change in case of a test winner. There we go. Simulation is running and we see there's a 96.4% chance on best up that would be up from Asia. Variation will be to control, and we can do the same for mobile. For mobile, we see 5000 visitors with 900 versions in control and five visitors with nine hundreds 45 conversions. Here, Desperation days is still 14. Average order value is 30 and again we want to make a minimal revenue. Yields off 5000 calculate the business case and there we go also on mobile phone 87.6% chance that be up from a Now do keep in mind that when we calculated a minimum simple size for this test, it said we need at least 7000 users per variation in this case for mobile phone, get 5000 users so strictly spoken. This is not statistically felt it so you don't want to report big on this, but you can use it as a learning as an indication. And this is a clear indication that this test will work on mobile phone as well. And the same goes for the business case. It is a nice total contribution, But be careful communicating this number because she have insufficient data in this segment . But still, we can learn from this and see that it is pretty good a mobile phone as well. Also, what you might have noticed is that this business case is based on six months time. That's because you never know what's gonna happen. It just might win now. But after six months time, it might not win anymore or competitors might have copied it. So that's why this total contribution is always based on six months. So going back to the report and Google Analytics, you can choose whatever segment you need. Whatever segment is useful for your business. Uh, and whatever segment is useful for your learning. Now remember, we also place an actual line of code in our test to record button clicks so we can see differences between control and variation in the number off clicks on the orange button on the website. It could make us reports for that, but you can also find it here in Google, and they actually go to behavior topping fans and year. See the event category, event action and event label, just like you set up in your coat. Now see the differences in the control and variation on these events. You can make segments, and we actually already make these seconds when we checked. If a B test is running fine, so we have them here, so you have to control and variation. Remember, you want all the recessions that include event action i d. Seven and defend label control Or in case variation, you want toe event label to be variation if you want to. Second this even deeper, you good at extra line and for answers. Want to see Defy scategory contains death with these seconds for control and variation, you can not only see the differences in defense being sent to Google analytics like clicks on the orange button, which you can use these events to go through every single report in Google Analytics. So you get the maximum number off earnings from your test results. So you segments to get extra learnings and extra insights from your A B tests. And if you do there two things that you have to take into account. The 1st 1 is that your segments should have sufficient data consistent with the calculation she made before starting a death and a 2nd 1 They said, When your segment a lot, you increase the chance of finding yourself a false positive if your find your A B test to be successful for a very specific segment. I had fights to run the test again, but only for that segment and see if your results hold. 47. Reporting on test results: Creating a nice looking report from your test results is useful for two reasons. The first reason is internal promotion. When you share your test results, your insights and learnings you got from a test with the stakeholders in your organization , you get more people on board. You get more people and majestic for your Converse optimization practices, and you will get more support. The second reason is, of course, documentation. Whenever you want to check back on a test and see the results, it's much easier and nicer to do this with the report, you'll get a quick over few on what was tested when it was tested and what the results were . And, of course, when you get a new conversion specialist on board, So new converts specialist in your team this birth can quickly read for all the reports and see what tests have been done and what the results were. If you make of reports and you're going to share it with stakeholders, it is very important to keep it simple. I never copy and paste roll data from Google Analytics right into a report. Make it nice and easy looking. A good report should have to following sections. It should start with the reasons why you did this test. And also, of course, the best hypothesis. Next, you want to mention the set up off the test. So the duration, the segmentation and in Maine, KP eyes off course, you want to include the results for each man KP I. And don't forget to include the business case when you find yourself a test winner. If you want, you can add other useful insights for instance, which you've got through segmentation and every report shoot and with learnings and recommendations and, of course, your name for internal promotion. 48. Draw learnings from the test: Now that you fully analyzed your A B test, it's time to draw. Learnings and learnings are very important because if you really think about what you've learned from your AP test, you will get her valuable insights. Your customers be a here and come up with new and very good A B tests and these air tests, which you can run in the future. Now. We spoke about aviation calculation and to the claret test winner. I advise probability off a least 75% with a positive business case, but for a valuable learning, a learning that says something about your customer behavior. He wanted probability to be a little bit higher, so you're absolutely certain you learn something valuable. You're not running into a false positive. Ford is learning. So for learning to be valuable and to be correct, you want the probability off at least 85%. And if you find preventative 85% it means you did a very good A B test and that the outcome of this test can confirm the behavioral hypothesis. It can be approved for the behavioral hypothesis. Besides that, when you find the probability of 85%. You can also add it to the period ization score for related tests. Now, when you did your test analysis, always ask yourself this free questions. What else can I learn? What do the results say about my customer? And what other tests can I think off for our book, a bus? A B test? With changing the value proposition, we found a probability hired at 85%. So this means it's proof for our behavior Apotheosis, which states that people need certainty and safety when they book a bus online for the group travel. And with this result, we can also add it to the characterisation for related tests. So what other tests carry? Think off now that we know the results from this A B test off book a bus. Well, we could run a test which repeats the value proposition on all the pages off the website. Or we could run a test where we at a top art of website with use piece communicating certainty and safety for the website Fister. When they book a bus or if the business is up for it, we could test with different forms off guarantees. Now we can add all these tests to our testing blend, prioritized them and test them later on. So, in summary, learnings are very valuable. They help you keep improving your A B test you can build up on every learning you get are valuable insights about your customers behavior. And most importantly, this all helps to generate MAWR test winners. 49. Example report: So here we have an example off reports you can make based on our test results, which you can sense throughout the organization into your colleagues. So this is really bored for the A B test off the value proposition. We started with the reasons and hypothesis. So the reasons are, which is so in your research. Why did you decide to set up this test? So there, so it. In scientific literature, we had proof fruit customer service with song analytics that are so I drop off from the front page and we saw in heat maps that website Fister's Miss the big benefits on the front page. After visions. You state that hypothesis, of course, once government communicated set up off the test, so they want to use screenshots for this as well. You see the control and you see the variation which has to change to value proposition. You see that gave the ice or to get the I see the segmentation and just duration, which in this case was two weeks. Of course, once communicate results and it's mentions, don't copy data from Google Analytics or from aviation calculator make it as easy as possible for people to understand, you have to keep in mind. Not everyone understands what we ask over to specialists are doing to make it as easy as possible. So in this case you see an increase of 5% for variation, there's a change will be out from a by 96.4% and we get a small table with the number of users quotes being asked and the conversion rate well, if you want, you can show segmentation as well. So in this case I decided to include the desktop and mobile, but also with the remark that there's insufficient data for this. But if you want, you get at more segmentation. It's whatever is useful for your business and for your test. And, of course, where your colleagues to learn enough get include the business case. So in this case, we saw that when it variations implemented, it will generate an extra revenue off on our 17,000 euros in six months time. This helps the the probes owner Teoh decides how important this change on the website should be. Of course, yes, to do with a very long I tea bag lock, and he can decided purity upon this business case. Enough getting through learnings and recommendations as mentioned learnings are very important in this case. The results good confirmed effect that users need certainty and safety when booking us for group travel. But of course, more tests are needed if want to confirm such a baby for a hypothesis, you need at least seven tests that confirm this, but you could definitely stated the good fight proposition does increase number of quotes being asked on the website off Book of us, don't you anyone to end states with recommendations And, of course, for this one implemented variation to generate extra revenue, I prefer be as fast as possible, and you want to keep testing with variations related to the certainty and safety. So an example. Repeated value proposition. The quote page at a top art to the website and test with guarantees and want to enter presentation. Your your report with your name here. It's quite big, however you like it, but do state your name so your colleagues know who did this test, and it's also useful for internal promotions. Now, if you have a UX designer designer in your team, you might want to ask this person if he can make it much tempered for you. So you your reports look very nice and very official. This is just a nation. Simple reports. I hope you like it. I hope you have an idea of how to set up a report. And why this importance for you and the business? 50. Finalize: we're almost done with the test. We just need a few more steps to finalize it'll and wrap it up. The first thing you want to do is to our guys to test and investing toe. By doing so, you're making sure that no one will see the test anymore. And David stops coming into Google Analytics. Next. You want to document all the learnings you've got from your A B test? A good place to do this is to documented where you documented all the learners you got from your research findings. Off course. You also want to document your new test ideas, your desk back lock and for the other tests in a test back lock. You want to read of the prioritization based on the inside she got from this AP test? And finally, when you find yourself a nice test winner, you want to send it to I t so that the fed uppers can implement it on your website. So let's go ahead and finalize this test. You will start off by archiving it best and infidel be Oh, this is easy. We find ourselves in a B test section. We check this check box and click are now. We're absolutely sure that no one will keep seeing this test and data will stop coming in now for the documentation. Ineffective experiments going here. We're here. We have to test stated in complete Benning. Report said gets to finalized click edit details and we can feel out everything we got from the test. So when when did it stop? We can state the number of visitors, the number off conversions, the significance level. Effective experiments does not allow a aviation on measurement yet, but in this case has just saying 95% significance. Pick the outcome. Basic, successful and again at the learnings and recommendations we got from the report. And there we go. We hit to finalize Glick. Yes. So now the test is finalized. We see the screen shelf, the data on everything. We implement it. So go infected experiments view You noticed death if has moved from complete too final lives. Not a nice thing about effective experiments is that it generates reports from your test. You can find it here and can customize this. But in my opinion, not enough yet. Your own report is much more customizable and looks much better than this. But if you are in a hurry and I need the facts reports you can use to search boards generated by effective experiments. So next up is documenting our learnings from our A B test. Now, in this case, we had a massive winner and got some valuable learnings and want to document that somewhere . And a good place to do this is in the research section. Now, as she might notice the layers off. This section has changed during the recordings off this course, but the French marriages are still the same. So in his research again documentary learning from a B test. So let's go ahead and go to this behavior or hypothesis that people need certainty. A few research The ad observation you could ask the A B test thanks you for like an observation. So this a B test confirms this behavioral offices. Now, if you want to include other learnings from your test in research, I recommend you to do so and could do it here as well, or in any other until you're used to keep track of everything that's going on. No. Next you want to document your new test ideas based on the papers. I beat a B test yet some new ideas and what's include them here as well. So with one idea for a value proposition unquote page, you can fill out this again as you like. We also had it's obvious Bieber related to certainty safety, security, and we had a test idea to tests with girl and teeth. So, as you see, we did one a beatus. But from the test, we got free new ideas. So in that way, your conversion practices will keep on going and going and going and will keep on improving your website, improving your learnings and getting much more tested winners every time. So after test aviation want to redo the irritation, and now we have only one test here and hypothesis, but I'm sure you have much more. But this test moving the over review rating up from the food or to the top off the page. It's also in some way related to people feeling certain when doing business with book a bus so we could times proven so the brief Xabi test most of big winner. So this has been proven before, So we had a one score to this and you'll see the privatisation moves from 14 to 15 now. The last thing, of course you want to do is Center a B test winner to the I T. Beck lock, and you can do this with a simple email or project management toe. However, you do this in your line of work and that's it. Congratulations for making it this far. We have finished the test. Now it's time to start a new one. 51. The success formula: success For a Syria specialist, it's usually the number of best winners, and a normal valuable learning should get from your A B testing. Well, there's a simple formula to make you more successful. And if you're aware of this, this can really help you. The former is your success equals the gentle success times to frequency. Chance is all the time you spent on finding the perfect test finding the test winner. This will definitely be your research, but also analysis off previous test You finalized It came your test set up like making a perfect coats or making a beautiful to sign and even meetings could be part of this. Your frequency is simply the number of tests you run. So an example. Imagine that you spend a lot off time on your research, so you increase the chance of finding a test winner to 100%. But you spent so much time on your research that your frequency is only one A B test for a month. This will result in 12 test winners per year. Now, if you spend a little less time on your chance and let's say it decreases to 25% which have more time to run tests. So your frequency is eight a B test for a month diesel result and 24th test winners per year, double that of the previous example. And this is so much more success in the long run, this will make a major impact on your website and the business. Take a look at this table. What you see is if that if you spent a little less time protest, so you decrease your chance. You do a little bit less research. Your coding is not as perfect. The same goes for a design, and you don't keep analyzing and segmenting the results of previous A B test. But therefore you increase your frequency on the long run in the year time. This can generate a big uplift, a big uplift and revenue for your organization. Off course. When you have a hard time finding a B test winners, it is useful to do the other way around. Try to decrease the frequency and spend more time on your research so you will find more and generate more best winners 52. Get people on board: another way to be a more successful Ciro specialist is to inform people in your conversion optimization practices. This will get them enthusiastic, and they will understand what you do now. We already spoke about reporting after every I'd be test. You do create a nice looking report and send it to people. Send it to your colleagues in the organization. This will give them a much better feeling what you do. They will understand what you do, and they'll get Maura and Majestic about Converse optimization. But besides reporting, there's March more you can do, you could, for instance, hold presentations about our work or hold crisis like have people predict which variation that test will win. You could keep track of who's the best predictor and give out prices. You could even hang a poster on a prominent spot in your organization where a lot of people will see the poster with the leaderboards off their quizzes. He could also hold Q and A sessions and lunch sessions. And, of course, you could hold big competitions where teams off colleagues come up with test ideas and you can actually test them and see which team came up with the best idea Which team had a test winner By involving people zero, you will get them on board. There'll be more and majestic about your work. They will support you, and you will be more successful at your come first optimization practices. 53. Overcoming the confirmation bias: people want to be right about how they see the world. They want to be right about their beliefs, and they want to be right about how they think that things work. And we don't like to be wrong and to confirm our beliefs and our few of the world's. We seek information which confirms that we are right and we overlook or ignore contradicting evidence and contradicting opinions. Take the original off the earth, for instance. Both biologists and religious persons have lots of arguments how they think the world came to be, and the same goes for conspiracy theories. If you really believe that something happens, you will find evidence for debt. But you also overlook or ignore contradicting evidence. This is called the confirmation bias. It is our tendency to favor information which confirms our beliefs while we ignore or overlook alternative information. Now ankle version optimization. This is a huge pitiful. When we have a certain strong belief about our websites, you will start making mistakes in our work. For instance, we will look differently at research findings. We will only look for findings which confirms our belief and ignore other research findings . We will also start to ask suggestive questions in our research. For instance, what do you love about Hispanics, or what do you find difficult on this page? Maybe the person in front of you does not love your product at all or doesn't find anything difficult on your page. And also, when analyzing your A B test results off course, you really want to find it a detest winner. So if you torture the data long enough, set up a lot of segments. Even if data is too low, you're probably able to find why are prosthesis is correct and why you have a test winner. Because we're certain that artist results in a winner. As conversion specialists, it is our job to overcome the confirmation bias. Of course, you want to be right, but it's so important to get trustworthy results and roll the right conclusions from your testing and from your research. And there are three things you can do to overcome the confirmation bias. The first of all is simply to accept effect that you can be wrong and be completely fine with that challenge. Your own beliefs. The second is discussed findings with your Ciro team or with product owners and really challenged each other on each other's findings. And the 3rd 1 is to create an environment where you celebrate failures, see few hours as learning There's nothing wrong with having and a B test that doesn't win or with research that doesn't get you any useful findings toe overcoming the confirmation bias. It really helps you to be more successful in Sierra Oh! 54. Build a knowledge database: by building and knowledge database. It will be a lot more successful in your conversion optimization work. If you keep on doing your research and you properly documents or your research findings, as well as all the findings and learning should get from your AP tests, you will keep on increasing your own knowledge. This will be really helpful to improve your websites because you are becoming more and more knowledgeable about your customer, and this will lead to much more success in your A B testing. He will get a lot higher percentage your best winners and it's mentioned you can share your learnings with other departments, such as the marking departments and the Product Innovation Department. But besides that, knowing so much about your customer and your customers behavior also gives you a competitive advantage because you know something about your customer, get your competitive, probably doesn't know and to increase his effects and to get even more knowledge, you can also get the learnings from other colleagues who work on experimentation like the page marking team or even marking team by sharing your learnings with them, you really strengthen each other now in your documentation, make sure you used to write taking, so it's east to find the learnings you've got and so you can keep on building your knowledge database. 55. The ethical side of our work: we have come to the final section off this course. Before we end this course, I'd like to tell you an important message in our line of work who will influence people on our websites without them knowing it, who also learned things about them. Things that don't even know themselves like certain types of behaviour will drive them or the motivations. So this means there's an ethical site to converse optimization, and I really hope with old and know that you got from this course and all the work you will do, you will use it for something good. Please don't persuade people to buy bet products or BET services. If you do, the website Fisher will regret doing business with you, and this will lead to by remorse. And it will backfire in your ratings and refuse and a number of products being returns to organization. Now, if you have great products or great service and give your website fish, they're the best possible experience on your website. The website Fisher will return, were often do more often business with you and tell others about it generating a positive word of mouth. So I hope, with all your knowledge that you will use it for something good and that you will inspire others to do the same 56. Thank you: Thank you. Thank you for following the whole course. Now, if you still have any questions, doubts, concerns or like to share successes piece of free to contact me. I'll be happy to help me out, and I want you to be successful in Converse optimization. Now, if you have any suggestions on how to improve this course, please let me know as well. I want to make this the best Converse optimization course on the Web and help as many people as possible in our great line of work. Now, if you like this course, I'll be very, very grateful if you can leave a review. This will help me as well as others who are considering to take this force. Now things again. Thank you for following the whole course and good luck putting everything into practice.