Transcripts
1. Introduction: Hi, my name is Veronica Pinchin, and I'm a product manager at Mixpanel. In today's world, there are so many products out there, and making sure that your users have a really delightful experience in your app or website is really critical, because there's no need for them to use your app or website. There are a lot of options out there. Product analytics enable you to understand at scale what works well, and what's not working in your app or website, so you can really improve user experiences, drive retention engagement and ultimately, keep your users happy over time. Classically, product analytics tools are used by the product teams. So, this could be a product manager, a developer, or a designer. However, product analytics can actually be useful for more rules than that. For example, an analyst that wants to understand which feature to invest in next, or possibly a marketer that wants to understand how people are behaving in their app or website after receiving a specific campaign. In fact, if you had any role in building or bringing to market a digital product, then product analytics can be helpful for you. So, before I joined Mixpanel, I was a product manager at Google, and there we had a whole team that would help us understand how our products are being used, what's working well and what's not. The thing that drew me to Mixpanel is that, we enable product managers that don't have the resources that Google has, to get that same depth of understanding, usage and behavior, and use it to shape really excellent products. In this class, we're going to start by walking through setting business goals. Then, we'll talk about how to create implementations back or tracking plan. And finally, I'm going to walk through three of the most common reports in product analytics; segmentation reports, retention reports, and funnel reports. So, my goal for you after taking this class is that, instead of using occasional Ad Hoc analysis and intuition to build your products, you actually are using data at scale in a product analytics tool to make data informed decisions in every step of building your product.
2. What is Product Analytics?: As your app or website starts to grow, there's no way you can possibly talk to every single person that uses your product or even watch them interact. Product analytics come in there to basically help you see at scale how people are using your product on average, and to start building better experiences based on the most common behaviors, patterns, and flows through your product. If you look at big tech companies today, companies like Apple, or Amazon, or Google, they're using every piece of data that's sent to them in order to improve your experience over time. As a quick side note, when people think about analytics, they're often thinking about marketing analytics. Marketing analytics and product analytics are a little different. Marketing analytics is really focused on how to get users. So, understanding campaign performance, understanding how to get people into the product. Product analytics, on the other hand, is really how to understand how to improve the experience once they're already in your product. So, if someone's using your app or website, how to get them engaged over time, how to keep them coming back, and how to make sure they have a really great experience. I'm going to be saying the word product a lot in this class. When I say product, I'm really talking about digital products. So, not things that necessarily are in the offline world. Digital products are often apps and websites, but it can also apply to things beyond that. Things like your fitness tracker or your Google home, or Alexa, they don't necessarily have a screen. Behind every product is a business school or multiple business schools. That could be something like increasing revenue, getting users coming back to your product more or user growth. Product analytics really is about defining what those goals are, identifying the steps to achieving those goals, and then, improving your product or experience, so that you're making an impact ultimately on your business schools. An example of the type of decision you might make using product analytics is Netflix auto-play. Netflix, I'm sure it could tell that people were binge watching shows. So, rather than making you play the next one manually, they decided to switch to a world where they auto-play shows for you and basically, eat up your entire Saturday. This was also a great business decision for them. So, it was a change to the product they made that ultimately helped them achieve their goals. Product analytics works by using really simple code, that tracks user activity on your app or website. Those activities are called Events. For those events, there are also details like where are the user is coming from, what country they're in. Those details are called Properties. Every event and property that you track is a piece of the puzzle that you can aggregate to really see the overall picture of how people are using your app or website. One of the trickiest parts about product analytics is that you can track an infinite number of things. If you do that, you're going to have all this data thrown at you and end up with a lot of charts, and not necessarily a lot of insights. So, what we're going to focus on in today's class is how to decide what you're going to track and how to ask the right questions. So, that you're getting valuable insights for your business. Next, we're going to start talking about the business goals that you set to provide a framework for your product analytics.
3. Defining Your Goals: The reason that it's important to set business goals is because there are millions of things that you could track in your app or website. If you track every single one of those, it's going to be very time consuming, super expensive, and you're going to end up with a lot of useless data and not a lot of insights. By setting business goals upfront, you can provide a structure and framework for the things that you're going to track to make sure that you're getting valuable insights out of your product analytics. Some example of product focused business goals are things like user activation. So where are people dropping off when they're on-boarding onto your product? Things like user retention. Which users are likely to come back and which users are likely not to come back? It's also things like user referrals, which users are the most likely to invite other friends to use your product? There's also user engagement. Which is, which features are people using the most or the least? Beyond product goals, there is a lot of other goals that you might have for your business. Goals around marketing, finance, or sales. All of these goals are part of the continuum that help really drive the success of your business. But today, we're really going to be focusing on the product goals. In my own experience, where I start is I look at what the business is trying to achieve overall. At Mixpanel, one thing that's really important to us for example, is making sure that people are coming back to Mixpanel on a daily basis to get value out of the data that they're collecting. So, as part of this, one of the goals that we have is around viewing reports. We want lots of people to come back and do a lot of different reports because for us that's a good indicator that they're getting value out of their data. One simple example that we're going to be using throughout this class is that of a music app. It's something like a Spotify or a Google Play Music. If you think about that app, the things that you might care about are the number of songs that people are playing, how often they're coming back, or maybe the number of people that are upgrading to a paid plan. Those would be your business goals and you can set up your implementation tracking plan, which we'll talk about next, to figure out if you're achieving those goals. When you're thinking about how many business goals you want to track, there really is no right answer. If you're a small website just starting out, you might only have a couple. There might be one, or two, or three things that you think are really important to driving impact for your business. But as you start to scale, that could blow up to having multiple goals across several teams, and so you could end up with a lot more goals than that in a large product analytics implementation. So, at Mixpanel for example, we do have some overarching goals that are really the north star for our business. But for me specifically, when I'm product managing the machine learning team, I have a specific set of things that I care about, that are very different than another product manager who's running the mobile team they care about. So, those goals that are specific to the machine learning team are really the things that help guide my day to day decisions in terms of product development. So, before we move on to the next step, you should take a moment to write down two to three goals that you think are inline with your role and your business. From now, we'll go on to create implementations back around those goals.
4. Creating an Implementation Spec: In this lesson, we're going to go over how to create an implementation plan or a tracking plan. An implementation plan is a document that helps you map your business goals to specific events and properties that you want to track in order to answer questions about those goals. Your implementation plan is also a kind of map for your developers to implement a product analytics platform like Micpanel. The reason that creating an implementation spec is really important is because there are so many things that you could track and if you track everything without knowing upfront the types of questions you want to be able to answer, you're going to end up with a lot of data and not really any idea what to do with it. So you actually don't need to be technical at all to write an implementation spec. All that you have to do is start with those business goals that you defined in the last step. Then, for each of those goals, break it down into the key questions that you really want to understand about your product and about user behavior in your product. Then from there, you identify the user flows or patterns of behavior that are going to help you answer that question. Once you have those flows identified, all you have to do is identify the specific events and any properties that you care about that map to those flows and that's your implementations spec. So you'll only create your tracking plan once from the ground up, but it really is a living document. So you'll have a tracking plan that you create upfront, your developer will implement it. But then as you make changes to your product, to alter the flows or the key behaviors that you're driving through your product or as you launch new features, you're going to go back to that tracking plan and update it with new events and properties that you are going to care about. So, let's look at a sample tracking plan for a hypothetical Music App. We find that using a spreadsheet to create your tracking plan works really well. Because it helps you organize both your business goals as well as the questions you want to answer and the events and properties that you want to track. You can see on the right hand side that we've identified the business goal that we're going to look out first. In this case, is acquisition. We want people actually logging in and using our Music App. So next we're going to look at how you actually fill this out. First you have to go back to your business goals that you originally stated. For example, if what you care about is new users in your product, then identify the key activity that actually will indicate that you have new users signing up. In the case of our Music App that would be completing the sign up process. So what that means is that we're going to need at least one event that's looking at people that are completing the sign up process, but there's also some important steps that happened right before that we're going to want to track as well. For example, the user will have to open the app, next they're going to want to actually begin the sign up process and then they're going to complete the sign up process. So those are really the three key events that we're going to track in that flow. The reason that it's important to track every step of the user flow is because if for some reason your sign ups are much higher than you expect or much lower than you expect or even if they're right where you expect them to be, you want to know why. So, at each step in this flow there could be reasons why people are being more or less successful at completing them. If for example, you find out that the people that are trying to log in with Twitter instead of Facebook have a much lower successful sign up rate, that could lead you to investigate what your sign up process looks like for people that are signing with a Twitter account and maybe make some improvements there. So, depending on the goal, there can actually be one or many paths that allow you to understand if people are achieving that goal. For example, in your music app, there might be a couple of different ways that someone can upgrade to a paid subscription. You're going to want to implement tracking for every one of those flows, if upgrading to paid is a goal that you care about overall. However, for other business goals such as new user acquisition, it could be that you've only built one sign up flow, especially if you're smaller in your business. In that case, there might just be one flow that you track. You'll notice that for each event we have listed here, we have multiple properties that we want to track. This is pretty common. These properties provide detail about the event that might give you insight when you're doing your analysis later. So for example if you look at the log in event, things that we might care about is how they're logging in Facebook or Twitter or other things like their account type, follower account or when they last logged in. Looking at this spec you might actually think of some other things that matter to you in terms of logging. So for example, I might think that people behave differently in my Music App depending on where in the world they are. So I might actually decide that in addition to all this I want to add a new property that is the country that they're logging in from. When you're building your implementations spec you definitely want to include any event and any property that you think might be meaningful when you're asking key questions about your business. That being said, it can sometimes be human tendency to want to track absolutely everything. If you do that, you do run the risk at some point having so much data that it becomes hard to extract the pieces of the data that are really meaningful and that do drive impact. So, the great thing about a tracking plan or an implementations spec is it provides that structure in that framework to make sure that the data that you're sending in is all high quality and high value and also understandable to your team when they're running product analytics reports. So, it's useful to note that when you're filling this out, there's no one right way to do it. For example, I like to fill out the metric category first, so things like acquisition and then go straight to the triggers and events that I care about. That being said, a lot of people prefer to first do the metric category and then look at the business insights that they want to get. So, for example, in our log in event, we want to understand potentially pattern's about how people are logging in and so that we can figure out which platforms to target for advertisements or promotions. That's sort of the business question we might want to ask about log in. From there, you can then define the triggers, events and properties that you want to track. Depending on your role, you might have a different level of comfort with talking to your developer about technical considerations. The great thing about an implementations spec is it really provides a bit of a map for you to have a constructive conversation with your developer even if you're not very technical. So, no matter where you are in the spectrum, this is something you should be able to share with the developer and basically walk through how you would actually track and record each of these different events. It shouldn't be something that you have to get into any specifics in terms of the code or the implementation on the technical side. There are also a tone of resources available online to help you when you're building your implementation plan. I'm going to link in the resources section to some documents that can help. So sample tracking plans and also some developer documentation that you can always send along to your developer so that you can be speaking the same language when you're having these conversations. This might seem like a lot when you're looking at the spreadsheet, but if you actually break it down and take it one goal at a time, it actually becomes pretty easy and gets easier the more that you do it.
5. How to Approach Your Data: So at this point, hopefully you have your implementations spec done, your event and property tracking in place, and you're sending data to a product analytics platform. This is when the fun part starts. From here, we can now walk through different types of reports and analysis that you might want to try on top of your product analytics data. Now, it's worth noting that I'm going to be using Mixpanel for our demos today, because it's a tool that I like and that I'm familiar with. That being said, there's a ton of different product analytics platforms out there, and the reports that I'm going to be talking about are types of reports that are really consistent across each one of those tools. So even if you're not a Mixpanel user, you'll still be able to apply these lessons to other product analytics tools. When you start running these reports, you're going to want to start with a hypothesis. A hypothesis is an educated guess that you can prove or disprove by digging into your data. A simple hypothesis might be that the likelihood to upgrade to a paid plan is different for users based on which country they download your app in. So in that case, you would look at your plan upgrades and segment it by country to see if you have a higher percent of users upgrading in Mexico or Canada versus the US. It's helpful to start with hypothesis because there's a lot of data and there's a lot of events and a lot of properties that you would look at in your product analytics tool. By starting with a hypothesis, it really gives you a frame for the right things to look at, the right reports to run to make sure that you're getting a lot of insight from the data you are collecting.
6. Report 1: Segmentation: So, we're going to start by talking about a segmentation report. Segmentation analysis is really all purpose, very powerful tool to help you answer complex questions about user behavior in your product. It'll help you answer things like, how different features are getting used, or the types of users that are using your product. So, let's say that I'm running a music streaming app, and one of our key business goals is around ongoing engagement, and the way that we measure engagement is through song plays. I might start by going to a segmentation report, and looking up song plays in the last 30 days. So, this at a very high level tells me there's been 1.7 thousand song plays in the last 30 days. From there, I might have a hypothesis that the number of songs that are played are actually different based on the country that the song plays coming from. So, what I would choose to do is group by a country. Here you can see that the number of song plays in the US is actually a lot higher than the number in Mexico and Canada. So here, we can see that the largest engagement, or the most engagement that we get in our app is all coming from the United States today. From here, let's say that another one of my business goals is driving revenue, and the way that we get revenue in our business is by having people upgrade to a paid plan. So, what I might do is choose to group by plan type, and this will actually show me how many free versus premium users I have in each of these countries. At a high level you can see that we just have a lot more users in the United States, which is something that we knew from the previous report, but if you switch here to a percent view, you can actually see that the percent of premium customers is the highest in Mexico, where we have 73 percent of all users on a premium plan, versus only 53 percent in the US, and 57 percent in Canada. So, you can see here that what we're doing is we're really just segmenting the data by different properties. Here, the segments are country, the US, Mexico, and Canada, and also the planned type, premium or free. So, really a segment is just a fancy way of saying a group of people. As you can see, this report shows us that Mexico has the highest percent of premium customers of any of the countries where our app is launched. It could be that from this point, I want to invest and dive in on getting more users in Mexico and figuring out why it is that they're more likely to be premium subscribers. First of all, I'm going to filter by country code to only look at customers in Mexico. From here, I might choose to segment by additional properties. For example, I might have a hypothesis that the type of music that someone's listening to correlates with the likelihood that there'll be a premium customer. So, first, I'm going to try segmenting my favorite genre. Here, if I change to a stacked bar view, you'll see that premium customers are actually much more likely to listen to classical music than any other music type. From here, it might be beneficial for me to actually invest and first of all, getting more users in Mexico and second of all, encouraging users across the board to listen to more classical music. Switching to another example entirely, another powerful type of segmentation report you can run is actually comparing multiple events. So, for example, if I wanted to compare the number of upgrades versus the number of downgrades in my app in the last 24 weeks, I can do that using a segmentation report. Here you can see that 97 people have upgraded in the last 24 weeks, and only 27 percent have downgraded. Overall, that means more people are upgrading in my app than downgrading, which is good news. I can also switch to a line view to see how the number of upgrades and downgrades is trending over time. You can see that it looks like planned upgrades are actually trending up, which is great. Unfortunately, downgrades are also trending up to some degree as well, though it looks like not quite as fast. So, from here I'm actually going to want to run a whole new report that really is just focused on downgrades. You can see show total planned downgrade, and I might have a hypothesis that people might be more or less likely to downgrade based on the genre of music they listen to. From this graph we can see that folk music listeners are actually making up the bulk of our downgrades, a 30 percent of all downgrades have folk as their favorite genre. This could mean a couple of things, and we might have to go do a little bit of qualitative research to flush it out. For example, it could be that our folk music offering is not that great, and so a lot of people that are on the paid plan don't find value in it. So, it might mean that we want to get better content there. On the other hand, it could just be that folk music listeners are less likely to pay for music subscription apps, and it could be that it's not worth our time to invest. This is one of those cases where you really need to augment your product analytics with sometimes some real color around the user and how they're using your product, and things like usability testing and focus groups can become really valuable. So, I think it's worth taking a step back now to see what we learned from the segmentation report. Here, we can see that people that listen to folk music are actually making up the majority of our downgrades. However, segmentation analysis, and really all of product analysis, doesn't actually tell you what to do about that. However, it does identify the areas where you should put your resources in order to move the needle for your business.
7. Report 2: Retention: So, retention helps you understand how sticky your app or website is for your users. There's different kinds of retentions but retention reports will let you understand for example, how many new users are coming back to your app or how addictive certain features on your app are. Retention is different from segmentation and retention really focuses on people completing events over time. The way that we do that is with a cohort analysis, where users are grouped into cohorts based on when they complete an event and your retention report will be structured based on when a group of users first completed an event. The opposite of retention is churn, which is the number of users that stop using your app or website. If over time, you're not keeping your users and you're not keeping them engaged and keeping them happy, they're going to stop using your product which ultimately will not help you achieve your business goals. So, first let's look at a first-time retention report. A first time retention report is useful because it really helps you understand how many of your users are coming back to do key events or really any event in your app or website after doing one key sign of event. So, usually you're going to start by identifying something like a sign up or log in that indicates a new user. Here we are using sign up. Then you can come back and identify what the event is that you care about users doing next. So, for us, it really matters for our music app that people come back and play at least one song. So, for our first time retention, we're going to show people that did sign up and then came back into the song play. Looking at the report down here, what this shows us is how people that signed up in each of these weeks in June, how many of them came back to play a song within one week, one week later, two weeks later, and so on. So, for example, of the users that completed the sign up event the week of May 29th, 60% came back and listened to a song within one week or one week later. You can also switch to view of this that has an absolute number, so here you can see that 12 people came back to play a song one week later of the 20 that signed up in May. So, from here you might care about overall trends, like on average how much are people coming back to play songs after signing up? If your percentages are lower than you would like then it might be worth investing for example, in a better on boarding flow that encourages first time signed up to play more songs. From here though, you can also segment by different features. So, for example, if you want to understand if a genre has an impact on how likely someone is to come back and use your product, you can run a retention report by genre. That basically shows you that people that are classical music listeners have the highest first week retention of any genre, whereas people that are folk music listeners have the lowest retention of anyone. This actually is in line with what we learned in our segmentation report and this could be contributing to the fact that folk music listeners have the highest downgrade rates of any of our users. So, you can see that as you run more of these reports, you'll start to see patterns and trends, that either allow you to get more detail on how users are using your product or also might just help reinforce things that you're learning. Next we're going to look at recurring retention. Recurring retention is a type of retention that lets you see how likely someone that completes one event is likely to come back and do that same event again. So, for example, I might want to understand how people that purchased a song, how likely they are to come back and purchase another song later? So, for example, in the July 10th cohort of 43 people, you can see that 40% of them came back and purchased an additional song within one week of their first purchase. Here's where you can really clearly see what a cohort of users is. So, looking at August 7th for example, this is a cohort of 100 users that in this case did a song purchase, that week of August 7th. We can now track that cohort of users over time to see the other events that they did. So, what this report shows us is that 45 people in that cohort of 100 purchased another song within one week, 41 of them purchased a song within one week or more and 38 purchased a song two weeks or more. So, part of the reason that cohort analysis is powerful is because it allows you to understand how user behavior is changing over time. So, for example, with song purchases, you would hopefully see that a cohort of users that joined this month has a higher percent of people coming back to purchase songs than a cohort of users that completed that action a year ago. Addictive attention lets you know how frequently people are doing events in your app or website within any given time period. So, for example, let's look at the addictiveness of song player. So, from here, you can see how a cohort of users that played a song in June of 2017, how many days they actually played a song in your app or website? So, for example, of the 142 people that played a song in June, 49 of them played a song on four different days in that month. Overall, this report lets us know things like the most number of days that anyone's playing song on our app right now is 11, which isn't that great because we really want to be a product that someone uses every day. So from here, this might let me know as a product manager, as a product team that I should invest in experiences that bring people back to the app more days in a month. So, I actually use retention reports a lot. At Mix Panels something that we really care about is making sure that our users are coming back to Mix Panel and using it day after day to really inform the decisions that they're making. So, for example, I'll run retention reports looking at how different reports and Mix Panel are used by our customers and how different reports correlate to overall user retention. So, if I learn from using analysis that there are certain reports that make it more likely that someone's going to come back over time, I can deep dive to understand why those reports are so useful for customers and why it brings them back to Mix Panel. I can also invest in a report that have worse retention to understand why they aren't as useful to customers and how we can make improvements that will make them come back to Mix Panel.
8. Report 3: Funnels: So, the last report we're going to talk about in this class is funnels. Funnels allow you to analyze a process or user flow in your app or website, to understand where people are dropping off from key conversion steps. Part of the reason that I really love funnels is because it really lets you dive into the patterns of behavior that you expect to see in your app or website. If you think all the way back to the implementations pack or tracking plan that you created upfront, in that step, you defined what the processes and flows are that you care about. Funnels really bring that to life, where you can actually see each of those steps and understand how people are progressing through those steps that you want them to. It could be that your funnel analysis shows you that people are using your app or website exactly as you want them to and are completing these flows really effectively, but funnels are great at calling the areas where maybe that's not happening as well as it could. So, diving into a real funnel example, I'm going to create a new funnel here. This funnel is going to be something that looks at the initial sign up all the way through to an upgrade to a paid plan. So, the first step in the funnel is going to be actually viewing the sign up page. From there, people have to actually sign up. We're going to add a third step, that is a song play, and a fourth step, that is a plan upgrade. So, in my implementations pack, these are probably some pretty key steps in the process that I wanted to see all users go through to ultimately get to a planned upgrade. I'm going to save this funnel. As it builds, you can see how many people made it through each step here. So here, we have 316 people that viewed the sign up page. Of those, 82 percent actually signed up, which is a pretty good conversion rate. Of those 259 that signed up, 211 played at least one song in this period, that's another 81 or 82 percent conversion rate. However, only 15 percent of people that played a song upgraded to a plan. I think a 15 percent upgrade rate actually sounds pretty good, but it could be that that's not on track with what our business goals are, so that might be something that we want to improve going forward. The other thing that you can do is you can actually look into any one of these steps to understand if your performance is getting better or worse over time. So here, for example, you can see over the months how your Song Play to Plan Upgrade conversion rate is going. So, you can actually see that there were no conversions prior to April. It could be that upgrading to a paid plan is actually a feature that only launched in April. You saw on an initial uptick, and then moving into June and July, you can see a small positive trend in the number of upgrade conversions. So, that maybe a good sign for your business. Similar to what we saw on segmentation, within the funnel report, you can also dive a little bit deeper to see how things like, for example, the country that you're in, impact your likelihood to convert. If you segment by a property, here, we can see that different countries have different completion rates for each step of the funnel. So, for example, in Mexico, we actually have a worse page view to sign up rate. It's only 67 percent versus high 70s and low 80s overall. So, what that means is that people that visit our sign up page in Mexico are less likely to sign up. That might be some more that I want to actually invest in creating a higher quality sign up page or sign up experience there, especially since from previous reports, we know that Mexican users are more likely to have an upgrade to a paid plan overall once they have signed up. So, when you're starting with the funnel analysis, it can be useful sometimes to start with pretty broad steps. So here, we have four steps that really describe the end-to-end experience through the entire process. However, once you have this kind of initial view of your funnel, you may want to dig deeper. So, in this case, the song play to plan upgrade is a pretty big drop off. So, I might actually want to create a new funnel that dive into what's happening between those steps to cause those people to drop off. In that case, I might create a new funnel that starts with the song play, then looks at song purchase and then finally, looks at a plan upgrade. We'll call this funnel Song Play to Upgrade. Creating this funnel, we can see that there's actually a 55 percent conversion from the people that play a song to purchasing a song, which is really high, but only a 13 percent conversion from song purchase to plan upgrade. So, that means that we're actually seeing quite a few people purchasing songs as long as they play a song, but our conversion to the upgrade is not great. So, it could be that we focus on these people that are purchasing songs and figure out a better experience to get them to upgrade their plans. So, looking at a funnel report or any report for that matter, it can be difficult to know whether your numbers are good or bad, and honestly, there's no right answer here. What's considered a good conversion or a bad conversion at any point in the funnel or in any report is different for different products and different businesses. For example, Amazon might only have a one or two percent conversion to purchase rate, but considering how big their volume is, that might be an amazing number for them. When you're looking at your numbers, there's two things that you want to think about. The first is, how is that number rolling up to your broader business school? So, if you have certain targets in terms of the number of users you're acquiring over time or the amount of revenue that you need to bring in in a quarter, what you want to do is take the numbers that you see in your report and actually roll that up to see if that trend continues if you're going to make those numbers. The second thing to keep in mind is how you're trending over time. So, for example, here, where we're looking at our plan upgrades, if over time, the percent of users that are converting to an upgrade is going up, that's great news for us and our business irrespective of what the absolute number is. If it turns out that fewer people are upgrading over time, that's a cause for concern and something we'd want to investigate further.
9. Conclusion: So, at this point you have a good overview of the basics of product analytics, and you're probably starting to get a sense of how powerful these different tools can be. What happens next is, you have all this data, you have its insight, and now you're going to want to go and take action on it. That could be making changes to your product, so improving key user paths or flows, or maybe investing in certain geographies or certain types of users that are using your app or website. Beyond that, there's also other ways you can take action, through things like AB testing, different experiences in your product, or sending notifications at key points in a user journey to drive the behavior that you're looking for. I'm also going to provide more resources in case there was a topic or area that you thought was particularly interesting. As you go through the process of creating your implementation spec, I'd also love it if you posted it in the project gallery, so that I can see some of the work you guys are doing. So, thank you so much for taking this class, I really hope it's been helpful and that this will be a starting point for you to make data-driven product decisions going forward.
10. What's Next?:
Veronica Pinchin, Product Manager at Mixpanel
