Choosing the Right Performance Metrics for Your Business

1. 1. Introduction: Hi, I'm set and welcome to my class. Choosing the right performance metrics for your business. All businesses and all organisations effect are swimming in data these days from data on which Web pages on their site customers visit most to more traditional general data like what profits or revenues were last quarter. But all this data makes it harder not easier for businesses and organizations to focus on those few key pieces of data that are really important those pieces of data that tell you how you doing or whether you need to make a fundamental strategic change in direction. I've created this class help anybody who uses data to make decisions as part of their job, from the CEO of a company down a business analyst, a company or for investors looking to invest in different companies that want to know which metrics they should pay most close attention. How are we gonna do it? We're gonna walk through the example of a fictional e commerce company that sells clothes on. We're gonna look at this company at every stage of its group from it being a start up all the way up to it being a mature company. And at each stage, we're gonna look at what the key metrics for that company of focus on that stage are y Then for your class project, you're going to apply what you've learned. Looking at a fictional ed tech company that's looking to expand into a new market, you're gonna learn about its business goals, what stage of growth it is at. And then you're gonna make recommendations to its leadership team about what metrics it should focus on in order to achieve those business goals. I'm really excited, and I hope you are too, so let's get started. 2. 2. About the Instructor: So who am I in this section? I'll introduce myself at a high level. I have five years of experience in data science and Data analytics at a variety of companies, and I've been working in data science specifically for the last two years, and I'm currently working at Metris. Before moving on, I'll talk a little bit about menace. Medicine is a premier data science education company, and we're dedicated to getting mortgage rate data scientists and great data science out there in the world. We do data science training for many companies, including several in the Fortune 500. And we also teach full time data science boot camps in New York, San Francisco, Chicago and Seattle. And we do part time evening courses in sub fields of data science like machine learning and data visualization. As a senior data scientist at Metris, I'm currently teaching full time who camps in Chicago, and I'll be doing so through the end of September 2017. And after that, I'll be doing corporate training for them. So now getting back to that first bullet consulting business analysis and data science that probably just sounds like a bunch of buzzwords to you. So let me break that down. I started off working for a quantitative consulting firm in Chicago called the Cambridge Group. We would do projects like surveying companies, customers looking for quote unquote clusters of demand based on customers answers toe survey question that we gave them. And then we make recommendations to our clients about which products they should launch or how to market existing products. After working there for a couple of years, I moved to Capital one. Now Capital One was launching a bunch of new digital products when I was there, and I was responsible for figuring out which metrics we should track and building reports tracking those metrics. Finally, I worked in data science at Trump Club, the e commerce clothing company based out of Chicago, helping them better serve and understand their customers by building lead scoring models, improving their recommend er systems and building something where I automated the creation of quote unquote outfits of clothes for new customers. And I did all this using Python. Finally, a huge part of my career over the last several years has been being involved with innovative technology enabled education companies. I've taken a ton of classes on udacity and coursera, which is how I was able to transition from the non coding rolls I had at the Cambridge Group in Capital One to my python centric role at Trump Club. I worked part time for Udacity for eight months. I taught the General Assembly part Time Data Science Course in Chicago in the fall of 2016 and now I've really enjoyed the process of putting together this class for skill share. All right, with all those preliminaries out of the way, let's get started. 3. 3. The Startup Stage: Welcome back in this section, we're gonna talk about starting the company and for a very early stage company. Which metrics matter? Let's say that you've launched a new e commerce company that sells clothes online. Your idea is that people are always looking for new clothing brands. But the information about clothes out there is fragmented their reviews on the websites of the clothing brands themselves. Their reviews on Amazon, There are things you hear from your friends. There's an overwhelming amount of information. Your idea is to have reviews posted by your Facebook friends and friends of your Facebook friends and on other social media channels, so you'll be able to see who bought what and who liked it. Your company will make money by taking a cut out of any sales that happen on this site, as well as having brands pay you directly to get their clothes promoted on site. You call the company my Clothing Reviews Inc and call the website my clothing reviews dot com. So what should you do next? Your first step will be getting some money either out of your own pocket from a bank from some private investor venture capital or otherwise. Then you'll have to get both sides of your marketplace started. You'll first have to get the seller side of your marketplace started, so you'll have to buy some clothes from some clothing manufacturers that other people will actually want to buy. And you may do this by hiring a sales force of some kind and having them call on brands and try to convince them to sell in your sight simultaneously. You'll have to convince people to sign up for your site, and this will almost certainly be is through some sort of marketing, whether traditional PR, such as magazine articles about your site or through digital marketing on Facebook, for example. So right away you have to be spending money to grow your marketplace. So what are the most important metrics for you to focus on this stage? Whether again you're the CEO of this company, you're an employee of this company or you're an investor deciding whether to invest in this company? Well, the most important metric to focus on during this part of your company's journey is your burn re. Your burn rate is simply the rate at which you are spending money. Let's say you started with $1 million after three months you checked your bank account, saw that you had 700,000 left after, for example, paying the rent for office space and paying employees for three months. That means that your burn rate is $300,000 over three months or $100,000 a month. And, of course, this metric tells you how long your company can survive for now. Why is this the most important metric at this stage? Well, the reason is, at this point, your priority is a business as just to survive. You want to find what is called product market fit, which means that you sufficiently understand your market, which in this case is both customers that want to buy clothes and businesses that want to sell clothes well enough so that you know how to tailor your whole product experience to serve this market. So finding product market fit is the number one goal of your business at this stage, and burn rate tells you how long you have left to find product market fit before you run out of business. So that's what metric is most important now. How do you know if you found product market fit? The answer is that you start seeing lots of new customers and new businesses wanting to sign up for your site organically without you having to do anything. Once you hit this stage of your businesses journey. Ah, whole another set of metrics start mattering and those are what we're going to talk about next. 4. 4. The Fast Growth Stage : Welcome back. Now let's talk about the metrics that matter for companies that are growing really quickly . Let's say that my clothing Reviews Inc has found product market fit and is now experiencing significant growth. Lots of new users air signing up for your site and buying things on your site. And you're getting clothing brands to increasingly agree to sell on your site as well. Now, whether you're the CEO of this company, a business analyst working for this company or an investor evaluating this company, which metric should you be focusing on? And for the remainder of this lesson, I'll suppose that you're the CEO, so I'll refer to my clothing reviews as your company toe. Understand which metrics you should be focusing on. It's important to understand that your company's value will increase exponentially with its size due to its network effects. So just by the company being larger, it becomes more valuable to both the customers and sellers that are using it. Why is that? Well, the more people there are that have reviewed close on your site, the more value your site provides to visitors. Since the value of your site is in the fact that people your visitors know have reviewed close so they can see if they like them. And the more people there are in your site, the more value your site will provide to sellers, since there will be a larger market for their clothes. So the overall value your site is able to provide both to its users and to the brands that sell on the site increases exponentially with the number of brands and users on your site. So at this point, your number one priority as a business should be generating that exponential growth. This is common for many businesses with similar business models, but let's dive deeper, which metric specifically should be growing exponentially. The answer is that you want to aim for exponential growth in every metric, for which the amount of value provided on your site grows exponentially with that metric and a catch all term for these kinds of measures is engagement metrics because they measure quote unquote engagement with your site. Let's look at some examples of good engagement measures. One common engagement metric that you should know is G M V or gross merchandise value. This represents the total dollar volume of clothing sold on your site. This is a measure that obviously corresponds to the amount of value your users are getting out of your sight and thus should be growing exponentially. But any metric that measures engagement with your site and corresponds to the amount of value that users air getting out of your sight should also be tracked. So another good example in this case would be the number of reviews left on your site, since users leaving reviews is something that drives the value of your site and is therefore metric that at this stage you would want to see growing exponentially. So would any metric related to activity on your site be a good metric for your site? Actually, there are lots of metrics that companies track and report on that are not directly related to engagement and are not so good as measures of engagement. The's air often considered vanity metrics. They can make you look good and feel good, but they don't really mean that much or add much substance. Ah, classic example is the number of registered users on your site. You can spend a bunch on advertising and grow your number of registered users. But if those users aren't doing anything on your site, it doesn't matter. Another common vanity metric is the number of people who have downloaded your app with a number of APP downloads. And the most important and over our king point here is, while someone downloading your app or registering on your site is great. That in and of itself does not generate or fuel your network effects. Someone signing up by themselves doesn't make the site a more valuable place for other people to visit. But someone leaving a review, at least in the case of this business model, does. What about classic business metrics such as revenue and profit? These are critical to assessing the health of any business and in businesses that require fewer network effects to get started. It can even be important to be profitable out of the gate, but they are often of secondary importance at this stage. Let's go over why The short answer for revenue is that while it's important at this stage, G M V is more important again because your business has network effects. Getting that network as large as possible should be the number one priority of your business at this stage. Now let's suppose that you decided to focus on revenue at this stage of your company's growth instead of G M V. Well, if you were focused on revenue for a given amount of stuff sold on your site, you could increase revenues just by increasing the percentage of each sale that you take as a fee. However, this would reduce the incentive for sellers to continue to sell on your site and with us stop. You're increasing network effects in their tracks and hurt your company's long term growth . So this is a great example of a huge lesson, which is just by focusing on the wrong metrics at the wrong time. Your company can make fundamental strategic errors Now. What about profit? Companies can be unprofitable for to a point, of course, for a long time as they grow. And that's especially true for marketplace businesses like the one we're describing but is true of many businesses. For example, Amazon was unprofitable every quarter from its founding in 1994 through Q four of 2001 even as it defined and became the gold standard in the e commerce space and Jeff Bezos was named Time Person of the Year in 1999. Of course, growth starts to slow. At some point, all businesses mature past this stage of exponential growth. When it inevitably does, the rial detailed business analysis must start, and that's what we're gonna talk about next. 5. 5. Diagnosing Slowing Growth: welcome back. So let's say that your past, the honeymoon phase, your number of users, your number of sellers isn't growing exponentially anymore. You actually have to do the hard work of figuring out how to grow this business sustainably , so growth is slowing. Gross merchandise value is no longer increasing exponentially. So what metric should you focus on to ensure that this company can continue to grow sustainably? Or if you're an investor, what should you focus on if you're evaluating this company and figuring out whether it can continue to grow? Well, one thing you should look at is how you're getting users. And broadly speaking, there's two ways of getting users. You can get them organically or in organically, and to put it in simple terms, organic user acquisition refers to any user acquisition channel where you don't have to do anything so word of mouth people tell their friends about your site and they navigate to it directly. They go directly to good looks dot com and start shopping, or when people google good looks directly and click on the site when it appears and search results. That's organic growth to inorganic user acquisition refers toe anything else. Anything you're paying for is considered inorganic. So any advertisements you do on Facebook, any advertisements you do in print? Even referral bonuses are inorganic acquisition, since those acquisitions might not have happened if you hadn't paid. So one big thing to look at is how you're getting users. One big metric to track that often changes for companies at this stage is the percent of users that you're acquired in organically versus organically. It's very often that at the beginning of a company's growth, most of their users air coming through organically. They're coming through through word of meth. They're coming through by Googling for the company directly because they hear about it in some way. And these were sort of the early adopters people that just love your idea, and it solves a really important problem that they have, and so on, and at some point that stops and people start joining in organically. They start joining due to all the marketing you're doing and when these users start joining more in organically, they have what's referred to as a cost per acquisition. So you pay some amount of money in marketing and some amount of user. Some amount of people actually join your sight as a result of that marketing and the overall if you just take all the money you spend on marketing and divide it by the number of new users you get that gets you your cost per acquisition. C p A. Is a common metric for companies to track Teoh. See how much it costs them to grow in organically by paying for new users. Now that we're talking about costs, profitability becomes important. So there's a lot of different ways to measure profitability. But roughly speaking, profitability is revenues minus costs. You can take an accounting course and learn about all the different ways of measuring profitability. But roughly its revenues minus costs. And what's really important when thinking about profitability is what are the key revenue streams of the company and what are the key costs? So let's think about what those are in the case of this company, so revenues are just the share that you take from sellers on your site. When sellers sell something on your site, you get a cut of it, and costs include the cost to acquire those sellers. For example, by having a sales force that goes out and calls different clothing brands and tries to get them to sign up or the cost you pay to acquire customers. Buy online advertising, for example. Finally, you have all your costs that are not directly related to buying or selling on your site, such as paying employees paying for rent of your office space, etcetera. So a summary of all this profitability metrics is lifetime value. So lifetime value is the amount of revenue that you generate over the lifetime of a cellar , for example, minus the cost to acquire that cellar. So in the case of a seller, this would be the amount of clothes that that seller sells on your site times, whatever proportion of those sales that you keep for your company, minus the actual cost that you paid to acquire that seller, meaning the cost of the sales people, the cost of all the marketing you did to that cellar, etcetera. And similarly, for customers, so lifetime value, to sum up is a way of seeing at a customer level or at a cellar level if you're profitable , so it sort of takes profit and breaks it down a little more granular Lee. And if customers and sellers on your site don't have positive lifetime value of certain segments of them, don't have positive lifetime value. That could be a key sign that you need to change something about your business and looking at a business from the outside Thinking about with lifetime value of the customers or the sellers for that business are is a good way to evaluate that business. Finally, let's talk about what's driving all of this. So all of these metrics, whether somebody continues to make purchases on your site, for example, whether someone signs up whether someone buys anything on your site at all. All of these are the result of people actually having a good experience on your site. And how do you track whether people are having a good experience? Well, one way, that's a very popular metric. It's a little bit unscientific, but it's very popular. Is a net promoter score and a net promoter scores calculated in the following way. You ask people either sellers or customers on your site. How likely are you on a scale of 1 to 10 to recommend this experience to a friend and from these scores, you can calculate a net promoter score. The details aren't too important. The most common way to calculate it is to take the nines and the tens and call them promoters and to take the ones through the sixes and call them detractors and to subtract the percentage of people that are promoters from detractors. And call that your net promoter score. But that's just one way of calculating net promoter score. The broader point is, NPS is one very popular way to measure our people, actually having a good experience on your site. So that leads to a question. What if people aren't having a good experience on your site? What if metrics aren't trending in the right direction? What should you do and how could you diagnose that and figure out what's going on? That's what we're gonna cover in the next section. See you there 6. 6. Funnel Metrics: welcome back for mature companies facing the issue of how to drive additional growth or optimize their user experience. A common framework used is the funnel free. So to be clear, the question here is how can you break down the experience of using your site into steps so that you can identify the biggest pain point where user experience could be improved? And again, this is an important thing to focus on, whether you're the CEO, whether you're a business analyst working for the company or whether you're an investor evaluating the company. The idea here is that customers must go through a series of steps from before they become customers to when their regular customers on your site, So the process for a customer visiting your site will be they must first visit your site. Then they have to create an account. Once they've created an account, they can make their first purchase, and you'll then want them to leave a review, since leaving reviews is a key part of the value proposition of your site. Finally, once they've left a review, you'll want them to return to your site to make another purchase, and each one of these congee thought of as steps in a funnel. It's called a funnel because it goes from wide at the top, where you have lots of visitors to your site to narrow at the bottom where you have a relatively small number of repeat purchasers, we asked at the end of the last section. How could you diagnose low net, promoter, score or load and PS on your site? And one way to diagnose that is to pay attention to the funnel metrics on your site to see where people are dropping off. So an example funnel metric would be the visitor to sign up ratio. The percent of people who visit your site who go through the whole account creation process . If this ratio or this percentage begins to drop or is just low overall, perhaps compared to similar sites, then that tells you the valuable information that the barriers to signing up once someone visit your site are too high. This lets you dig deeper and determine what is going on there. Perhaps you find that the value proposition of your site isn't clearly stated on your home page. Or perhaps there's no clear call to action leading users to sign up once they visit your site. Similarly, if customers are making purchases on your site but not leaving reviews, that tells you that you should consider making reviews mandatory or strongly encouraged before their next purchase, just like uber and lift do. Finally, let's say the sign up to purchase conversion rate is too low. So the ratio of people that create accounts on your site versus people that actually make a first purchase that percentage is too low. That tells you the valuable information that you need to prioritize that area of your product experience. Perhaps, for example, after somebody creates an account recommending products that people could buy next, based on their preferences immediately after they join your site. And it is this example that will drill down on deeply in the next section with a little exercise 7. 7. Funnel Metrics Example: Welcome back. Now let's go through an example of a company that wants to launch a new feature and how they might look at their funnel metrics when deciding whether the feature is having a positive impact or not. So let's say the current state of my clothing reviews, Inc is that people can visit your site, and when they do, they can create an account just by putting in information like their name and email address . And once they create an account, they can immediately start shopping. You have an idea of making people go through an on boarding process, a common thing that APs like Apple music do where, when you sign up for in this case, my clothing reviews Inc. You give some information about yourself, like where you shop for clothes, what brands you like, or even how you like your clothes to fit. Based on this, my clothing Reviews Inc has hired a couple of data scientists to use this information and use machine learning to recommend products to your new customers based on their answers to these questions. So the high level question here is, how will we know if this is making a positive impact or not. Just overall is adding this feature a good thing for our business, whether you're a CEO, whether you're a business analyst working for this business or whether you're an investor evaluating this business. To analyze this systematically will use the funnel framework, and we'll look at how this new feature will impact each stage of the funnel. Well, the first stage of the funnel is from people visiting your website to people being done creating their accounts. Now this stage of the funnel, we would actually expect to be negatively impacted by this feature. Why? Well, we're required more steps than before for people to create their accounts or were increasing the total amount of friction involved. So this part of the funnel will actually be hurt. On the other hand, among people that do get through the on boarding process, we would expect that more of them will make a purchase than before. Why? Well, now those people that get through the on boarding process and create accounts, we have mawr information on them than before, and that information should allow us to recommend products that they'll look at and go, Oh, I want to buy that right away. So what's the answer? What should you focus on here in general? The answer is you should focus on the impact on the entire fund. And oftentimes, when making the decision is a business, you have to combine metrics or even create new metrics to make sure you're focused on the right thing. Here. The key metric or combined metric is what percent of people that visit your site end up making the first purchase. This takes into account the fact that this feature will decrease the proportion of people who visit your site that finish creating an account. But it also takes into account that it will increase the proportion of people who create an account who end up buying something. So this is an example of the kind of reasoning you'll have to go through when making decisions in your business. You'll have to think critically about what impact you're trying to drive and focus on measuring that specific impact directly. I hope you take this example and apply this same kind of reasoning when your business tries new things 8. 8. Closing Thoughts: Welcome back. Here are some closing thoughts on metrics based on what I've seen in a number of companies that I've worked in and with first, it has continually astonished me how often companies simply do not pay attention to the right metrics, even though using the right metrics might seem like common sense. A lot of the time start up type companies often focus on vanity metrics like the number of registered users they have, or they'll talk about the number of quote unquote active users they have or active might be measured just by visiting the site, rather than by doing something on the site that actually corresponds with engagement. Ah, Classic one for tech companies is focusing on profit too late While they're growing now, it's absolutely true that for many businesses you have to get to a certain scale and get a certain amount of traction before the economic start to make sense. Nevertheless, it's still true that companies frequently focus on profit too late, enjoying the phase of fast growth for far too long before pivoting into becoming profitable . Another one for more traditional companies is actually focusing on growth too early before they have really nailed the user experience. This will manifest itself when companies begin to grow rapidly and their NPS immediately starts declined or their c p A goes through the roof. What's happening in these cases is that the company may have had traction among early adopters but hadn't really figured out an experience that people in the broader market would find appealing. Sam Altman of the major venture capital firm Y Combinator has an essay about companies that focus on growth to suit that I'll post a link to in the class, notes another closing thought. I focused very little in this class on revenue and profit, and that is because revenue and profit are the result of doing everything else well in your business. Revenue, for example, is the result of customers using your site and loving it and telling their friends about it so that their friends make purchases and your customers returning to your site making to make more purchases on the seller side, the sellers must have a good experience using your site whether that means a smooth process to get started selling on your site or good customer service once they're on your site, so revenue and profit are just the result of doing these things the actual running of your business. Well, if there is perhaps one point to take away from this course in the area of how you should evaluate businesses, it is to focus on the actual experiences and kinds of value that the business is giving to its customers, to its sellers and to all of its stakeholders. And then next, focus on metrics that a business contrive directly that are directly related to that value . That it's addict, such as gross merchandise value such as the individual funnel metrics we discussed, like the percent of visitors that actually end up creating an account or metrics of engagement directly like NPS. That's all I have for the lessons. Hopefully, you've come away with this with a better understanding of how to choose the right performance metrics for your business again, whether you're the CEO, whether you're a business analyst for this company or whether you're an investor in a company, thanks, everyone