Email Marketing Essentials: Measuring Impact with Metrics

1. Introduction: I'm John Foreman, the Chief Data Scientist at MailChimp. MailChimp is the largest email service provider with eight million users. We send 15 billion emails a month. What makes being a Chief Data Scientist usually interesting is actually that we track a bunch of information off of those 15 billion emails. My job here is to collect all of our data and look at it to answer questions, as well as to build products that use our data to better help our customers perform email marketing, and speak to their audiences. So, I wrote a book that explains a wide variety of data science techniques. The interesting thing about email is when you think about direct mail, I'm going to send you a letter. I have no way of knowing what you did with that letter. Emails are technological products actually produces a lot of data. When someone receives it, you can track opens, you can track clicks, you can track unsubscribes, and a lot of things. So, today we're just going to talk about those metrics and that data you can gather from the email you send, from the marketing you send, and what's the best way to use that to improve your business in the way you speak to customers in the future. The audience for this class is anyone who has an audience they're already speaking to, they want to speak to them with email. They care about improving that communication and improving that engagement with their audience might have. If you're interested in email metrics, the reason why you're interested is because you want to do better. You just want to batch up something and shoot it out of a cannon. You should care about how these people are engaging with their emails. 2. Your Project: So your project for this class is to go into your inbox and find an e-mail that's unread, something like a promotional e-mail that you've received, some kind of bulk email that you've received that you've just scrolled past and not engage with. Open it up. Take a look at it and what you might be able to change or play with or test to improve the performance of that email. Think about someone like you who maybe skimmed past it in their inbox and what you might have done differently to grab a hold of you. So the reason why an exercise like this is valuable, is just to emphasize the fact that an audience that you're marketing to is a lot like you in the sense that that audience is full of people. So by looking at the promotional e-mail that you received and thinking about what it takes to engage someone like you naturally says something about your own practices and engaging with your audience, right? That, as you begin to think of yourself as an audience member for other businesses, in other organizations, so think about what does it mean to have these conversations with people like me. For this particular assignment, you don't need a whole lot of gear. If you're going to screen cap the email, you might need to screen cap it with some sort of software or just take a picture of your computer monitor, I think would work too and just e-mail it to yourself. Then you're going to need to think about what you would change, right? So it's just going to take some time. If I were looking through these different submissions and I think about what would make a good submission for this particular project, it would be one that was real thoughtful around the various elements that comprise an e-mail newsletter. When it was sent, what was the subject line was, what the from name and from address was, what was in the content, and actually, think constructively about what this business could do, really put yourself in the shoes of that business. So I'd encourage you not just to submit your own project, but actually to go into the gallery and the discussion sections and reach out to other people who are participating this class and strike up discussions, challenge what other people are saying about what they might change versus what you've seen in ways you've done marketing or you might have other ideas on a piece of content because you get to learn a lot more in conversation with others than you will just thinking about these things on your own. 3. Overview: Metrics Matter: When we say the word Metric or email Metric, what we're talking about is that when you send email marketing or any kind of bulk email, this email generates data. In this class when I refer to email metrics, what I'm referring to is just any number of account or a rate that is generated from sending your email. Many studies have found that businesses generate a fair bit of their revenue, the email marketing. It's a great way to make money, it's a great way to have a personal conversation with a customer. So, if you care about making someone a customer or a previous customer buying something new, understanding how they're interacting with your email now is going to give you a sense of what you can improve to get them to come back. So, I think that's why an entrepreneur would care about it is because email is a great channel. Actually is much more effective than say social at making money. So, if that's your goal to whether it's to sell merchandise or services or whatever, this is just a really good channel to do that and so it behooves you to improve your performance by actually looking at how you're doing now, which means looking at the data looking at the metrics. There are two ways actually that I think email metrics have really evolved over the past few years. One is that via integrations between email service providers and other web analytics products, you actually can now get a very good picture of not only how people are engaging with your email, but once they've engaged with your email and left that email to go to your website, what are they doing down stream? It used to be just like, "Oh, here's who opened, here's who clicked." Now you can actually see, here's who clicked, left, went to my website, opened this particular product, added it to their shopping cart, but didn't buy it. That's actually more informative and really more interesting than just who clicked. Another thing that has substantially changed is the rise of mobile. So, if I'm writing Bob Loblaw's Law Blog and everyone who reads it are lawyers and they're reading it on desktops, that maybe I can engage with people in the same way that I used to. But if I'm in e-commerce company and most of my customers are young, intend to read on iPhone or maybe Android, then you need to think about how does the email show up on that mobile device, are the images and print clear there? Am I putting the content right in front of them as opposed to making them scroll down which they might choose not to do on the phone? Things like that. So, that's another thing that's substantially shifted from even just a few years ago. So, this course is going to be covering a lot of different metrics, different data that your email could possibly generate. Not all of it is going to apply directly to your business necessarily, if you're an e-commerce company, e-retailer, some of the things that I say regarding tracking purchases for instance would apply, but might not if you're a nonprofit and what you really care about is perhaps a donation page on your site. So, just take that with a grain of salt, take away which you can, and metrics that might not apply to you, just don't worry about them. Most email service providers allow their customers to track who's opening their email. This is done generally by embedding a single pixel image in the email, where it's encoded to be specific to each recipient, and so, the email service provider can tell like, "Oh, this person just open because they downloaded this tiny little image in the email when they opened it." That's the first metric you become acquainted with in email marketing. If you have URLs in your email marketing, so hey, click this to go buy this thing, then you can actually track clicks. So, who clicked what? Not only did this person open, they went in, they looked at this T-Shirt, they click to go to my website to buy it. So, you can track clicks in the same way and you can create a click rate by knowing how many people you sent to and out of that what percentage actually clicks something. They're also some negative metrics you're going to want to track that are part of this group of traditional baseline email marketing metrics. Specifically, their unsubscribes, so whenever you send email marketing, you need to make sure that folks on your list, parts of your audience can say, "Hey, I really don't want to converse with you in this manner anymore, I want to opt out." You need to track those. Those are good just for maintaining less quality, but also you want to look at unsubscribes in terms of just the individual unsubscribe, who was it? Or you could look at the unsub rate which would be, "Oh, I actually got five unsubscribes out of 100 people, that's like an unsub rate of five percent." Maybe that's really bad, maybe that says that rate actually says I'm doing something wrong. Some other ones would be the bounce rate, which would be, how many these email addresses I'm sending to or actually dead addresses and no one uses the email account anymore. Then there's the abuse rate which is even perhaps more negative than an unsubscribe, these are people who are saying what you're sending is spam. i.e. you do not have permission to speak to them and it's really important that you pay attention to that. One thing that these metrics will do is actually tell you about how stale your list is. If you're not talking to your customers on a regular basis, you should fully expect them to check out. So in terms of your positive metrics things like open rate, click rate, you'll see those go down, and then in terms of negative metrics like bounce rate, abuse rate, if you're not constantly sending to your list maybe, "Oh, every month I'm going to check in with my customers, tell them what's new in the shop that's going on." If you don't do that, those negative metrics are going to climb. One thing to think about when it comes to the performance of your list is its' growth. So, one way to look at list growth is to track both subscribes to your list and unsubscribes. Those two numbers combined, of course, give you your list growth. You want to look at how many people are coming onto your lists and maybe that's new customers, new readers, new donors depending on your business and then how many people are deciding to leave. So, one thing I will say about negative metrics on your list. So, we can talk about unsubscribes for example. There's going to be someone who comes into your list because they bought a product from you, because that's where they were at at that particular time in their life and then they might move on and they might unsubscribe from your list. These baseline metrics provide a lot more data than just looking at a total count of let's say clicks, or a click rate will actually convey. So, let's take clicks for example. A click map or a hierarchy of clicks is something you'll often see in email service provider, it's a more complex metric to look at than just the click rate and is one that's actually really interesting. You also get things around the person's device and location when they click. So, you can actually see things like, "Hey, you know what? You're performing really well with people who use iPhones, but not so well with people who use desktop." Maybe that's interesting, maybe that says something about the way you're designing your email, you can actually see that in your email marketing metrics because each individual click might actually come with a location. Oftentimes, email service providers MailChimp included, will be able to show you a map of here's the US and here's where your engagement is coming from within the US, and that can be very informative. This can help inform the way that you speak to your audience in a few different ways. If I know the clients that folks are using, I can test my email on those clients. Oftentimes, email service providers will give you some way to preview the way your email looks on all those clients. Another way to use that data is just to give you a picture into the age and stage or the demographics of your audience. If your clients are predominantly Outlook in desktop, what does that say about your audience? If your audience is primarily engaging with your email on an iPhone, maybe that says something not only about how you should design your email, but what these people are in the times they choose to engage with your content, then you can think about, "Oh, what's the best way to speak to them?" They're reading this while they're on the go, maybe it's going to be harder for them to do a checkout process if they're clicking the email from their mobile device and you're thinking about how I can enable them to buy stuff on their mobile device. So, one thing you can also look at it's kind of similar to client is what ISPs you can actually, a lot of email service providers will allow you to break down your email engagement by ISP. Gmail happens to be free, Comcast, you got to buy something from Comcast, you have this so maybe your email's working better with people who have a little bit of disposable income or a job that could inform the way you speak to your audience or who you focus on "Oh, these people like it more, I'll speak to them in this way." Maybe this isn't resounding with a Gmail audience, I'm going to change the way I speak to them. So, some email marketing products will actually allow you to compare your performance between one email and another. It can be informative to just see, "Oh, this is how many people engage with this email." It can actually be more informative to look at your performance over maybe the last quarter and maybe the last year and see whether engagement is going up or is going down or maybe it's going up with one segment and down with another. If I build segments around gender, I can actually see perhaps that the male segment is disengaging recently. What does that mean about the products I'm advertising? Maybe I need to segment them out and send them something else. So, that comparative style reporting where you're still looking at the same metrics. You're still looking at things like click rate or people who purchase something, but you're going to look at it over time and across emails, that can actually be really interesting. So, that's something to consider too. 4. Types of Email: When it comes to all of these email metrics, it's good to focus on the ones that directly apply to the goals of your email marketing. There are few different types of email, for instance, maybe your email is the Buy Me type of email, where you're trying to get people to buy something. In that case, sure you can look at opens, but that's perhaps not the best metric. Click rate, i.e someone clicking from the email to go to your website would be closer to that purchase. So, that would be a better metric. If you happen to have some type of e-commerce technology that could integrate with email service provider, you can even go further into, hey, they should went to the private page and add it to their cart, or maybe they bought it, and you could push that data back versus, let's say, you're a business that just want people to go to a particular event. Maybe it's a tutorial or a networking function and it's a Join Me type of email, right? "Hey, I'm putting on this event I'd like you to join me here." In that particular case, maybe there is nothing to click in the email. Maybe you're just giving details around the event. Then, open rate may actually be a great metric, because that's how far people need to get, right? They need to look at the subject line, see you, it's from, and actually open it and read it to get the event details. Although perhaps, you have an RSVP button in there, people need to click on that button, fill out a form to say, "Oh I'm going." In that case, you actually care about the click rate of that particular RSVP button. So, depending on what type of Join Me email it is, how formal it is, how much you want to track, it could be open rate, it could be click rate, just depending. You could be a publisher, or a blogger, or someone who actually just wants to directly converse with people in your email. I'm actually going to put an article in this email. Then, if I just care about people reading the content of my email and that's it, open rate is the best way to track that. Then, there's a fourth type of email that would just be generally categorized as transactional. This could be a lot of different types actually, right? So, transactional could be a receipt or it could be tracking information around the shipment of a product, or it could be maybe it's staged out individually per recipient, but, oh, every week, you're going to get a new lesson in a course I'm teaching. That type of transactional. When it comes to automated workflows, you might actually care about how long that person is going to go with this workflow. Oh, yeah, a lot of people really engaged with lesson one, and they really engage with lesson two, but then they fall off with lesson three. Where did I miss the mark? What is it about my content between that second and third lesson that people aren't following on in this particular course? Automation workflows should be very valuable to see where people are just falling out. Same with unsub rate, or if you're trying to get people to click on various things within the email, in the automation workflow, you can even look at click rate. So, depending on the goals of my email, let's take Buy Me versus Read Me. You're going to look at different metrics and because of the metrics you are looking at and the goals you're trying to achieve, there's only certain things you can change. If my goal is to get people to read a particular article, that's in the body content of the email, then naturally, click rate doesn't apply. And because what I'm tracking is open rate and I want to know who's opening to read this, there are only certain things I can change to affect that rate. Since it's open rate, I can affect the From Address or the From Name, so, rather than have my From Address be like, "Don't reply at Johnforeman.com." Maybe it's "John@Johnforeman.com." And that might be more welcoming. So, if I'm trying to get people to read than focusing on open, so I need to think about what I can change before people open to drive them to open. If I'm thinking about clicks, or purchases, or something beyond the email, where someone is going to engage in the email by clicking URLs and going to my website to do something, then all of a sudden content becomes fair game. Now, I'm thinking about what can I change before someone clicks to persuade them to click. Things like images in the email, layout, the order of the URL, such that the most important stuff perhaps comes first. All of those things become elements in the content that I can play with because I care about clicks which are sort of downstream from the layout of the email itself. 5. Metrics in Perspective: So, a lot of people when they approach email for the first time they think,"Oh, all these customers, they all want my email. They're all going to open every email and they're all going to click every link I have in that email." You might have sort of wildly inflated expectations around how people are going to engage with their content. It's not necessarily that what you're sending is something that people are not interested in. So you need to have realistic expectations around how many folks on any given particular email are actually going to engage with it. But then the question comes in, what is a realistic expectation for the performance of my email? How often people engage with a nonprofit email might actually differ from a hobbyist email, which may actually differ from a publisher or an e-commerce company. So based on your particular industry type, you should expect engagement to change. You should be aware of that, so you can actually get industry benchmarks and so you can compare yourself to that benchmark. So, putting those metrics in context can actually be very helpful. So, one thing I touched on earlier is that for any given subscriber, they may be interested in what you're doing as an organization or a business only for a time and then they may move on. The rate at which that happens actually changes based on the industry you're in, and that's something you should be aware of. So for instance, if you're a publisher and you publish foreign policy news and then when someone signs up for your service, they may actually be engaged with you for a very long time. There aren't that many people that say, "I really want in-depth foreign policy coverage," and so if you're a business like that, you may see very level metrics from those subscribers for a very long time. However, if you are something more like let's say a gym, you may see engagement with your content evolve over time. Someone may be actually very engaged with your content at first and then that that engagement may degrade over time, and that's something that's natural. Right? They may have a New Year's resolution that they just completely abandoned by April. So, you'll see that to a lot of businesses will see that. We will see an ebb and flow or a petering out after so much time. It's important to realize that for a lot of businesses, you have a limited number of times to send to someone where they're still interested in what you're doing. You want to capitalize on that time when they're interested before they move on and think about getting new folks, new subscribers, new customers in the door that will be those same people down the road. They're now engaged for a time before they move on. 6. Testing: So, one interesting thing about email marketing metrics is that they can actually be used to perform tests or experiments on your marketing. It's informative to look at this, each week I send out a newsletter here's what the click rate is. But you might ask yourself, what if I had done the email in this way instead? Just changing it up the next week and sending out an email that looks some other way or has a different subject line that only gets you so far, because maybe in that week your list has changed. So, what you can do instead is actually run a test. These will often be called AB tests, or multivariate tests, or univariate tests. The idea here is that I've got some metric I can track. Maybe it's open rate, maybe it's click rate, and I can take my list, my audience, my subscribers, and I can divide them up randomly. Then, I can take two different versions or maybe it's eight different versions of my newsletter and send them to these random groups of people and see how engagement changes. Now, there are lots of different things that can be tested. You could test subject lines, you could test from name and from address, you could test send time, or you can even get into the email and test content itself. Based on what you're testing, you want to focus in on a metric closely related to that change. If I care about people reading an article that I'm putting in the content of the email, then I want to look at perhaps open rate because I'm not caring about people clicking away from the email, I care about people reading that email. So, I want to get people into the email. So, I want to play with things like subject line, or send time and in that case, open rate is a perfectly acceptable metric to look at. So, you want to choose the metric that best shines light on what you're testing. So, there are multiple reasons why you might want to test. One reason to test is just to learn about your audience. So, think about performing a multivariate test or maybe I have got two subject lines, I've got two different ways I can present the content. So, maybe it's picture a, picture b. One of the reasons why you might want to do a test like that is just to learn. Wow, you know what? When I send at 1:00 PM, people seem to engage better with this particular thing than 9:00 AM and they really like it when I do images that look like this. Maybe it is my shirts on a model versus my shirts hanging on the rack. So, you can learn about all that stuff and then going forward, just engage with people in that way that seems to work best. Your list is constantly evolving. New people are subscribing, other people are unsubscribing, the world around you is changing, people are getting new devices, maybe they used to all use BlackBerry now they all use iPhone. Because your audience is changing, their situation in life is changing, their interests are changing, you can't just test and then never test again and assume that what you learned on that test still holds true. It's important to just always be testing. Maybe it's once a quarter you run some AB tests, or some multivariate tests to think about. You know what, I used to send in the morning, now it seems best if I actually send in the evening and I structure my content like this. That way you've got something to operate on that's more current, and then you would engage with people in that way until it came time to maybe test again. Making it a consistent practice is just going to improve your performance more. One thing to keep in mind is that, it's best to actually think about ideas that may actually affect how your audience is going to engage with you for the long term. So, hey, I could do a subject line that really says what's going on in the content, or I could just call out this one particular element really loudly. Maybe that would be a good test versus just gimmicky. Putting the sushi emoji first in your subject line, in one subject line and not the other. Sure, that's a change you can do, but whatever you learn there is not necessarily going to be sustainable. If you put the sushi emoji in every subject line from now until you die, you're not going to maintain that same additional couple of opens because people were confused about the sushi emoji. So, actually think about testing things that you think actually matter for having an effective conversation with your audience. When it comes to testing content, one thing to keep in mind, is you have at your disposal this whole newsletter. Maybe it's a bunch of different links, a bunch of different images. If you change everything around, if you change too many things, you might not actually be able to understand what caused people to engage more or engage less. So, you want to be conservative and focused in what you change. In that way, when you get back the results of the test you can say, hey, more people clicked and this is the thing I changed, and so that's what I've learned. If I change the layout of this email how do I feel about that. You can come up with hypotheses, and the more you think about that and the more you think about those reasons, the more it's going to help you conduct a test that actually matters versus just randomly throwing things against the wall as if your audiences is just mindless robots. I think it's important to realize that your own experience is a great way to generate ideas. Those are interesting things for creating maybe a list of ideas of things you could test. Wow, look at the way these folks are structuring their content, maybe they're putting some animated GIFs in the content. What can I do around that? I have this new interface for the app I'm selling, I can actually put an animated GIF of how that works in the content. Just by looking around your own life, you can come up with these ideas, but the way you remove subjectivity from it is by testing those ideas, not just going in directly implementing them necessarily but conducting tests on them to see if they resound better with your audience or not. 7. Demonstrating the Project: Okay, so we've been talking about testing and what I think would be actually pretty interesting to do, is to run through real quick an example of a test. So, this test comes from my own life. Specifically I work at MailChimp and one thing male chimp does is we send email to our customers to tell them about various things in the application that they might be interested in, blog posts might want to read, other customers and what those customers are doing. So, I have app in front of me here, two versions of an email campaign that we sent and the question we had was oftentimes in these newsletters we'll have some content that won't necessarily have an image associated with it, it'll just be like, hey here's some blog posts you might want to read. In this particular case, we've got some content related to our new emoji support. So, within the application, you can insert emojis into subject lines going to let people know about this right. But, on our list, there's a group of people who don't click a whole lot. I've got two different versions of content up in front of me, and I want to figure out for those non clickers, how can I get them to engage? You can see it in sort of our control version, it's just this is feature spotlight emojis and it's in red and it's underlined, but then in the second version that we tested out, this content doesn't have an image. So, what we're gonna do instead, is just draw a box around it. Just try to call the I more to the content and figure out, is that going to get people to stop and look at this box. Okay, we're spotlighting it. So, we just built a segment of non-engagers and then test it out content a versus content b on them. In this particular case, it's kind of depressing, it was actually a statistical tie. That's fine though, we realized, you know what, either of these ways of presenting the content work just as effectively. So, this is something that you can do and think about. Is okay, I've got a particular group of people maybe it's a list, maybe it's a segment on that list and there's something I want to try out on them. I can pull out that segment and pull out that list, generate two different types of content, two different newsletters, send it out and see which one works best. I want to show you this example of a test we run and we actually run this test not against our entire list, but actually against a segment of people who had ceased to engage with this particular newsletter. You don't have to run a test against your whole audience. If something is working for a group of people and you don't want to mess with it, you can actually segment then out, maybe into a separate list or separate segment and then focus on the people that you want to get some improvement out of or learn something about. Just to give an example of a newsletter, I have this email from Herschel and let's just talk about what they could potentially track in this email. They've got lots of different links in here, you've got some high-level links at the top around O click over the shop or the blog. Then they've got some new merchandise right at the top here, new wallets and pouches then you scroll down further, and they've got additional merchandise. Specifically here they've got some things about a journal and some other links. Then down at the bottom, they've got some social links. So, at a high level, they could just track, click rate coming out of the newsletter. Or even just general purchases or cart ads coming out they've integrated Google Analytics or their e-commerce technology into MailChimp. But then they could also look at, okay we've got some high-level links at the top that are new merchandise then some links down at the bottom, what are the click rates for the individual URLs, right. So, they could pull up maybe a heat map of the campaign and see where people are going. Which ones are going to show up when people open up the email on those particular devices, is that causing people to maybe not engaged with the lower stuff because they're not scrolling far enough, do they know their stuff below or recalling that out somehow. So, they can think about that by looking at engagement by device within the application as well. 8. Final Thoughts: As a data science guy, I like numbers. However, it's important to keep these numbers in their appropriate context and not become obsessed with moving the needle for the purpose of just moving the needle, right? Sometimes people can become metrics obsessed. What that will often lead to is playing games with the numbers and doing gimmicks. But if you're a business selling things or if your goal is to nonprofits to get donations. You might not actually see purchases happen. You might not actually see donations happen. Instead all you've done is played around with some gimmicks to drive one particular metric that doesn't matter. So it's important to keep those in context. If I just started my business and I'm emailing only 100 customers, some of which happened to be my buddies. Then of course my stats are going to be great. But as I gain traction and my business grows those rates are probably going to fall, right? A 50% click rate when I'm sending to my 100 closest customers, makes a lot of sense. A 50% click rate when I'm sending to 200,000 customers is almost unheard of. So keep that in mind. Be realistic. Picture taking the time today to go through this class. I hope you've learned something about how to better use data and that you'll begin to think in terms of data, right? Not only that I'm just going to blast out content into the ether and maybe something hits with my audience but rather that I can actually look at how they're responding to things using data and use that to alter my behavior in the way that I speak to people. So, now it's your turn go into your inbox find an interesting email that you want to talk about. Think about what you might test to improve its performance. What metrics you would look at to understand whether those tests were successful. Share that particular idea and start talking with your classmates about what they might change.