Let’s start with a story.
Once, there were two brothers, John and James, who moved west to seek their fortune in the California Gold Rush.
Upon arriving in San Francisco in the late 1860s, they realized that panning for gold was mostly unprofitable. Determined to succeed, they decided to become traders, selling whatever people needed in the growing city.
The older brother, James, was convinced that the city needed a big store, so he toured the entire countryside, doing careful research on what people were buying, hitting up potential investors and trying to find data that corroborated his hypothesis.
The younger brother, John, on the other hand, bought the smallest shop he could afford and started selling the first thing he could think of – shovels. To save money, he slept at the shop itself. After a few weeks when sales started slowing down, he switched to selling denims. Sales shot up again.
This way, he started experimenting with small quantities of different products: panning equipment, foodstuff, even livestock. The products that sold well, he kept; others, he discarded. Soon, profits grew, as did his shop.
By the time James turned up in San Francisco with a thousand-dollar investment and a ton of data on what people were buying, John already had a prosperous business with hundreds of loyal customers. Despite his big investment and data, James couldn’t compete and soon folded shop and went back to Oregon.
Fictional though it may be, the story shows two very fundamentally different ways of doing business: the top-down, go-big-or-go-home approach, and a smaller, leaner model that relies on real-time, actionable data.
While James was busy finding data that justified his original hypothesis, John built a business through small, incremental changes based on data he gathered through observation and feedback from his customers.
The Conventional Approach to Statistics Doesn’t Work
The conventional theory of statistics is broken. Nobody would tell you in business school, but it’s pretty apparent to anyone who has ever worked in tech fields how slow and outmoded the conventional approach really is.
Traditionally, if you wanted to, say, learn the market demand for a type of ice cream, you would first conduct exhaustive market research, hire a bunch of food tasters, get some attractive packaging designed and try to strike up a distribution deal with grocery stores. This approach could be broken down into multiple steps:
- Understanding what needs to be measured, i.e., creating a statistical model
- Collecting data that works within said model
- Summarizing collected data
- Interpreting collected data
- Creating an action plan that works on interpreted data.
This is a rational, scientific approach that has mostly worked well enough, except for a few major flaws:
- This approach assumes that you actually know what needs to be measured; that is, your statistical models are correct. It cannot account for things that may lie outside your models. Maybe your model considers flavor, brand name and smell as important factors in ice cream purchases, but consumers actually buy your ice cream simply because of the shape of the tub it is sold in – something you never thought of measuring.
- The process of collecting, summarizing and interpreting data is slow – a big hindrance in fast-moving industries.
- Collected data is often used in hindsight to justify past decisions. Nicholas Nissim Taleb calls it the problem of inference, where you use data to perpetuate your own narratives (just ask the people over at Lehmann Brothers) instead of using data to make future decisions.
That the conventional approach to statistics doesn’t work is no secret. People have known it for years. The only problem was that there really was no alternative; broken as it may be, this was the best we could do.
This has changed rapidly in the last few years, thanks to technology. We can now test new things and get almost instant feedback. Instead of spending thousands of dollars on market research, you can throw up a website selling your new ice cream flavors, buy a bunch of ads on Google and see what sticks. You can collect data in real-time, see which flavors are selling, which aren’t, and modify production accordingly.
This, ladies and gentlemen, is the new approach to statistics, and it is changing the way we do business.
Data Is a Lie
Businesspeople, bureaucrats, economists, town planners and policymakers across the world suffer from a particularly troubling delusion – that all data is good data.
There’s something comforting about data. It makes our inferences sound more “legitimate” and couches our intuitions in the language of science. “Washing hands reduces the spread of disease” is probably right, but “Research indicates that washing hands reduces incidents of disease by 45%” sounds way more convincing.
Blame it on our rational, Enlightenment-influenced mode of thinking, but we tend to turn to data whenever we are faced with difficult decisions. This is a fine way to approach problems, except for the fact that we often end up using wrong data to make decisions.
When Metrics Fail
All data can be basically divided into two categories:
- Actionable data
- Vanity data
As you would’ve guessed, actionable data (or metrics) is data that helps you make actual decisions. Vanity metrics, on the other hand, are pure fluff. This data looks nice on paper but doesn’t really do much for decision making.
For example: if you want to lose weight, you should weigh yourself at the same time every day (or every week); collecting data on the number of times you washed your hands each day would be redundant. Sure, the data will look good and may even help you cut down on your soap usage by a smidgen, but it wouldn’t help you in your desired goal of losing weight.
Measuring your waist is an actionable metric; calculating the amount of soap you use daily is a vanity metric.
You see vanity metrics all the time. When someone says they increased their customer count by “100%,” it sounds impressive but doesn’t really tell you anything important.
Maybe they had zero customers before and just acquired their first customer, shooting up the customer count by a 100%. Or, when someone says they had 1M page views last month, you first need to know their page view count the month before, the bounce rate, the amount of money spent in acquiring those page views, and most importantly, the conversion rate, before you can decide whether the metric is actually useful.
Actionable Metrics vs. Vanity Metrics
Any data that helps you make decisions and move closer to your stated goal would fall under actionable metrics.
It’s important to understand that these aren’t watertight categories; the same data might fall under “actionable” in one instance, “vanity” in another. For example, page views is a key actionable metric for a digital media company that sells ad space.
A company that sells a downloadable app, on the other hand, needs to pay attention to conversion rate, not number of page views. Higher page view numbers would look nice on paper (and please VCs), but it wouldn’t really help you increase your downloads, which is the metric that matters here.
The traditional approach to statistics, with its slowness and static models, spews a lot of vanity metrics. This is because most vanity metrics are easy to calculate.
A simple hit counter can tell you how many people visited your website. It can also tell you how long they stayed on a web page. What it can’t tell you is what elements on the web page convinced them to stay and what elements you can change to convince them to come back.
Actionable Metrics and the Lean Startup Model
Let’s go back to our original story. John, without really knowing it, was following the lean startup model. He wasn’t really using any fancy analytics software, of course. Instead, he was simply observing what people around him were actually buying, i.e., real-world data, and what items were selling well in his store. Based on this feedback, plus the verbal feedback from his regular customers, he was changing his inventory constantly – the quintessence of the lean approach.
For John, the actionable metrics would be:
- Amount of time an item-type spent unsold
- Number of requests for an item-type
- Cost of procuring and stocking an item compared to its selling price
- Observable demand for an item-type.
Perhaps more importantly, John was making decisions in real time. He wasn’t making grand models and conducting in-depth research to come to conclusions. He was simply looking at what was selling and buying more of it.
This should be your approach when using actionable metrics. Instead of making decisions and then gathering data that support the hypothesis (like James), use data to make small, incremental changes. This data would fall under the following three categories:
- Customer-Focused Data: Try to implement data collection models that emphasize per-customer data. This means that instead of looking at bounce rate for the whole month, you should consider bounce rate for new versus returning customers, customers on a particular screen type or customers from a target geographical market. The idea is to eschew general data in favor of focused metrics. At the same time, you should strive to collect subjective feedback as well – what individual customers are saying about your product or website.
- Conversion-Related Data: Regardless of what business you run, the most important metric you can measure is the conversion rate. To this effect, you should measure not only the conversion rate, but also the path taken by the customer to reach the conversion point, i.e., the conversion funnel. Ask yourself, “What steps did the customer take to click the ‘buy’ button? Where did this customer come from – search, social, or paid ads? How long did it take for the customer to reach conversion point? Is the customer a repeat buyer? If yes, what was their previous purchase amount? Is there a noticeable drop/increase in engagement or order value, and if it is, what prompted this change?”
- Marketing-Related Data: The last metric you should measure is related to your marketing. This can not only indicate the efficacy of your marketing, but also reveal a lot about your customers. Customers who come through generic, broad search keywords such as “graphic cards” are different from customers who came to your site through long-tail keywords like “sapphire ati/amd hd 7790 graphic card.” Similarly, first-time customers who bought during a promotion will be different from regular customers who found your site through word of mouth.
To gather this data, you should master split tests and find analytics software with strong segmentation capabilities. Your analytics software should also help you identify customer funnels and gather subjective feedback.
The lean startup approach to metrics is a whole lot different than the way businesses traditionally use statistics. The focus is on real-world, actionable data gathered in real time. Any data that doesn’t help make decisions is discarded. And perhaps most importantly, data drives decisions, instead of the other way around.