4 Levels of Measurement
Companies at all sizes are afflicted by a paradox of measurement. The more data we have, the better decisions we make. In support of that, we want to measure everything and measure it precisely: clicks, opens, command-line invocations and website visits. We want to produce a vast sea of data and transform it into a beautiful set of dashboards that tell us what’s working, what’s not, and where we should spend more time.
The problem is that analytics work can cost a significant amount of time and money, especially in small companies. Collecting, aggregating, analyzing and presenting data all take real effort, effort that’s not being spent building product, writing content, or talking to customers.
Adding to the confusion here is that the results of such an effort are impossible to predict. We can’t know how better data will inform better decisions until we have that data.
I’d like to equip you with a framework for navigating this paradox and figuring out how you can get the biggest bang for your buck on analytics work you choose to perform. The four-levels framework is a simple heuristic designed to allow you to talk about measurement in a simple and pragmatic way. It will get you out of the weeds and give you a structure to make common-sense decisions about where you should invest in better data and where you can get by with the data you have.
The 4 levels
Level 1 is the simple act of reporting that you did something: “We wrote a blog post!” The advantage of level 1 is that it’s incredibly affordable: it takes very little effort to communicate at level 1, and it’s still valuable communication.
Level 2 is when you measure the size of an effect but don’t tie it to business impact. For a newsletter, this could entail measuring the number of people who open the newsletter or click a link. For a feature, it would mean measuring the number of people who use that feature.
Level 2 is a great place for small startups to aim to hit consistently. It gets you to start thinking about the impact of your work in a quantitative way and can highlight early analytics obstacles that warrant attention.
Level 3 is when you measure business impact in a way that hints at causation. A level 3 measurement for a newsletter would be to see how many readers sign up for your product after reading the newsletter. An even more sophisticated measurement would be to see how many of those signups convert to active or paid users.
Level 3 is often where the cost/impact discussion surrounding analytics starts to get interesting. Sure, it would be lovely to tie all your efforts back to revenue impact. However, performing this kind of analysis is often costly and may not be worth the effort it entails. Before engaging in this kind of analysis, have an honest conversation with your team about how the results of this research will affect your actions. If you can’t produce a realistic story about how sophisticated analysis will impact the work you’re doing, don’t perform the analysis.
Level 4 is quantifying impact in a scientific way, such as hiding a blog post from half of the site visitors and trying to determine if there’s a significant difference in signup behaviour between visitors who see the blog post and those who don’t.
Level 4 is the apex of both sophistication and effort. Unless you have a dedicated analytics team, I rarely recommend performing this kind of analysis.
Picking the Right Level
Let’s be clear: higher is not better. While moving up this framework results in increased insight, it also means you’re spending more time looking at numbers instead of building product or talking to customers.
Instead of aiming to always hit level 4, use the following questions to determine what an appropriate level is.
- What level are we at right now?
- How do we imagine moving up a level would affect the decisions we’re making?
- How much would it cost to do so?
- How would moving down a level affect the decisions we’re making?
- How much time and money would that save us?
