Measuring experiences: the essential KPIs for data-driven design work
The goal of a designer is usually to renew/transform the customer/user experience. This starts with strategic action, but the rubber meets the road only when touchpoints are changed — for example, the UX of a website, the logic of an app or the brand of a company.
Measuring the impact of a new experience is always tricky. No offence, but most business intelligence analysts suffer from a chronical lack of imagination. Designers, on the other hand, often show little concern for practical issues — such as the business impact of their work. Let's get the best of both worlds.
When it comes to experience, there are two tasks at hand: quantifying and measuring. And quantifying something abstract requires a lot of imagination. How to kick-off the process? In order to illustrate this, let's imagine an ice-cream parlour working on the launch of a new flavour.
You will need a snappy name, cool brand, fresh colours, and a hashtag. At the end of the week, you have Peachy Keen, a creamy blend of peaches and mango (yum!) that should generate buzz and loads of sales next summer.
State the benefit, clearly
To measure an experience, a professional needs to think of the benefit that users actually experience.
What benefits does the experience bring? Well, Peachy Keen is delicious. And secondly, let's say it's set to be a new hip thing, so the benefit is to show you are in the now and on the loop, playing with the breezy flavour around Instagram. All in all, say you want that Peachy Keen makes customers feel rewarded and happy.
Now, how do you measure the happiness experience offered by Peachy Keen? It gets tricky, doesn't it? This example will show you how you must think metrics in design work.
The gold standard for measuring happiness is the increase in dopamine levels in the brain. Dopamine is the hormone connected to the feeling of happiness. It's released in our brain when we receive some gratification, such as sex, food and other pleasurable sensations.
You are not very likely to be able to measure, in real-time, the levels of dopamine released in the brains of your customers. If you could, you would come up with something like this:
Customers who take Peachy Keen get an increase of 150% of dopamine released right after consumption, compared to the normal brain levels (for us to have an idea, this is a similar concentration level when we eat (compared to 250% increase in concentration after sex, according to Di Chiara et al., Neuroscience, 1999., Fiorino and Phillips, J. Neuroscience, 1997.).
That would be a very weird ice cream parlour, but probably they would have a quite strong base of loyal customers. So if brain meters are out of question, what can you do to ensure you are measuring experiences?
The instinctive answer is usually the same: "measure how much it is selling". Unfortunately, this is the wrong answer. You would be measuring your product's performance, not your customer's experience.
And this matters because if you measure only the performance numbers, which happens after you deliver a great product ("Peachy Keen sold 1,000 this week). On top of that, you must keep in check that the actions that make your product great are being carried on (every customer got their 150% increase in dopamine).
Look for signals
As we already established we cannot measure brain hormones, we need to look into what do the happiest customers of Peachy Keen do after they have the ice cream? We need to look for signals, preferrably, measurable signals. It is a proxy. It is the next best thing from dopamine level measurement.
Perhaps you can keep the record of the amount of "yummy" exclamations right after the first try.
Perhaps you can measure how many customers return the next week and order, again, Peachy Keen.
Or... you can instruct customers on how to express their appreciation — by using a the hashtag #TeamPeachyKeen or something like that. That you can measure effortlessly.
This is what statisticians and finance accountants call a leading indicator. It doesn't measure growth and revenue, it ensures growth and revenue. For a designer, this matters because it starts to create attribution to your work. That is, it starts to tell that it is your work that is generating a certain amount of revenue.
Why not only measure revenue?
Because revenue only tells your first purchasers. It's an expensive way of measuring things. Sean Ellis has a lot of literature about what he calls the North Star Metric.
Scenario 1 — Measuring only revenue
Say you invested in advertising enough to bring 10,000 people to your shop. From those, 5000 people bought Peachy Keen. But you don't know if any of them liked the product or if they will ever come back.
Perhaps on the following month, you end up having only 100 units sold. You will need to spend big again, and bring a new batch of 10000 people to buy Peachy Keen. Because your formula probably sucks. Because you are not releasing any dopamine in their brain.
Scenario 2 — Measuring experience / retention
Instead, if you measure that from 5000 people who bought the product, 2500 of them used the hashtag, you know that your product does not suck. You know that at least 50% of every customer will love it. You may run a survey or have a chat with part of these people, and look carefully into the profile of these people and understand more their context, demographics, psychographics.
The question to explore: what do the best customer love about Peachy Keen? And how to bring more people like this to the shop?
Lastly, build your parameters
Once you give it time, you will be able to see increases or decreases in the hashtag use, viral loops and if the same users use the hashtag more than once. You will be able to see what type of Instagram account uses the hashtag, the demographics and even sentiment analysis (are they complimenting or complaining about the product?).
More importantly, you will be able to introduce new elements to the mix, and measure how they affect the experience measurements. You have, then, a great laboratory for experimenting with ways to increase retention and the expansion of your product. A real science lab, even without dopamine tracking.
State your experience benefit clearly.
Find the next best measurable metric to assess it.
Make sure you measure over time.
Observe over time how it relates to revenue.
Introduce new elements and observe how it affects your numbers.