Your Metric is Overloaded

Have you ever been in an elevator with too many people in it? Not just an uncomfortably close amount, but so many people that the control panel dings that it’s overloaded?

The same thing happens with metrics, only there’s not a handy “ding” to tell you something is wrong.

I was talking with a client the other day, and they relayed a major shift they needed to make in how they categorized a process. For years, they had ranked what was important along one dimension using a single metric. I’ve been in that conversation more times than I can count.

When it’s healthy, turning data into action is rarely static. The people, processes, and data are constantly in motion. It’s inherently iterative. But there are periods of stability, and sometimes those periods turn into stagnation.

The Metric No Longer Means What It Says

The biggest sign that it’s time to re-evaluate a metric is dislocation.

When people sense a difference between what the metric measures and what it means in reality, they look at two things that are rated the same and can clearly see that they are not equivalent. Only by digging into narratives or adjacent metrics are they able to get the full picture of what a single measurement truly means.

Soon, there are attempts to subdivide the main metric into subcategories with decimals, hyphens, and other add-ons, all of which take away from its true meaning. An easy way to tell is by asking someone if X really means X when looking at two records. They will know in their gut that it doesn’t.

A knock-on effect is eroding trust in the data. People start thinking, “If this metric isn’t representative, then can we trust any of this?” It can become a threat to the standing of you and your team with your customers.

When They’re Arguing, the Metric Is Already Dead

When users spend more time arguing about how something is categorized, it’s time for a change.

The entire purpose of a measurement system is to create a shared and efficient understanding of an event. We’ve all sat through these arguments, endlessly going around in circles asking, “What does this even mean?” That’s your ding.

The closer a metric is to someone’s personal identity or livelihood, the more passionate the arguments. Think of public school ratings, grade inflation, or test scores. The more dislocated a metric becomes from its original system, the louder the arguments become.

Also, if there was a period of arguing that didn’t result in a shared resolution, that’s a massive warning sign. It means your customers no longer trust the data and have moved on to a shadow system, or they’re simply ignoring what you produce.

A Note on Changing Metrics

Sometimes metrics are fully within your control. Other times, they are subject to input from a massive community.

Take hurricane categories. One person can’t simply come up with a new category ranking. The category system is massively load-bearing in science, insurance, and politics.

The metric is overloaded. The system won’t tell you. But the arguments will.

That client found a way through. They introduced a paired measurement stream, adding a new dimension to how an event was scored. It’s like that moment in Inside Out where a ball of emotions can have two dimensions. Riley can start to feel not just Joy, Sadness, Fear, Anger, or Disgust, but combinations of those emotions that produce Anxiety, Ennui, and more.

Next time you’re arguing about a dislocated metric, take a step back and ask yourself, “Is this overloaded?” If so, it’s time for a change.

The elevator has a ding. Your data system doesn’t. But now you at least know what to listen for.

Be kind to your future self.

— Shaun