The Paradox of Abundant Data

Actionaut Shaun Davis’ “Analytics Advantage” is a weekly newsletter of actionable insights, proven strategies, and top tips for getting the most from your data and making high-stakes decisions with confidence. Here’s a sample issue. We hope you’ll subscribe.

***

Data nerds only want one thing and it’s disgusting.

This obsession with more data always leads to the paradox of abundant data.

The Burden of Too Much Data

The paradox of abundant data is a burden. The paradox is that as we get more data, we understand less.

We think that if we collect just the right pieces of data, assembled in the right way, we’ll be enabled to make a decision more confidently.

But it’s a mirage.

We already have more data than we can ever use. And that can get people into a lot of trouble.

Remember the Cambridge Analytica controversy? Or Snowden revealing the large-scale surveillance gathering programs by the NSA? Regardless of what you think about these news stories, we know these people got in a lot of trouble for collecting too much (sensitive) data. At some point, someone approved the reckless strategy: “Let’s collect the data and worry about making sense of it later.”

As seen in nation-wide examples, the burden of the Paradox of Abundant Data can have serious consequences, like leading to a loss in consumer trust or citizens of a country losing their trust in government.

And, as consumers who have had our personal data probably gathered at some point in time, we’ve given up our right to privacy. It no longer exists.

How many times has someone sworn that their phone or speaker is listening to their conversation?

Now, the real question is not if our data is being collected, but how it’s being used.

Analysis Paralysis

The rise of privacy-first browsers like Brave, search engines like DuckDuckGo, and regulations like GDPR (the wide-ranging privacy regulations from Europe) are a response to the unchecked data collection that stems from our insatiable desire for more information.

Ironically, an overwhelming amount of data can lead to “analysis paralysis” rather than clarity.

What did Cypher in The Matrix say? Sitting across from Agent Smith, savoring a steak he knows isn’t real, Cypher says, “You know, I know this steak doesn’t exist. I know that when I put it in my mouth, the Matrix is telling my brain that it is juicy and delicious… Ignorance is bliss.”

Data Bloat

Data bloat. The struggle is real.

Imagine you start collecting a new stream of data. You estimate how much storage and computing power it will need, but you underestimate it–by a factor of 10.

Now your cloud costs have skyrocketed, and a Senior VP is asking what your plan is for managing all this data.

You thought that through, right?

Or, heaven forbid, someone takes all of your data in a breach because someone in IT didn’t get their annual pizza party. Now, the company’s earnings are taking a hit just as everyone was planning to cash in on their stock options. Oops.

But you knew what you were going to do with that data all along, right?

The Paradox of Abundant Data leads to the exact opposite of what you want. Instead of driving clear, data-informed decisions, the volume becomes overwhelming, leaving decision-makers unable to act.

So what’s a data nerd to do? Steal from Private Equity, of course!

Zero-Based Data

Private equity firms are companies that provide financial services to private companies for profit. They use investors to invest in private companies, grow them, or buy out a public company.

These firms use a financial strategy called Zero-Based Budgeting where every expense has to be justified each year or it gets scratched, instead of relying on past budgets.

I’d like to take their concept of Zero-Based Budgeting and apply it to data: Zero-Based Data (ZBD).

Every piece of data has to earn its keep, every year, or it gets deleted.
Every piece of data has a purpose and is tied to an outcome.

The process looks like this:
1. Keep an inventory of all the data you collect and create
2. Tie every piece of data to an outcome
3. If it doesn’t have an outcome, turn off the pipeline
4. If you really believe in ZBD, delete the data, the dashboards, everything.

Imagine a retail company that collects vast amounts of customer data from online purchases, loyalty programs, and in-store interactions. To implement Zero-Based Data, the company conducts an audit to tie each data source to a specific business outcome, like improving customer retention, or optimizing product offerings.

Data streams without a clear link to these outcomes are either removed or deprioritized, resulting in streamlined data use and reduced storage costs while maintaining a focus on actionable insights.

Implementing Zero-Based Data

To implement the Zero-Based Data strategy, make a plan for the data you have and how you think it will drive your intended outcome.

In a year, evaluate whether the data is actually driving that outcome. If the data didn’t achieve the outcome, resist the temptation to say you need more data or more granular data. Look hard at the way in which you use the data. Look hard at how the data is communicated.

This is also the time to ask your audience, the end-users, some hard questions. Are they getting the insights they need? Is the data being communicated in a way that drives action? Or, is it overwhelming and unclear?

Understanding the Real Issue

Maybe it’s not the data that’s the problem, but the culture and process around the data that’s lacking. So often, data nerds assume that the audience fully understands the data and what it represents, but that’s rarely the case. As data workers, we think deeply about all the details of the data, but most of our audience doesn’t operate at the same level of understanding.

Depending on your industry, it can take years to build the institutional knowledge required to use a particular data set effectively. The more stable and risk-averse your industry and company, the longer it takes for it to adapt to change. As a result, even with the right data in hand, it can take longer to implement new strategies, or improve how data is communicated or understood.

The Path to Tangible Outcomes

The solution isn’t more data, it’s a clear plan. Make sure the data you have is tied to real, measurable outcomes you want to see in your audience: increased sales, better uptime, fewer abandoned carts. If there’s no clear path from your data to an outcome, apply ZBD, and dump the data!

Happy deleting!