Newsletters
Spooky Data Horror Stories
Actionaut Shaun Davis’ “Analytics Advantage” is a weekly newsletter of actionable insights, proven strategies, and top tips for getting the most from your data and making high-stakes decisions with confidence. Here’s a sample issue. We hope you’ll subscribe.
***
Data nerds know that with great power comes great responsibility. Leaders trust us a lot and we run on something tricky: implied trust.
Every decision we make is a step up the ladder, and every step depends on others believing we’re getting the data right.
Sometimes, we have a clear source of truth to back us up. But let’s be real, that’s the exception, not the rule.
In the spirit of Halloween, let’s look at some spooky and frankly horrifying data stories and learn how to avoid your own Texas Chaincode Massacre.
Knight Capital’s Zombie Code Gone Wild
Who doesn’t love a good zombie movie like 28 Iterations Later?
Unfortunately for Knight Capital, their version of a zombie outbreak wasn’t playing in theaters. It happened on the stock market floor.
In 2012, Knight Capital was the largest US trader in equities. Their Electronic Trading Group (ETG) handled an average of 3.3 billion trades every day. They moved over $21 billion in daily volume.
On August 1st, a glitch caused their system to buy $7 billion worth of stocks in less than an hour. Knight had no choice but to sell them at a loss, bleeding out $440 million before they could stop the hemorrhaging.
Zombie Code Comes to Life
According to news articles and insider retellings, the disaster happened when Knight’s team used old code in a new system. One server didn’t get updated properly, resulting in conflicting versions of code running simultaneously.
Their system started buying and selling at random, ignoring trading algorithms, and in classic corporate fashion, IT blamed the quantitative analysts instead of looking at their own code.
Lessons from the Knightmare
As the saying goes, “Slow is smooth, smooth is fast.”
Knight sped through their deployment without balancing speed with caution. Reusing old code without fully understanding how it would impact the new system was their downfall.
In the end, they had all the time in the world to think about that mistake–after getting fired.
Technical debt? Manage it!
Zombie code doesn’t belong in your systems, and if you leave it, it can come back to haunt you. Always have safeguards in place; more than you think you need.
It took Knight 45 minutes to staunch the bleeding. If they’d pulled the plug in 15 minutes, there might’ve been some frustration, sure, but maybe their company (and jobs) would’ve survived.
Tax Dollars Go Boo-m in Space
2001: A Space Odyssey? More like $700 Million: A Space Disaster.
Data errors haunt the world of rockets, too. Just ask Ariane 5 Flight V88 and the infamous Mars Climate Orbiter.
Ariane 5: The Curse of Copy-Paste Code
In June 1996, the European Space Agency (ESA) launched the Ariane 5 from French Guiana. Designed to be more powerful than its predecessor, the rocket’s mission was to deploy four Cluster spacecraft to study Earth’s magnetic field and its interactions with solar winds.
But instead of a successful launch, Ariane 5 went up in flames.
The problem?
Copy-paste coding and a bad hand-off between teams.
The software was borrowed from Ariane 4 and included a piece of code that converted speed data into a limited storage format. Nobody told the new team about this limitation.
The more powerful Ariane 5 produced speeds too large for the code to handle, which resulted in what programmers call an integer overflow–like trying to pour a gallon of liquid into a tiny cup. The system overflowed, interpreted the error as a command, and everything spiraled out of control. The onboard safety system stepped in and blew up the rocket to protect people on the ground.
Mars Climate Orbiter: When Units Go Missing
The Mars Climate Orbiter saw the Ariane 5 disaster and said, “Hold my ASCII table.”
In one of the most most infamous data blunders, the Mars Climate Orbiter failed to do what they tell you in elementary school math class: show your units.
Engineers at the Jet Propulsion Lab ran all of their calculations in metric units: Newtons. However, when these calculations were handed off to Lockheed Martin Astronautics, the engineers there assumed the data was in Imperial units (pound-seconds).
The probe’s launch and trip to Mars were flawless. But when the craft tried to enter orbit, the unit mix-up threw the navigation system off course. Instead of orbiting the red planet, the probe plunged into its atmosphere and burned up.
Several investigations ensued and it turned out that the units got mixed up between the two teams.
Measure 4 times, check twice, cut once
Show your units. Always.
It was the first time NASA’s Jet Propulsion Lab and Lockheed Martin had collaborated on a project. There were bound to be issues, especially with government-adjacent companies like Lockheed.
But these kinds of errors arise when assumptions, like everyone using the same measurement system, go unchecked.
Also, when using tax-payer dollars, as we like to say at Action: “Measure four times, check twice, cut once.” Because, millions of dollars are on the line.
Don’t Let the Data Errors Come Back to Haunt You
What’s a busy leader to do? Learn from the ghosts of data failures past.
These failures came down to a misunderstanding of the risks that come with change and speed.
When designing a new data product, identify the risk of making a wrong decision. Not every product carries the same weight. Some involve low risk, while others involve bigger consequences. Push your leaders to ensure the development teams understand the impact of errors.
Innovation is the baseline, but it’s crucial to understand the real cost if things go south. Watch out for those who might be driving change just to pad their resumes.
Have a plan. Document everything. And if things go wrong, exorcize the process, not the people. Slow is smooth, smooth is fast.
So, what data horrors keep you awake at night?
