On the Complexity of Data Perfection

Why accuracy gets harder to achieve as systems get better

Written by Ari Gross

Intuition suggests that improving a system’s accuracy should make the remaining work easier: fewer errors, less to fix. In practice the opposite happens. As a system, whether it is AI-based or not, climbs from mediocre to good to excellent, the marginal cost of each additional point of accuracy rises sharply, and the humans responsible for catching what the system misses become progressively worse at doing so. This paper describes that dynamic, why it happens, and what it implies for how organizations should design human oversight of high-performing systems.

The Diminishing-Returns Curve

Most systems follow a rough progression:

• 0–70% accuracy: errors are frequent, large, and obvious. Humans catch them almost by accident, simply because there are so many.
• 70–95% accuracy: errors are still common enough to form patterns. Humans can build habits and checklists around them.
• 95–99.5% accuracy: errors become rare, individually distinct, and often only visible in context that the reviewer must reconstruct.
• 99.5%+ accuracy: remaining errors are frequently edge cases the system has never been tuned for, adversarial in nature, or artifacts of a compounding chain of small, individually-plausible decisions.

Each step requires disproportionately more engineering effort than the last — a well-known effect in machine learning, where the last fraction of a percent of accuracy can cost as much as everything that came before it combined. But the harder problem is not engineering. It is that the humans meant to catch the remaining errors are, at the same time, losing their ability to do so.

Why Human Correction Ability Degrades

Three well-documented human factors compound as system accuracy rises:

Vigilance decrement. Sustained attention to a rare-event detection task degrades measurably within the first 20–30 minutes and continues to degrade the longer the task continues. A system with a 10% error rate keeps a reviewer’s attention active through frequent confirmation. A system with a 0.1% error rate offers almost no such reinforcement — the reviewer’s attention has nothing to hold onto.

The prevalence effect. Signal-detection research (most notably in airport baggage screening studies) shows that as the base rate of a target falls, the miss rate for that target rises — even when the reviewer is told the rate is low and is motivated to find it. People are calibrated, unconsciously, to what they expect to see. A very accurate system trains its own reviewers to expect nothing wrong, which makes the rare wrong thing harder to see, not easier.

Automation complacency. The more consistently a system performs well, the more its output is trusted by default, and the more review shifts from active verification to passive approval. This is sometimes called the irony of automation: the better the automation, the more critical — and the more atrophied — the human’s role becomes, because the skill of catching errors is itself something that requires practice to maintain, and a high-accuracy system starves that practice.

These effects interact. A system that is 99% accurate does not present its reviewers with a proportionally smaller version of the same review task. It presents them with a qualitatively harder task — rarer signals, higher trust, less practiced attention — performed by people who are, in effect, being systematically deskilled by the very success of the system they are reviewing.

Why This Matters More for AI Systems Specifically

Three properties of modern AI systems sharpen the problem:

• Fluency masks error. Outputs are often well-formed, confident, and internally consistent even when wrong, removing the surface-level cues (garbled text, obvious formatting breaks) that once signaled “check this.”
• Errors cluster in the tail. A model’s mistakes concentrate in exactly the inputs that are rare, ambiguous, or outside its training distribution — which are also the inputs a time-pressed human reviewer is least equipped to evaluate quickly.
• Scale removes the safety net of redundancy. At low volume, a human might independently redo the task and catch a mismatch. At the volume AI enables, independent redundancy is usually the first thing cut, precisely because the system “doesn’t need it anymore.”

The net effect: the point at which a system becomes accurate enough to be trusted broadly is also the point at which the humans overseeing it are least equipped to catch what it still gets wrong.

Implications for Practice

If human vigilance cannot scale to meet rising accuracy, oversight has to be redesigned rather than simply relied upon more heavily. A few directions follow directly from the diagnosis above:

1. Stop optimizing review for volume; optimize it for detection. Sampling strategies that deliberately inject known-bad or synthetic edge cases into the review stream keep reviewers calibrated to what errors actually look like, countering the prevalence effect directly.
2. Rotate and time-box review tasks. Since vigilance decrements with duration, short, frequent review sessions outperform long ones for rare-error detection.
3. Measure the reviewer, not just the system. Track human catch-rate on injected test cases over time. A dropping catch-rate is a leading indicator that oversight has stopped functioning, well before an outage or failure reveals it.
4. Treat “human in the loop” as a claim to be tested, not a control to be assumed. A named reviewer is not evidence of review; documented catch-rates are.
5. Push some error detection upstream, into structural checks Consider utilizing automated methods that do not depend on sustained human attention to check for problems that include anomaly detection, adversarial testing, and disagreement between independent models or methods.

Conclusion

Data perfection is not a destination a system approaches smoothly. It is a moving target that gets harder to hit precisely because hitting it changes the behavior of the people responsible for the last mile. Organizations that treat “we have a human reviewing this” as a static safeguard, rather than a capability that degrades as the system improves, will find their oversight quietly failing at the exact moment their system looks most trustworthy.

References

1. Bainbridge, L. (1983). “Ironies of Automation.” Automatica, 19(6), 775–779.
2. Wolfe, J. M., Horowitz, T. S., & Kenner, N. M. (2005). “Rare Items Often Missed in Visual Searches.” Nature, 435(7041), 439–440.
3. Wolfe, J. M., Horowitz, T. S., Van Wert, M. J., Kenner, N. M., Place, S. S., & Kibbi, N. (2007). “Low Target Prevalence Is a Stubborn Source of Errors in Visual Search Tasks.” Journal of Experimental Psychology: General, 136(4), 623–638.
4. Mackworth, N. H. (1948). “The Breakdown of Vigilance During Prolonged Visual Search.” Quarterly Journal of Experimental Psychology, 1(1), 6–21.
5. Parasuraman, R., & Manzey, D. H. (2010). “Complacency and Bias in Human Use of Automation: An Attentional Integration.” Human Factors, 52(3), 381–410.
6. Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). “Does Automation Bias Decision-Making?” International Journal of Human-Computer Studies, 51(5), 991–1006.
7. Sculley, D., Holt, G., Golovin, D., et al. (2015). “Hidden Technical Debt in Machine Learning Systems.” Advances in Neural Information Processing Systems (NeurIPS) 28.