Ten years of disasters, zero AI
Take the best known software incidents of the decade preceding the mainstreaming of coding assistants. Not the sophisticated state-sponsored attacks, not the finely crafted zero-days: the ordinary disasters, the ones that made the front page. Almost all of them fall into six families of structural stupidity.
The port left open on the Internet, with no authentication. In 2015, the Jeep Cherokee exposed a port of its embedded system running a service as root, with no authentication: two researchers steered and braked the vehicle remotely. Between 2015 and 2017, tens of thousands of MongoDB databases listened on the Internet with no password, the default configuration: in January 2017, more than 33,000 of them were wiped and held for ransom by simple scripts. As of 2026, 3,100 were still exposed.
The default credential. The Mirai botnet (2016) exploited no vulnerability at all: it tried 62 username and password pairs of the root/root and admin/admin variety, enlisted 380,000 devices and took down a chunk of the American Internet. Its authors were three students. At Equifax (2017), while 147.9 million records leaked through an unpatched flaw, the company's Argentine portal was protected by admin/admin. SolarWinds left "solarwinds123" sitting on a public GitHub repository for seventeen months. Colonial Pipeline shut down because of a VPN account with no second factor.
The sensitive data in plain sight. Missouri, then. But also First American Financial (2019): 885 million documents accessible by simply incrementing a URL, a flaw known internally for months before it was disclosed. Or Optus (2022): a public API with no authentication, fixed on the main domain in 2021, never on the secondary one. 9.8 million customers.
The freshman mistake. CrowdStrike (2024): a template expected 21 fields, the code supplied 20, and the out-of-bounds read brought down 8.5 million Windows machines. The gap travelled through several layers of validation and testing. The incident postdates ChatGPT and has strictly nothing to do with AI. Cloudflare (2019): a regular expression with catastrophic backtracking saturated the CPUs of the global network for 27 minutes; the safeguard that should have limited the damage had been removed by mistake during a refactoring... intended to reduce CPU consumption.
The single command that sinks everything. Facebook (2021): one faulty maintenance command, an audit tool meant to block it but itself buggy, and the company erased itself from the Internet, to the point where its employees could no longer badge into their own server rooms. GitLab (2017): an rm -rf on the wrong server, 300 GB wiped, and above all the discovery that not one of the five backup techniques in place actually worked.
The classic. macOS High Sierra (2017): type "root," leave the password blank, click twice. You are an administrator.
Three observations run through these fourteen incidents. None of these disasters was sophisticated: a patch applied on time, a password changed, a port closed would have been enough in almost every case. The failures are organizational, not technical: a patch forgotten on one domain for four years, a flaw documented internally with no follow-up, five backup systems never tested. These are chain defects, not syntax defects. And stupidity survives every technological wave: if AI had introduced a genuinely new class of bug, we would know its name. We do not, because it does not exist.
“
Fourteen disasters, not one of them sophisticated, not one of them written by a machine. The constant is not the technology, it is the organization.
Bugs, then, did not wait for AI. What is new is something else: our intolerance. The same error, made by a human, is forgiven; made by a machine, it disqualifies. This asymmetry is not a hallway impression. It has been measured for twenty years, and it has two floors: a psychological floor and an accounting floor.
First floor: the machine gets no right to be wrong
The founding experiment dates from 2015. Berkeley Dietvorst, Joseph Simmons and Cade Massey, at Wharton, had participants work with two forecasters: a human and an algorithm. The central result: after seeing both make the same error, participants lost confidence in the algorithm far faster than in the human, and kept avoiding it even when it remained objectively more accurate. The authors named the phenomenon algorithm aversion. Two years later, Prahl and Van Swol isolated the mechanism in its purest form: as long as nobody errs, participants follow the human adviser and the machine adviser indifferently; the moment an identical piece of bad advice comes from both, only the machine is penalized. Identical advice, identical error, different punishment.
Why the asymmetry? Human factors research explains it through what it calls the perfect automation schema, described by Dzindolet as early as 2002 and then formalized by Madhavan and Wiegmann: we approach machines with two implicit beliefs, very high performance expectations and all-or-nothing thinking. The consequence is counter-intuitive: before any incident, the trust granted to an automated system is often higher than the trust granted to a human. But that initial trust is fragile by construction. Human error confirms our model of the world (humans are fallible, everyone knows that); machine error breaks a schema (the machine was supposed to be perfect). An expected error is absorbed. An error that contradicts the mental model disqualifies: it is no longer "the system got it wrong," it is "the system is broken."
Three aggravating circumstances follow. First, once the schema is broken, we durably overestimate the system's actual error rate: the memory of the incident weighs more than the statistic. Second, and this is the point that should interest anyone who has read the corpus above: it is the easy errors that destroy trust the most. Madhavan, Wiegmann and Lacson demonstrated this experimentally in 2006: an automated aid that fails at a task a human performs effortlessly is judged globally incompetent, far more harshly than an aid that fails at a difficult task. That is why a trivial AI-generated blunder circles social media within hours, while twenty years of admin/admin disqualified nobody. The corpus of this article is made exclusively of easy errors. It is precisely the category of error we forgive endlessly in humans, and never in machines.
Finally, the bar we set for the machine is not the bar of the real human: it is the bar of the human we believe ourselves to be. The best documented ground is the autonomous vehicle. Acceptability studies converge: the public does not require a self-driving car to do better than the average driver, it requires it to do several times better. Some work places the adoption threshold at four or five times human safety, and the British data ethics committee noted that aligning the requirement with rail or air travel would mean doing a hundred times better than the average driver. A large 2021 study (more than 4,500 participants) identifies the driver of this: algorithm aversion combines with the better-than-average effect; the better people judge themselves as drivers, the more they demand of the machine. Transpose that to software development, a field where nobody thinks of themselves as below average: the developer assessing generated code does not compare it to real human output, the output of the corpus above. They compare it to the image they hold of themselves.
That leaves the social layer, which is not a bias but an architecture: human error comes with a narrative (fatigue, the deadline, the context) and above all with someone accountable, someone who can answer for it, apologize, promise to do better. Our tolerance for human error rests on that possibility of recourse. Machine error appears faceless and excuseless: you do not summon a language model to a crisis meeting. Where there is nobody to forgive, there is no forgiveness.
Second floor: an accounting intolerance
It would be comfortable to stop there and conclude that our severity toward machines is just a bias to be corrected. That would miss half the subject. Because the demand for near-perfection from automation also has a perfectly rational foundation: the economics of delegation.
Human error is already in the price. Every organization has always paid for a complete infrastructure dedicated to catching humans: management, code review, QA, acceptance testing, audit, financial control. These costs are so old and so diffuse that they have become invisible: nobody books them under "human non-quality," they are called payroll and org chart. When a human makes a mistake, the cost of watching them was already provisioned.
Automation, by contrast, is sold on the opposite promise. The business case for a code generation tool, an agent, an automated pipeline rests precisely on the supervision work you will no longer have to do. Every machine error therefore reintroduces exactly the cost the investment was meant to remove: you have to reread, verify, test, redo. Machine error does not just damage the deliverable. It attacks the profitability of delegation itself. We do not forgive the machine because its reliability is not a nice-to-have quality: it is the product we bought.
That tipping point is not an intuition, it is quantified. In 2007, Wickens and Dixon compiled twenty experimental studies on automated diagnostic aids and found a sharp break: below roughly 70 percent reliability, working with automation produces worse results than working without it. Not a halved benefit: a net loss, because the cost of monitoring and catching up exceeds the benefit of the aid. Reliability is not a refinement of automation; it is its condition of economic existence. And Lisanne Bainbridge had already described, back in 1983, in her "ironies of automation," the structural trap that follows: the more you automate, the more thankless and difficult the human's remaining share becomes, monitoring a system that almost always works and catching its rare failures. A task we are notoriously bad at, and one that does not shrink through more automation: it moves.
“
“The designer who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate.”
Lisanne Bainbridge, Ironies of Automation, 1983
The most dangerous thing about this verification cost is that it is invisible to whoever pays it. In the summer of 2025, METR published a randomized controlled trial on sixteen experienced open source developers and 246 real tasks: allowed to use AI tools, they took on average 19 percent longer to deliver, while remaining convinced, afterwards, that they had been around 20 percent faster. METR has since qualified the reach of the figure itself (later waves of the study suffer from selection effects), but the result that held is the gap between perception and measurement: nearly forty points. Nobody feels the time they spend verifying. Usage data tells the same story at team scale: where the volume of merged code explodes, review time swells almost as much, and developer surveys put "almost right" solutions at the top of the frustration list, the ones that cost more to fix than to rewrite.
Intolerance toward machine error is therefore both a bias and a calculation. The bias sets the bar at perfection; the calculation demands that verification cost less than the delegated work, failing which automation destroys value. And both floors converge on the same question, one almost no organization has ever asked head-on: what does producing trust cost, at your company?
Trust never rested on the author
Because that is the blind spot of the entire discussion: we never really trusted developers. No serious organization ships code to production on the grounds that its author is a good person. What we trust, when the system works, is a chain: reviews, tests, integration environments, progressive rollouts, post-mortems. The chain is the historical answer to the economic problem of the previous paragraph: it pools and industrializes the cost of verification. It turns that cost from a variable one (each deliverable inspected by hand by someone) into a fixed one (mechanisms that inspect everything, all the time, at the same price).
The corpus lets us distinguish two regimes of that chain. At CrowdStrike, the validation layers existed and the gap travelled through all of them: controls that catch nothing, compliance ritual, what this magazine's editorial line calls parasitic friction. At GitLab, by contrast, the disaster did happen, six hours of data lost for good; but the fully public post-mortem and the live-streamed restoration produced a paradoxical effect: trust in the company came out stronger. Load-bearing friction does not guarantee the absence of accidents. It guarantees that the accident teaches something, and that the system can prove it.
Research on algorithm aversion also lights the way out of the trap. In a second series of experiments (2018), Dietvorst and his colleagues showed that participants go back to using an imperfect algorithm as soon as they are given the ability, however marginal, to adjust its outputs. Trust does not return through a promise of perfection. It returns through control. Transposed to our organizations: nobody will durably trust a largely automated production chain because they are assured the model is reliable. They will trust it because it exposes control points that can be verified to catch what they are supposed to catch.
The right question is therefore not "does AI write bugs?" It does, like everyone else. The right question is twofold, mirroring the two floors: is your chain able to catch defects, whoever their author? And does it do so at a cost below the value of what you delegate? An organization that answers yes to both can absorb generated code without flinching. An organization that answers no was already in no state to trust its humans; it simply was not measuring it, because the cost was diluted into the payroll.
What AI actually changes
AI did not invent bugs. It changed two things: throughput, and the visibility of the price of trust.
Throughput first. Where a team used to ship ten changes a week, code generation makes fifty possible. Every decorative link in the chain, every rushed review, every test that tests nothing was tolerable at low volume: the limited number of changes gave humans time to compensate informally. At high throughput, informal compensation no longer keeps up. What was a latent defect becomes a visible failure.
The price second. As long as verification was a diffuse human gesture, buried in salaries and rituals, no executive team had to fund it explicitly. Automation brings it into full daylight: every hour spent rereading generated code is a line item the tool's business case did not anticipate. Trust stops being a feeling and becomes an accounting entry. That is uncomfortable, and it is excellent news: what has a measurable cost can finally be industrialized, arbitrated, improved.
This is the thesis of this magazine applied to software quality: AI does not create the dysfunction, it reveals it. The organizations discovering today that they "cannot trust AI code" are in fact discovering that they never built the mechanisms that produce trust, for anyone, and that they never knew its price. The corpus above is the empirical demonstration across ten years; experimental psychology explains why we did not see it; the economics of automation explains why we can no longer ignore it.
The logical next step is not to slow the machine down, nor to wait for it to become perfect: the perfect automation schema is precisely the trap. It is to finally industrialize what should have been industrialized long ago: controls that control, backups that restore, post-mortems that circulate, and a verification cost that is known, budgeted and falling. The teams that do this work will be able to delegate massively, to machines and to humans alike. The others will go on debating trust while staring at their open ports.