Listen to this article
In this issue — 4 parts
Part 1 of 4
The File Nobody Checked
The Problem: The Memo From Your Own Department Never Gets Proofread
Every organization of any size has a version of this habit. Documents from outside the company get read carefully, checked for red flags, maybe even run through a review process before anyone acts on them. Documents that originate inside the company, from a team everyone trusts, tend to skip that scrutiny. A memo from your own engineering department does not usually get the same suspicion as an email from an unknown sender. It is "us," so it must be fine.
On November 18, 2025, Cloudflare found out what happens when that habit is built into a piece of production software instead of just office culture. A file that Cloudflare's own systems generated, automatically, from Cloudflare's own database, was never validated the way an external input would have been before it was pushed to every machine on Cloudflare's global network. The file was malformed. It had doubled in size past a limit nobody thought it could reach, because nobody expected their own system to produce a file like that. The software reading it crashed. Because that software sat inside Cloudflare's core traffic proxy, not off to the side, the crash did not just disable one feature. It took down the path that a very large share of the internet's traffic runs through.
Cloudflare's own postmortem calls it the company's worst outage since 2019. For roughly five and a half hours, starting at 11:20 UTC, a long list of familiar names went dark or degraded at the same time: X, ChatGPT, Spotify, Shopify, Dropbox, League of Legends. So did New Jersey Transit, New York City's emergency management systems, and France's national rail operator, SNCF. Cloudflare confirmed quickly, and SecurityWeek covered explicitly, that this was not an attack. Nobody broke in. Nobody stole anything. A permissions change that was supposed to make things more secure had a side effect nobody caught, because the file it produced was treated as trusted the moment it left Cloudflare's own database.
Why Internal Config Gets a Free Pass
Most organizations draw a hard line between two kinds of data flowing through their systems: input from the outside world, which gets sanitized, validated, and treated with suspicion by default, and configuration generated by their own internal tooling, which gets treated as ground truth. The logic is not unreasonable on its face. Your own database query, run by your own automated pipeline, using your own schema, feels categorically different from a file uploaded by an anonymous user. Why would you need to defend against yourself?
The answer, as Cloudflare's own engineering team now states plainly in their postmortem, is that "yourself" is not a single, static thing. It is a permissions system that gets changed for good reasons. It is a query that references a table name that can, under conditions nobody anticipated, resolve to more than one database. It is a downstream consumer with a hardcoded assumption, written years earlier by someone who had no way to know that assumption would ever be tested. Internal systems drift, get modified, and interact in ways their original designers did not model. A file that has always been safe because it has always looked a certain way is not safe because someone guarantees it will always look that way. It is safe only until the day it does not, and nothing was watching for that day.
Why This Is Not a Hack
It is worth dwelling on the fact that this was not a cyberattack, because the instinct after any major outage is to ask "who did this to us." Cloudflare answered that question directly and quickly: nobody did this to them. This was change management, not an adversary. A team made a permissions change they believed would improve security. That change had a consequence three steps removed from the change itself, in a system that had no gate checking the consequence before it reached production.
This distinction matters because it means the standard security playbook, firewalls, intrusion detection, threat intelligence, none of it was ever going to catch this. There was no intrusion to detect. There was no malicious payload to scan for. There was a piece of Cloudflare's own software, doing exactly what it was built to do, acting on data that nobody had validated before trusting it. This is the failure mode that a change-validation discipline exists to catch, and it is the failure mode that most organizations, like Cloudflare before November 18, have not built a gate for.
One security-hardening change, three steps removed from its consequence, cascading to every machine on a global network with no gate in between.
Why This Matters to You
You do not need to run infrastructure at Cloudflare's scale for this to apply to you. Every organization has "trusted" internal config: feature flags, permission tables, generated manifests, cached lookup files, pipeline outputs that downstream systems consume without a second look because they came from inside the house. If none of those paths have a validation gate, you have the same exposure Cloudflare had, just with a smaller blast radius.
Roughly 20% of the web runs through Cloudflare in some capacity. When its core proxy went down, the outage was not contained to Cloudflare's own products; it took a visible slice of the internet with it, from social media to public transit systems. Scale changes the size of the story. It does not change the mechanism, and the mechanism is available in every environment that trusts its own output without checking it.
Coming up in Part 2 — the five ordinary steps that turned a security-hardening change into a global outage.
Part 2 of 4
Five Steps From Routine to Global
What you missed: Cloudflare's own internally-generated config file was never validated the way external input would have been, and this was change management gone wrong, not an attack.
What Happened: Five Ordinary Steps, One Global Outage
None of the five steps below is, by itself, unusual or reckless. That is what makes this incident worth studying closely. This is not a story about a careless engineer or a skipped code review. It is a story about five individually reasonable decisions and defaults that, stacked together with no validation gate between any of them, produced a five-and-a-half-hour global outage.
The Permissions Change
The everyday analogy: A building manager tightens who can access a storage room, for good reason. Nobody checks whether the room's inventory list is still accurate afterward, because the change was about the lock, not the list.
At 11:05 UTC, Cloudflare deployed a routine permissions change to a ClickHouse database cluster, intended purely as a security hardening measure. Nothing about the change targeted or touched the Bot Management system directly. It was the kind of change that, in most organizations, would not trigger a second look from the teams whose systems depend on that database, because it was a permissions update, not a schema change or a data migration.
Impact: A change with no apparent connection to production traffic serving was deployed without triggering validation against the systems that actually depended on the data it touched.
The Query Lost Its Filter
The everyday analogy: A filing clerk who used to pull records from one specific drawer now, after the lock change, can see two drawers labeled almost the same way. Nobody told the clerk to only pull from one, so both go in the folder.
Cloudflare's Bot Management system relies on a periodically regenerated "feature file," a machine-learning input file normally listing around 60 features used to score incoming traffic for bot-like behavior. The query that builds this file was scoped to a specific database name. After the permissions change, that scoping broke down: the query began pulling duplicate rows from both the `default` and `r0` databases, rather than the one it was meant to read from alone.
Impact: A silent loss of database-name scoping caused the same underlying data to be read twice, with no error thrown, no alert fired, and no indication that anything was wrong at the source.
Past the Hardcoded Limit
The everyday analogy: A shipping crate is built to hold 200 items and has never once held more, so nobody built a sensor to check the count. The one time 400 items arrive, the crate does not know to reject the extra 200. It just gets loaded onto the truck.
The duplicated rows silently doubled the feature file's size, pushing it past a hardcoded limit of 200 features baked into the downstream software that reads the file. That limit had never been tested against a file that actually reached it, because under normal conditions the file never grew anywhere near that size. The file was not rejected. It was not flagged as malformed. It simply became a file the reading software was never built to handle, and it kept moving through the pipeline as if nothing had changed.
Impact: A hardcoded assumption, safe for years, became a landmine the moment the data feeding it changed shape. Nothing in the pipeline was watching for that boundary being crossed.
Automatic Global Propagation
The everyday analogy: A company memo goes out to every office worldwide the instant it is finalized, by design, because speed is the whole point of the system that distributes it. Nobody proofreads company memos before they go out. This one had a typo that broke every printer that tried to read it.
Cloudflare's infrastructure is built to distribute updated configuration to every machine on its global network automatically and quickly. That speed is a genuine strength for legitimate updates; it is also exactly why the malformed feature file did not stay contained to one data center or one region. Because the file was generated internally rather than submitted by a user or a customer, it did not pass through the kind of validation, staging, or canary rollout that Cloudflare applies to other categories of change. It went everywhere, all at once.
Impact: The same automation that makes legitimate updates fast made a malformed update just as fast. Global propagation with no staged rollout turned a local data error into a worldwide event in minutes.
The Core Proxy Crashed on Ingest
The everyday analogy: A single malfunctioning sensor on a factory floor is supposed to be a minor, isolated problem. It becomes a plant-wide shutdown only if someone wired that sensor directly into the main power switch instead of a breaker built to fail safely on its own.
Bot Management is not a peripheral feature bolted onto Cloudflare's network; the software that reads the feature file runs inline, inside the core proxy that serves all of Cloudflare's traffic. When that software encountered the oversized, malformed file and crashed, it did not just disable bot scoring. It crashed the process handling the traffic itself. Every request that depended on that core proxy path, across every product built on top of it, failed at the same time, in every region the file had already reached.
Impact: A non-critical subsystem crashing took the entire critical traffic-serving path down with it, because nothing isolated the two. The blast radius of a bot-scoring bug became the blast radius of the whole network.
How It Works: What Cloudflare Expected vs. What Actually Happened
Every step in this chain made sense from where the people responsible for it were standing. The database team saw a permissions hardening change with no obvious link to traffic serving. The Bot Management system saw a feature file that looked, at the byte level, like every other feature file it had ever consumed, just bigger. The distribution system saw a routine internal config push, the exact category of update it exists to move quickly. None of them saw the whole chain, because no single gate existed that was responsible for checking the file against what a safe file should actually look like before it reached production.
Same File, Two Different Realities
A feature file, roughly 60 features, generated from one database, read once by every downstream machine.
The shape the file had always had, so nothing checked whether it still matched that shape.
A feature file with duplicated rows from two databases, past the 200-feature limit the reader had never been tested against, propagated globally within minutes.
No validation step existed to compare the arriving file against the expected shape before trusting it.
Cloudflare's own postmortem states the fix directly: harden ingestion of Cloudflare-generated configuration files the same way you would for user-generated input. The file was never untrustworthy by nature. It became untrustworthy the moment nobody checked it, and nothing downstream was built to notice.
Coming up in Part 3 — what nearly six hours actually cost, and why this keeps happening industry-wide.
Part 3 of 4
What Nearly Six Hours Cost
What you missed: five ordinary steps, a permissions change, a lost query filter, a breached hardcoded limit, automatic global propagation, and a core-proxy crash, stacked with no gate between any of them.
By The Numbers
5h 38m
From First Impact to Full Resolution
200
Hardcoded Feature Limit the File Doubled Past
~20%
Of the Web Runs Through Cloudflare
0
Attackers Involved, Confirmed by Cloudflare
Financial Impact
Economic estimates vary sharply by methodology and should be read with that caveat: a mid-range analysis put lost revenue in the roughly $180-360 million range; a broader estimate ran past $1.5 billion; at least one much higher outlier figure has circulated citing losses as high as $60 billion, but that number comes from a low-rigor source and should not be treated as authoritative. What is not in dispute: roughly 20% of the web runs through Cloudflare in some capacity, and for nearly 6 hours a large share of the internet - commercial, social, and civic - went dark at the same moment, from a single unvalidated file.
Risk Severity Analysis
Cloudflare's incident is not one failure but a stack of them, each survivable alone and catastrophic together. The following maps the individual gaps to their business risk.
| Gap | Severity | Business Risk |
|---|---|---|
| Internally-generated config trusted without validation | Critical | Any malformed artifact your own systems produce can reach production with zero scrutiny, because "we made it ourselves" was treated as a safety guarantee. |
| Non-critical subsystem sharing the critical path | Critical | A bug in bot scoring should degrade bot scoring, not the entire traffic-serving proxy. No fault isolation meant the smallest failure had the largest possible blast radius. |
| No staged or canary rollout for internal config | Critical | Automatic, immediate, global propagation is a strength for good updates and a catastrophe for bad ones. Speed with no gate is not resilience, it is exposure at machine speed. |
| No global kill switch for a misbehaving subsystem | High | Without a way to instantly disable one failing component network-wide, recovery depends on diagnosing and fixing the root cause under live pressure, not on containing the damage first. |
| Single-vendor concentration across the wider internet | High | With roughly a fifth of the web behind one provider, an internal change-management failure at that provider becomes a systemic event for every customer downstream, none of whom made the mistake themselves. |
Why This Keeps Happening: The Line We Draw Between "Ours" and "Theirs"
Cloudflare is, by any reasonable measure, one of the more sophisticated infrastructure operators on the internet. This was not a company with an immature engineering culture. It was a company that, like nearly every other organization running production systems, drew a line between validated input and trusted internal output, and never revisited whether that line still made sense as its systems grew more interconnected and more automated.
That line exists everywhere for a practical reason: validating everything, all the time, is expensive, and most internal config genuinely does not need it, most of the time. The problem is that "most of the time" is exactly the trap. A control that only needs to work once in a while, on the one time the internal system produces something it has never produced before, is a control organizations rarely build, because it rarely seems necessary until the day it was necessary and nobody built it.
The second driver is architectural: speed and blast-radius containment are often in tension, and speed usually wins the design argument, because speed is visible in normal operation and blast-radius containment only pays off on the rare day something goes wrong. Cloudflare's global propagation system is fast because being fast is almost always correct. The one day it was not correct, nothing in the design stopped the fast path from being exactly as fast for a broken file as for a good one.
Coming up in Part 4 — six practical steps to close this gap, plus a governance checklist.
Part 4 of 4
Closing the Gap
What you missed: nearly six hours down, a stack of individually reasonable defaults, and a line between "validated" and "trusted internal" that nobody had revisited.
What You Can Do: Six Practical Steps
Cloudflare's own remediation plan, published in its postmortem, is a reasonable starting list: harden ingestion of internally-generated files, add more global kill switches, improve fault isolation. Here is that same discipline generalized into six steps any organization running production infrastructure can apply, regardless of scale.
Recovery meant rolling back to a validated known-good version of the feature file. The step worth taking is validating the file before it ships, not only after it fails.
Validate what your own systems produce, not just what arrives from outside
"Internally generated" is not a synonym for "safe." Any file, manifest, or config artifact your own pipelines produce should pass through the same class of check applied to third-party input before anything downstream treats it as trustworthy.
In practice, this means defining what a valid artifact of each type looks like, shape, size, expected ranges, and rejecting anything that falls outside that definition automatically, rather than assuming the source can be trusted because it is internal.
Build a bounded, safe failure state, not a crash
A file that exceeds an expected limit should be rejected and fall back to the last known-good version, not consumed anyway by software that was never tested against that condition.
Every hardcoded assumption in a production system is a boundary that will eventually be tested. Treat exceeding it as an expected, handled case, not an unhandled crash.
Isolate non-critical subsystems from the critical path
A bot-scoring feature should never be able to crash the traffic-serving proxy it runs alongside. Fault isolation means a failure in one subsystem degrades that subsystem, not everything sharing its process space.
Ask, for every feature bolted onto a critical system: if this component fails completely, right now, does the critical system keep serving traffic? If the answer is no, that is an architecture gap, not a bug to patch later.
Build and rehearse a global kill switch
You need the ability to instantly disable a misbehaving subsystem network-wide, independent of diagnosing or fixing the underlying cause. Containment and root-causing are two different jobs and should not be blocked on each other.
A kill switch that has never been exercised outside a real incident is a kill switch you are not sure works. Rehearse it on a schedule, the same way you would rehearse a disaster-recovery failover.
Stage and canary config changes the way you stage code deploys
"It's just a config file" is not a reason to skip the staged rollout discipline applied to code. Push internally-generated config to a small percentage of the fleet first, verify behavior, then expand.
A canary deployment would have surfaced this failure on a handful of machines, not the entire global network, and turned a five-and-a-half-hour worldwide outage into a contained, quickly-reverted incident.
Keep a validated known-good artifact ready for instant rollback
Cloudflare's actual recovery step was reverting to a known-good version of the feature file. That path should be pre-validated and ready before an incident, not assembled under pressure during one.
A known-good baseline that you have to reconstruct during an outage costs you the minutes, or hours, between "we know what broke" and "we have restored what worked."
Governance Checklist
Does your change-management process include these controls?
Most organizations currently lack the controls marked with ✗. Cloudflare's own postmortem commits to closing several of them; the gap it describes is common across the industry, not unique to one vendor.
AuthorityGate Governance Framework
AuthorityGate's 8-gate model exists precisely for changes like this one. Gate 1 (Pre-Validation & Backup) requires a known-good baseline and a usable recovery path before any change proceeds, internal or external. Gate 5 (Dependency Health) would have flagged a feature file that no longer matched the shape its downstream consumer expected. Gate 6 (Behavioral Resilience) tests a change against a representative lower environment before it reaches production, exactly the step that would have caught this file crashing its reader before it ever touched live traffic. Gate 8 (Recovery Readiness) requires a tested rollback path be proven, not assembled under pressure.
The framework treats every change, human, pipeline, vendor update, or internally-generated artifact, as subject to the same validation discipline. Nothing gets a pass for coming from inside the house.
The Bottom Line
Cloudflare's November 18 outage is valuable precisely because nothing about it was exotic. No attacker, no zero-day, no novel technique. A permissions change had a side effect three steps removed from itself, and nothing was watching for that side effect because the file it produced came from inside Cloudflare's own systems. That is the most common failure mode in production infrastructure, and it is the one most organizations are least prepared to catch, because "trusted internal source" quietly substitutes for "validated" in almost every environment.
Cloudflare's own words are the clearest summary available: the fix is hardening ingestion of internally-generated configuration the same way you would for untrusted input. That is not a Cloudflare-specific lesson. It is the argument for validation-before-production stated by one of the most capable infrastructure teams in the industry, after their own internal trust assumption cost them nearly six hours and a meaningful share of the internet.
The organizations that take this seriously will stop asking only "is this input from outside our walls," and start asking "has this artifact actually been validated," regardless of where it came from. The organizations that do not will find out, the way Cloudflare did, that the file they trusted the most was the one nobody checked.
This article is part of our incident analysis newsletter series. Subscribe to receive complete analyses with timeline tables, risk matrices, governance checklists, and actionable recommendations.
