Listen to this article
Some postmortems age into required reading, and Datadog's writeup of its March 8, 2023 outage is one of them. Two and a half years later, it is worth rereading for one reason: the company that watches everyone else's infrastructure - the premier observability vendor, the people you page when your systems misbehave - was taken down for roughly two days by an operating system patch that nobody approved. Not a bad deploy. Not a botched migration. A routine security update that applied itself.
At 06:00 UTC that morning, a security update to systemd was automatically applied to Datadog's virtual machines through a legacy OS update channel. On restart, systemd-networkd deleted the network routes managed by the Cilium CNI plugin, and the nodes simply fell off the network. Detection came fast - 06:03 UTC. It did not matter. Between 06:00 and 07:00 UTC, tens of thousands of nodes went offline.
Services were declared operational March 9 at 08:58 UTC; full resolution, including backfilled data, came March 10 at 06:25 UTC - roughly 48 to 50 hours end to end.
Multi-cloud was supposed to prevent exactly this
Here is the detail that makes this postmortem canonical. Datadog did everything the resilience playbook prescribes: five regions - US1, EU1, US3, US4, US5 - spread across three different cloud providers. And all five went down simultaneously, because the same legacy update channel was enabled everywhere, and the same patch rode the same automation into every region at once. The redundancy was real. It just was not redundancy against this.
Multi-region did not help. Multi-cloud did not help. An unvalidated change does not respect availability zones - it ships to all of them on schedule.
The independent analysis by The Pragmatic Engineer put numbers on the blast radius: roughly half of Datadog's Kubernetes fleet knocked offline, and an estimated $5 million in customer credits - the estimate is Pragmatic Engineer's, not Datadog's. Whatever the exact figure, the shape of the incident is undisputed and comes from Datadog's own account: one unsupervised update path, global simultaneous impact, two days of recovery.
The remediation list is a confession
Datadog's fixes, in its own postmortem: disable the legacy auto-update channel, change the systemd-networkd configuration so it preserves routing tables, and audit the fleet for other unsupervised update paths. That last item is the one to sit with. The remediation was not "patch less" or "patch slower." It was an admission that updates had a route into production that no process ever looked at - and a hunt for how many more such routes existed.
That is the honest condition of most infrastructure, then and now. Every fleet has update channels that predate the current team: OS packages, base images, agent auto-updaters, vendor daemons that phone home. Each one is a change pipeline into production with no gate on it. They stay invisible right up until 06:00 UTC.
The AuthorityGate take
The lesson of this outage is not that security patches are dangerous. Unpatched systemd is its own incident waiting to happen. The lesson is that "routine" is not a validation category. A systemd update is a change to production, and it deserves what every change to production deserves: validation against your environment - your CNI plugin, your network config, your restart behavior - before it applies, staged rollout so region two waits on evidence from region one, and a human decision on anything that touches the whole fleet. That is what we mean by update validation: every update, including the ones a vendor marks routine, passes the gates first.
Rereading this in late 2025, the stakes have only moved one direction. The industry is wiring more automation, more agents, and more auto-remediation into production than it had in 2023 - more things with standing permission to change systems on their own schedule. Each one is a legacy update channel in the making, and the fix is the same: no change applies itself, no matter how trusted its source. Anything less, and your continuity plan is one unattended patch away from being a postmortem.
Datadog's engineers recovered a global platform in two days and published an unusually candid account of why they had to. The postmortem's real gift is the question it forces on every reader: how many update paths into your production environment could fire tonight at 06:00 UTC without a single person approving them? If the honest answer is "we would have to audit to find out," that is the same answer Datadog had on March 7.
Sources
Go deeper
Every agent action, validated before it takes effect
AuthorityGate's newsletter breaks down real AI incidents and the governance failures behind them. Our configurable 8-gate validation model is how organizations keep a named human accountable for what their AI actually does.