Risk register · entry
Q3 · EngineeredMeta global outage
One backbone command erased Facebook from the internet worldwide.
Tightly coupled systems where one small fault cascades and takes down the whole machine.
Why this room
The payoff structure is a simple, well-modeled engineering failure (a bad command, a buggy safeguard) but the tail behavior is anything but simple: because the backbone, DNS, and physical access systems were all tightly coupled, the failure jumped from one routing mistake to a total global blackout in under 20 minutes, and the same hardened design that limits everyday risk (secure data centers, self-protecting DNS) is what stretched recovery to six hours, the classic Q3 signature of an optimized system whose rare failures are fast, cascading, and self-defeating.
The record
- Outage window: 15:39 UTC to roughly 21:28 UTC on Oct 4, 2021, about 6 hourscertain
- Facebook stock closed down about 4.8-5% that Monday, wiping roughly $47 billion in market valuecertain
- Mark Zuckerberg's net worth fell by about $5.9-7 billion that daylikely
- Fortune estimate: Facebook lost roughly $65-100 million in ad revenue during the outageuncertain
- Standard Media Index conservative estimate: about $3.3 million in lost ad spenduncertain
- NetBlocks estimate: roughly $1 billion lost to the global economy over the six hoursuncertain
- Cloudflare's 1.1.1.1 resolver saw DNS query traffic spike about 30x above normal during the outagecertain
- India had an estimated 15 million or more small businesses using WhatsApp Business at the time, out of 390 million WhatsApp users in-marketlikely
- Root cause per Meta: a routine backbone-capacity audit command wrongly withdrew all BGP routes to Facebook's data centers; a bug in the audit tool that should have blocked the command failed to catch itcertain
Sources
The book
This entry is one of 111 in the register. The full story, and what it cost the people who lived it, is in Risky Business by Claudia Zeisberger, David Munro and Joanna Reijgersberg-Siew.
Join the waiting list