Risk register · entry
Q3 · EngineeredMeta global outage
One backbone command erased Facebook from the internet worldwide.
Tightly coupled systems where one small fault cascades and takes down the whole machine.
Why this quadrant
One command withdrew the BGP routes to Facebook's data centres, and the audit tool built to catch exactly that command had a bug. Because the backbone, DNS and building access all ran over the same network, a routing mistake became a global blackout in under twenty minutes. The hardening is what made it last six hours. Engineers could not get into the buildings, and the DNS servers had withdrawn themselves.
The record
- Outage window: 15:39 UTC to roughly 21:28 UTC on Oct 4, 2021, about 6 hourscertain
- Facebook stock closed down about 4.8-5% that Monday, wiping roughly $47 billion in market valuecertain
- Mark Zuckerberg's net worth fell by about $5.9-7 billion that daylikely
- Fortune estimate: Facebook lost roughly $65-100 million in ad revenue during the outageuncertain
- Standard Media Index conservative estimate: about $3.3 million in lost ad spenduncertain
- NetBlocks estimate: roughly $1 billion lost to the global economy over the six hoursuncertain
- Cloudflare's 1.1.1.1 resolver saw DNS query traffic spike about 30x above normal during the outagecertain
- India had an estimated 15 million or more small businesses using WhatsApp Business at the time, out of 390 million WhatsApp users in-marketlikely
- Root cause per Meta: a routine backbone-capacity audit command wrongly withdrew all BGP routes to Facebook's data centers; a bug in the audit tool that should have blocked the command failed to catch itcertain
Sources
The newsletter
One risk story a week, taken apart the way this one was: what was known, what was ignored, and which quadrant it really belonged to.
No spam, one email a week.