Risk register · entry
Q3 · EngineeredAWS US-EAST-1 outage
A routine scaling action hit a latent bug and took down half the internet.
Tightly coupled systems where one small fault cascades and takes down the whole machine.
Why this room
The trigger was a narrow, engineered payoff structure (one internal automation bug with a knowable, containable failure mode) but the tail behavior turned fat and correlated the instant it touched shared infrastructure, so a single-region software defect produced simultaneous, uninsured losses across unrelated industries worldwide.
The record
- Outage began 7:30am PST / 10:30am ET, December 7, 2021certain
- Internal DNS traffic migration completed at 9:28am PST, resolving DNS errorscertain
- Most core services (EC2 API errors, instance launches) recovered between 1:15pm and 2:40pm PSTcertain
- Full resolution (Amazon Connect, EventBridge) not until 4:41pm-6:40pm PST, roughly 7-9 hours total depending on servicecertain
- Companies affected include Netflix, Disney+, Coinbase, Ring, Amazon retail/warehouse operations, Roomba, Ticketmastercertain
- Amazon Flex and Dolphin scanner logins failed across warehouses in at least 14 named US states plus Ontario, Canadalikely
- A second US-EAST-1 disruption followed on December 10, 2021likely
- No official AWS-disclosed dollar cost estimate for the December 2021 event specifically (unlike the much larger, separately documented October 2025 AWS outage)uncertain
Sources
The book
This entry is one of 111 in the register. The full story, and what it cost the people who lived it, is in Risky Business by Claudia Zeisberger, David Munro and Joanna Reijgersberg-Siew.
Join the waiting list