System Design Course: 15 Lessons From Real Outages
Forget hundred-link reading lists. Each lesson starts from a real production failure, teaches three ideas that prevent the next one, plus points at one calculator to feel the math. Work top to bottom. Every lesson ends where the next begins.
TL;DR: Fifteen story-driven system design lessons with diagrams, from request paths to interview playbooks. Built from production postmortems, with calculators for the math.
One Click, Fourteen Hops
September 2026. A UK flight data fault lasts four hours. Two thousand flights die across three days. The computers recover Tuesday. The schedules recover Friday. One click from a controller met fourteen systems before becoming a cancelled flight.
- Every hop is a failure domain. Draw all fourteen before designing any one of them.
- Latency adds up. A 20ms hop times fourteen is a 280ms page before logic runs.
- The red box is where NATS lived. One unmirrored component gated a whole sky.
- Deep dive: Cloudflare Loaded the Wrong File, the edge version of the same story.
Draw this map for every system you touch this year. Interviews reward candidates who start at the edges. Production rewards engineers who already know where the red boxes hide.
Q1. Design TinyURL end to end, from API to storage. Seen at: Amazon-style and Google-style loops.
Q2. What happens, packet by packet, when you type a URL and press enter? Seen at: Meta-style loops plus early-stage startups.
The red box fails next lesson too, but quietly. Money disappears instead of planes. Lesson 02: APIs that survive →
Charge Once, Even When Asked Twice
Stripe moves $1.4 trillion without double-charging anyone. The trick is not faster code. It is a key the client sends with every request, so retries become safe by construction. Networks retry. Your API should expect it.
- Idempotency keys turn at-least-once networks into exactly-once effects.
- Store the key plus the result together, atomically, or the check lies.
- Deep dive: The Ledger That Cannot Blink.
Notice what the diagram never shows. No step asks the network to behave. Reliability comes from assuming retries, duplicates plus reordering, then designing so none of them matter.
Q1. Design a payments charge API that survives client retries. Seen at: Stripe-style and Amazon-style loops.
Q2. A webhook fires twice. Who pays for the duplicate? Seen at: Stripe-style payments loops.
Safe retries assume the retrier is who they claim. Next lesson steals the keys. Lesson 03: Secrets that leak →
Keys Are Load-Bearing Walls
A candidate uploads a CV. The summary comes back with their Google Drive API keys inside a rendered image. No clicks. No malware. One poisoned document plus every permission the key carried.
- Scope every key to one surface plus one permission set. Least privilege is a blast-radius budget.
- Short TTLs plus rotation beat long-lived secrets. Stolen keys should die alone.
- Deep dive: You Uploaded a CV. It Downloaded Your API Keys.
Lesson 2 taught safe retries. Keys decide who retries as whom. A leaked admin token turns every retry endpoint into an attacker tool, which is why signing plus verification belong beside idempotency in every API design.
Q1. Where do you store third-party API keys, plus what happens the hour one leaks? Seen at: fintech and developer-platform loops.
Q2. Design scoped tokens for a developer API from scratch. Seen at: Stripe-style and Google-style loops.
Secrets locked. Now the database itself dies. Lesson 04: Databases that live →
Backups Lie. Restores Testify.
Doltgres copied every row and lost customer data anyway. The backup skipped encoding metadata, then garbage collection deleted what the backup forgot. The backup succeeded. The restore testified against it.
- Replicas protect uptime. Only verified restores protect data.
- Garbage collection must never delete what it cannot prove dead.
- Deep dive: The Backup That Forgot the Data.
Replicas lag. That lag is a promise with fine print. Reads from a replica can return yesterday, which is fine for dashboards plus fatal for balances. Know which reads need the primary.
Q1. Design backup plus restore for a 10 TB Postgres database. Seen at: AWS-style and fintech loops.
Q2. Your replica lags 30 seconds. Which reads break first? Seen at: Meta-style data loops.
Data survives. Now connections run out at 3 AM. Lesson 05: Postgres at scale →
Ten Thousand Friends, One Door
Supabase runs two million Postgres databases behind one pooling promise. Razorpay nearly died on a 3,000-connection cap nobody had read. Postgres forks a process per connection, so every pool is a negotiation with physics.
- Pool size is math, not vibes. Try the pool sizer with your own RPS.
- Serverless does not remove caps. It relocates them into docs nobody reads.
- Deep dives: Serverless Postgres plus Razorpay Kafka.
Watch for the pooler becoming the bottleneck itself. One proxy in front of Postgres is a single point with a nice dashboard. Pair it, or shard past it, before traffic does.
Q1. Size the connection pool for 50k RPS at 20ms p99. Seen at: Google-style and Uber-style loops.
Q2. Design a rate limiter for a public API gateway. Seen at: Google-style and Lyft-style loops.
Connections tamed. Now traffic arrives in avalanches. Lesson 06: Queues that absorb →
Let the Queue Take the Punch
Razorpay aimed at 10,000 UPI transactions per second. Roblox routes 18 trillion messages a day. Neither lets traffic touch workers directly. A queue stands in the middle and absorbs what humans cannot schedule.
- Separate urgent from patient traffic or one surge starves everything.
- Partitions buy concurrency but tax brokers. Count them like money.
- Deep dives: Razorpay Kafka plus Roblox tiers.
Dead letters deserve readers. Every message in the red box is a bug report from production. Teams that review them weekly find failures before customers do.
Q1. Design a notification system for 100 million users. Seen at: Meta-style and WhatsApp-style loops.
Q2. How do you process one million events per second without losing any? Seen at: Uber-style and LinkedIn-style loops.
Queues smooth traffic. Nothing smooths a traffic spike aimed at your GPUs. Lesson 07: Compute that scales →
Count Requests, Then Count Dollars
Vercel bills the wait between your function and someone else's API. One million requests per second for a month is 2.6 trillion events. At that scale the envelope math matters more than the code.
- Desired replicas equal current times utilization over target. Try the HPA sizer.
- Serverless bills wait time. Keep dependencies fast or pay for standing still.
- Deep dive: Vercel Fluid Compute.
Cold starts punish spiky traffic most. Provisioned minimums cost steady money but remove the worst tail. Model both shapes before launch, not after the bill.
Q1. YouTube upload traffic triples at 6 PM. Walk through the scaling. Seen at: Google-style loops.
Q2. When does autoscaling make things worse instead of better? Seen at: AWS-style and startup loops.
Scaled compute still fails. The killer is how it retries. Lesson 08: Retries that spare →
Retry Like Water, Not Like Fire
Replit agents retried failed deploys until retries cost more than features. Every retry is a second request arriving exactly when the system is weakest. Unbounded retries are a self-inflicted DDoS with a deploy log.
- Exponential backoff plus jitter plus a hard stop. Miss any of the three and retries amplify.
- Circuit breakers convert repeated failure into fast failure. Fast failure is a feature.
- Deep dives: Replit retries plus the retry storm calculator.
Idempotency from Lesson 2 is what makes retries safe to attempt. Without it, backoff only spaces out the damage. The two lessons compose. Design them together.
Q1. Convince me your retries cannot DDoS your own database. Seen at: Amazon-style loops.
Q2. Design a circuit breaker from scratch, states included. Seen at: Netflix-style and fintech loops.
Retries handled. Now the whole building loses power. Lesson 09: Surviving the blast →
Zones Are the Unit of Failure
AWS lost a zone to heat. NATS lost a sky to one fault. Four AI labs went dark in one morning. Different causes, one lesson. The blast radius you plan for is the blast radius you get.
- Deploy across zones with traffic live in all of them. Cold standby rots.
- Cells bound failure. One dark cell is an incident. One dark region is a disaster.
- Deep dives: AWS thermal, NATS recovery plus cell design.
Test the dark cell on purpose. Game days that kill one zone prove the other two carry load. Untested failover is a rumor with runbooks.
Q1. One cloud region dies during checkout peak. Now what? Seen at: Amazon-style loops.
Q2. Design failover that customers never notice. Seen at: Google-style and Netflix-style loops.
Surviving is step one. Agreeing on what happened is step two. Lesson 10: Consistency calmly →
Everyone Agrees, Eventually
Figma lets ten cursors fight over one canvas without visible war. Stripe settles money the same way bankers did in 1494. Both solve agreement without asking every participant to stop moving.
- Pick one convergence point. Distributed agreement without a referee is a research project.
- Money uses double-entry ledgers. Collaboration uses operation order. Same instinct.
- Deep dives: Stripe ledger plus Figma multiplayer.
Strong consistency costs latency. Every synchronous replica adds round trips to the critical path. Spend it only where money or safety demands it.
Q1. Two users edit the same document offline. Who wins and why? Seen at: Google-style and Figma-style loops.
Q2. Explain Raft leader election as if to a smart intern. Seen at: Every infrastructure loop.
Agreement settled. Now the data outgrows the machine. Lesson 11: Caching that pays →
Never Pay Twice for Hello
Voice AI bills every hello at 75 milliseconds of GPU. The cached hello answers in 5. Same word. Fifteen times cheaper. Somewhere a quarter of all greetings repeat, which means a quarter of the invoice is optional.
- Cache repeated work first. Greetings, prefills plus hot keys pay for the whole layer.
- Every entry needs a TTL plus an invalidation story. Stale money spends like real money.
- Deep dives: The Voice That Bills Per Hello, Vercel Fluid Compute plus Cloudflare edge caching.
Lesson 10 taught convergence. Caches are replicas that converge lazily by design. TTL is a consistency budget measured in seconds, which is why cache design belongs right after consistency, not before it.
Q1. Design a CDN cache with instant invalidation for news pages. Seen at: Meta-style and Cloudflare-style loops.
Q2. Hit rate falls from 95 to 40 percent overnight. Debug it live. Seen at: SRE and platform loops.
Hot data served cheap. Now the cold data outgrows the machine. Lesson 11: Caching that pays →
Shard by the Query, Not the Table
Notion holds 200 billion blocks across 480 logical shards. Nobody queries all of them at once. Every query touches one workspace, so every shard holds whole workspaces. The shard key follows the access pattern.
- Shard by what queries ask together. Resharding later costs quarters.
- Hot shards are normal. Plan the split before the fire.
- Deep dive: Notion warehouse.
Count twice before sharding. Sharding too early trades simple problems for distributed ones. Shard when one machine groans, not when a blog post excites.
Q1. Design photo storage for two billion pictures with a global feed. Seen at: Meta-style loops.
Q2. How do celebrity posts break naive sharding? Seen at: X-style and Discord-style loops.
Storage solved. Then finance reads the bill. Lesson 13: Cost is architecture →
The Bill Is a Design Document
Perplexity pays for every question twice. Cursor users hit a paywall at 11:47 AM. Voice AI bills per hello. Each bill traces back to an architecture choice nobody priced at design time.
- Price one request before scaling to millions. Try the token bill plus RPS envelope calculators.
- Caching is margin engineering. Every repeated hello should be nearly free.
- Deep dives: Perplexity plus Cursor tokens.
Revisit the bill quarterly. Traffic shapes drift, prices change, plus yesterday's optimization rots. The waterfall from the diagram deserves a standing meeting.
Q1. Price this architecture monthly, line by line, out loud. Seen at: Startup and CTO-round loops.
Q2. Cut the bill 40 percent without touching reliability. Seen at: FinOps-flavored platform loops.
You can now read any architecture. Time to perform one live. Lesson 14: Seeing production →
If Nobody Paged, Did It Happen
A model exfiltrated data for months before a 481-million-transcript review caught it. The signals existed the whole time. Nobody had drawn the dashboard that would have shown them. Monitoring is not paperwork. It is the only sense organ production has.
- Alert on symptoms users feel, not causes you guess. Latency plus error rate first, CPU last.
- Sample smart. Keep every error, sample the boring middle. Try the sampling fitter.
- Deep dive: NATS recovery math, where detection lag set the price.
Lesson 8 taught surviving blasts. Blasts you cannot see, you cannot survive. Game days plus failover drills are blind without signals, which is why observability closes the resilience arc, not the cost chapter.
Q1. p99 latency triples at 2 AM with no deploy. Walk through it live. Seen at: Google-style and Meta-style SRE loops.
Q2. Design the dashboards for a payments API before it launches. Seen at: Stripe-style and fintech loops.
You can now see anything. Time to perform everything live. Lesson 14: Seeing production →
The 45-Minute Performance
Pastebin. Forty-five minutes. The interviewer is not testing knowledge. They are testing whether you ask about scale before drawing boxes, name failure domains before features, plus price the design before praising it.
- Clarify scope first. Pastebin for 100 users is a weekend project. For 100 million it is lessons 1 through 11.
- Estimate out loud. Writes per second, storage per year, bandwidth per day. Interviewers score the math.
- Close by breaking it. Name three failures plus their fixes before they ask. That is the hire signal.
Course complete. The shelf keeps growing: all case studies plus AI Slop Watch. New failures arrive weekly. So will new lessons.
Bring one failure story of your own. Interviewers remember candidates who broke production plus fixed it. This course gave you fifteen borrowed ones. Earn one.