How-tos

Learning Systems Engineering From Other People's Outages

31 October 2026 · 7 min

Somebody else already paid for this lesson. Run the lab instead.

Most of what makes a production system reliable was learned the expensive way: something broke, at scale, in front of customers, and a team wrote down what they would never do again. You don't have to repeat those incidents to learn from them. You can read what other teams concluded, then rebuild the defence yourself on a laptop until it makes sense.

That is the idea behind modern-systems-engineering, a curriculum I've been putting together. It pairs short case-study notes on production systems from companies like AWS, Cloudflare, Stripe, Discord, Netflix and Uber with hands-on labs that have starter code and a reference solution. This post walks through how to use it, with the reliability lab as the worked example.

How the repo is laid out

There are seven courses plus a capstone:

1. Engineering foundations (APIs, reliability, version control)
2. Distributed systems (a gRPC chat system and a Kafka pipeline)
3. Cloud infrastructure (a Kubernetes deployment)
4. AI systems (a small retrieval-augmented assistant)
5. Real-time systems (WebSocket streaming with backpressure)
6. Financial systems (a double-entry payment ledger)
7. Advanced system design (five interview-style exercises)

Each course folder has a syllabus.md, lecture notes where they exist, case studies, and labs. Every lab has a README.md, a starter-code/ folder with the parts left for you to write, and a solution/ folder to compare against afterwards.

The loop the repo suggests is simple: read the case study, do the lab without peeking, then compare with the solution and write a short note on what was different.

Step 1: Read a case study for its failure lessons

The case studies are short. Each one has the same shape: an overview, the key challenges, architecture highlights, the tech stack, and a Lessons Learned list. That last section is the part to read slowly.

Take the AWS cell architecture note. Its core idea is the blast radius: split a big service into independent cells, so that when one cell fails it takes down only its own share of customers, not everyone. The note's lessons are about keeping cells isolated, mostly through asynchronous messaging rather than chains of synchronous calls between them.

Or the Cloudflare DDoS note: spread attack traffic across many locations, filter it as early as possible, and layer defences, because no single layer catches everything.

Neither lesson is about a specific feature. Both are about what happens when something goes wrong and how far the damage spreads. That is the habit to build: when you read about any system, ask how it fails and who notices.

Step 2: Feel the failure in a lab

Reading "use retries with backoff and a circuit breaker" is easy. Seeing what those do to your numbers is what makes it stick. The reliability toolkit lab in course 01 gives you a fake downstream service that misbehaves:

def flaky_dependency() -> str:
    if random.random() < 0.25:
        raise RuntimeError("downstream_error")
    if random.random() < 0.05:
        time.sleep(0.25)
    time.sleep(0.01)
    return "ok"

A quarter of calls fail outright, and about one in twenty of the rest is slow. That is a fair picture of a dependency having a bad day.

Retries with exponential backoff and jitter

The first defence is to try again, but not immediately and not all at once:

def _backoff_delay_s(*, attempt, base_delay_s, max_delay_s):
    exp = base_delay_s * (2 ** (attempt - 1))
    capped = min(exp, max_delay_s)
    jitter = capped * (0.5 + random.random())
    return jitter

The wait doubles with each attempt, is capped at a maximum, and is then multiplied by a random factor between 0.5 and 1.5. That random part, the jitter, matters more than it looks. Without it, a thousand clients that failed at the same moment would all retry at the same moment too, and hit the struggling service in a synchronised wave.

A circuit breaker

Retries help when failures are occasional. When a dependency is properly down, retrying just piles more load on it and makes your own callers wait. A circuit breaker stops that. The lab's breaker has three states:

• Closed: calls go through normally, and consecutive failures are counted.
• Open: after failure_threshold failures in a row, calls fail immediately with circuit_open, without touching the dependency.
• Half-open: once recovery_timeout_s has passed, one trial call is let through. Success closes the breaker again; a failure opens it straight away.

What the numbers say

I ran the reference solution's simulate.py, which makes 200 calls in each of three setups (fixed random seed 7, three attempts, a breaker that trips after three failures and waits 0.5 s). This is what came out:

This table is the whole lesson in miniature. Retries took success from 70% to 98%, but the slowest calls got much slower, because a caller that fails twice spends real time waiting before its third try. Adding the breaker gave back some of that success rate, but cut the 95th-percentile latency by about two thirds, because while the circuit is open, callers fail fast instead of queueing behind a sick service.

Neither row is "correct". Which one you want depends on whether your users would rather wait or get a quick error they can handle. That is a design decision, and the point of the lab is that you now make it with numbers instead of instinct.

The lab's README suggests two experiments, and they are worth the ten minutes: raise max_attempts and watch the tail latency, then change failure_threshold and compare success against latency.

Step 3: Make retries safe

There is a catch in Step 2. The lab's own README says its retry logic is meant for idempotent operations: ones where doing them twice has the same effect as doing them once. Reading a record is idempotent. Charging a card is not.

The payment ledger lab in course 06 shows the standard fix. Every transfer carries an idempotency_key chosen by the caller. Before writing anything, the ledger checks whether it has already seen that key:

existing = conn.execute(
    "SELECT * FROM transactions WHERE idempotency_key = ?",
    (idempotency_key,),
).fetchone()
if existing is not None:
    conn.execute("COMMIT")
    return _row_to_tx(existing)

If it has, it returns the original transaction instead of creating a second one. So a client can retry a payment after a timeout without any risk of paying twice.

The same function also writes every transfer as two ledger entries, a debit and a matching credit, checks that they add up to zero, and records an audit-log entry, all inside one database transaction that rolls back on any error. Money is neither created nor lost, even when something fails halfway.

Put Steps 2 and 3 together and you have one of the most useful rules in the whole curriculum: retries make a system more available, and idempotency is what makes retries safe.

How to use it yourself

1. Pick a course from the roadmap. If you are new to this, start with course 01; the repo notes that course 02 is the foundation for the rest.
2. Read the case studies for that course, and write down the lessons-learned items in your own words.
3. Do the lab in starter-code/ first. Only then open solution/.
4. Change one parameter, rerun, and write down what moved.
5. When you're ready, try one of the four capstone templates (an AI knowledge platform, a global chat platform, a payment platform, or a feed-ranking system).

The case studies are compact summaries, not full incident reports, so treat them as a map. When a lesson catches your eye, follow it to the company's own engineering blog, and then come back and build the small version.

Other teams have already paid for these lessons with real outages. The least we can do is learn from them on a laptop, where a failure costs nothing but a rerun.