Postmortem: The Checkout Outage
A blameless review of one bad afternoon: what happened, why, and what we will change.
Preview
Plays with its recorded voice-over.
🛠️
PostmortemBlameless
Postmortem: The Checkout Outage
A blameless review of one bad afternoon: what happened, why, and what we will change.
8 scenes · narrated · captions included
The JSON behind it
This is all it takes.
{
"version": 1,
"title": "Postmortem: The Checkout Outage",
"subtitle": "A blameless review of one bad afternoon: what happened, why, and what we will change.",
"topic": "A blameless postmortem of a production incident in an online shop",
"domain": "technology",
"emoji": "🛠️",
"kicker": "Engineering",
"chips": [
"Postmortem",
"Blameless"
],
"style": {
"theme": "midnight-tech",
"progress": "glow",
"header": [
"kicker",
"title"
]
},
"narration": {
"voice": "F1",
"speed": 0.97
},
"scenes": [
{
"kind": "intro",
"say": [
"Last Tuesday, customers could not pay for forty-seven minutes, between two oh two and two forty-nine in the afternoon.",
"Here is what happened, and what we are changing so it does not happen again."
]
},
{
"kind": "timeline",
"heading": "What happened, minute by minute",
"events": [
{
"label": "14:02",
"title": "Release goes out",
"text": "A new cart page is deployed.",
"icon": "🚀",
"tone": "violet",
"say": "At two oh two, a new version of the cart page was released."
},
{
"label": "14:09",
"title": "Errors climb",
"text": "Payment errors reach 40%.",
"icon": "📈",
"tone": "rose",
"say": "Seven minutes later, errors at the payment step reached forty percent."
},
{
"label": "14:21",
"title": "Team paged",
"text": "The on-call engineer joins.",
"icon": "📟",
"tone": "amber",
"say": "Only at two twenty-one did the on-call engineer get paged."
},
{
"label": "14:49",
"title": "Back to normal",
"text": "The release is rolled back.",
"icon": "✅",
"tone": "emerald",
"say": "And at two forty-nine the release was rolled back, and checkout recovered."
}
]
},
{
"kind": "motion",
"heading": "What the database felt",
"lead": "This picture shows the load on our database during the incident.",
"canvas": "wide",
"shapes": [
{
"id": "limit",
"type": "line",
"x": 10,
"y": 20,
"x2": 150,
"y2": 20,
"stroke": "amber",
"strokeWidth": 0.4,
"enter": "grow",
"at": 0.3
},
{
"id": "limitLabel",
"type": "text",
"text": "Database limit",
"x": 128,
"y": 15,
"size": 4,
"fill": "amber",
"enter": "fade",
"at": 0.6
},
{
"id": "deployLine",
"type": "line",
"x": 60,
"y": 12,
"x2": 60,
"y2": 52,
"stroke": "violet",
"strokeWidth": 0.35,
"enter": "grow",
"at": "line1"
},
{
"id": "deployLabel",
"type": "text",
"text": "Release 14:02",
"x": 60,
"y": 57,
"size": 3.6,
"fill": "violet",
"enter": "fade",
"at": "line1"
},
{
"id": "load",
"type": "path",
"d": "M 10 50 C 30 49, 45 48, 60 46 C 75 44, 80 30, 95 10 C 105 -2, 130 2, 150 4",
"stroke": "sky",
"strokeWidth": 0.9,
"enter": "draw",
"at": "line1",
"duration": 3
},
{
"id": "dot",
"type": "circle",
"x": 10,
"y": 50,
"size": 2.6,
"fill": "white",
"glow": true,
"enter": "pop",
"at": "line1"
},
{
"id": "crossing",
"type": "ring",
"x": 88,
"y": 20,
"size": 7,
"stroke": "rose",
"strokeWidth": 0.6,
"loop": "pulse",
"enter": "pop",
"at": "line2"
},
{
"id": "errors",
"type": "text",
"text": "Checkout errors",
"x": 122,
"y": 34,
"size": 4.5,
"fill": "rose",
"enter": "type",
"at": "line3"
}
],
"script": "tl.to(\"dot\", { motionPath: { path: \"load\" }, duration: 3, ease: \"none\" }, \"line1\");",
"lines": [
{
"say": "Right after the release, the load on our database began to climb."
},
{
"say": "Each checkout now asked the database a separate question for every item in the cart, so the load soon crossed its limit."
},
{
"say": "From that moment, every new payment ran into errors, until the release was rolled back."
}
],
"transition": "blur-zoom",
"header": false
},
{
"kind": "tree",
"heading": "Why it took so long",
"root": {
"title": "Checkout errors",
"icon": "🛒",
"text": "47 minutes"
},
"branches": [
{
"title": "Database overloaded",
"icon": "🗄️",
"tone": "rose",
"children": [
"A query for every item",
"No cap on connections"
],
"say": "First, the database was overloaded, because the new page made a query for every item and nothing capped the connections."
},
{
"title": "Slow to notice",
"icon": "🔔",
"tone": "amber",
"children": [
"Noisy alerts",
"No error-rate alarm"
],
"say": "Second, we were slow to notice. Our alerts were noisy, and none of them watched the error rate."
},
{
"title": "Slow to fix",
"icon": "🧰",
"tone": "sky",
"children": [
"Manual rollback",
"Unclear owner"
],
"say": "And third, the fix was slow, because rolling back was manual and nobody knew who owned the decision."
}
]
},
{
"kind": "code",
"heading": "The line behind it",
"filename": "price-cart.ts",
"lines": [
{
"code": "for (const item of cart.items) {",
"say": "For every item in the cart,"
},
{
"code": " total += await db.getPrice(item.id);",
"say": "the new code asked the database for its price, one query at a time. This is the line that caused the trouble."
},
{
"code": "}",
"say": "So a cart with fifty items meant fifty queries, and thousands of shoppers were too many."
}
],
"transition": "glitch"
},
{
"kind": "email",
"heading": "Telling customers",
"from": "Shop Status <status@shop.example>",
"to": "all customers",
"subject": "Checkout is back to normal",
"time": "15:10",
"body": [
{
"text": "Between **14:02 and 14:49** some of you could not complete a purchase.",
"say": "We told customers exactly when it happened."
},
{
"text": "**No payment was taken twice**, and every delayed order will ship as planned.",
"say": "We said that nobody was charged twice, and that delayed orders would still ship."
},
{
"text": "We are sorry, and we are working to make sure it does not happen again.",
"say": "And we apologised, without excuses."
}
],
"lead": "Good communication matters as much as the fix.",
"outro": "Short, honest and specific beats long and vague."
},
{
"kind": "checklist",
"heading": "What we will change",
"items": [
{
"label": "Fetch prices in one query",
"detail": "Done on Wednesday",
"icon": "✅",
"tone": "emerald",
"say": "The code is already fixed: prices are now fetched in a single query."
},
{
"label": "Cap database connections",
"detail": "Owner: Platform team",
"icon": "🔌",
"tone": "sky",
"say": "The platform team will cap the number of connections to the database."
},
{
"label": "Alert on error rate",
"detail": "Owner: On-call team",
"icon": "🔔",
"tone": "amber",
"say": "The on-call team will add an alarm that watches the error rate."
},
{
"label": "One-click rollback",
"detail": "Owner: Release team",
"icon": "↩️",
"tone": "violet",
"say": "And the release team will make a rollback a one-click job."
}
]
},
{
"kind": "callout",
"tone": "tip",
"title": "Blameless, on purpose",
"text": "We fix the system, not the person. People who fear blame hide problems, and hidden problems come back.",
"say": "A final word: a good postmortem is blameless. We fix the system, not the person, because people who fear blame hide problems, and hidden problems come back.",
"transition": "fade"
}
]
}