A SellTrove payment processing outage. Duration: 23 minutes. Impact: 340 payment attempts returning errors. Revenue impact: quantifiable. Reputation impact: also quantifiable, and worse.
The timeline: 14:32 — PagerDuty alert fires on payment error rate exceeding 1%. 14:34 — On-call engineer acknowledges. 14:38 — Root cause identified: a Lambda function dependency update had introduced a breaking change in the payment gateway client library. 14:41 — Rollback initiated. 14:55 — Rollback complete, error rate nominal, all failed transactions retried via idempotency.
What slowed the response: the runbook for "payment processing errors" linked to the wrong CloudWatch dashboard. Four minutes lost navigating to the correct log stream.
What changed: runbook updated with correct dashboard links, tested by two engineers who were not the author. Lambda dependency updates now require a canary deployment (10% of traffic, 15 minutes, error rate check) before full rollout.
Every outage teaches you something the architecture review did not. Write it down. Build the fix. Repeat.
— Dick Bassey | DevDick | 2025