The most important habit: write a runbook for every alert before you go live. When you are woken at 3 a.m. by a payment alert, you do not want to think from first principles. The runbook removes cognitive load.
The second most important: when you resolve an incident, write a post-mortem. Not to assign blame, but to capture what happened, what the immediate fix was, and what permanent fix prevents recurrence.
The third: reduce alert noise aggressively. An alert that fires 20 times a week and is always benign is not an alert. It is noise that trains engineers to ignore alerts.
The goal of on-call is not to respond to incidents. It is to build systems that need you on-call less often.
— Dick Bassey | DevDick | 2024