Every production system has taught me: the failures you do not instrument for are the ones you discover from a user complaint at 3 a.m.
The minimum viable observability stack: metrics (request rates, error rates, latency percentiles, queue depths), logs (what happened in detail when something went wrong), and alerts (automated notification when metrics cross problem thresholds).
Alert on symptoms, not causes. Alert on "5xx error rate exceeds 1% for 5 minutes" not "CPU usage exceeds 80%." Most CPU spikes are harmless. All 5xx spikes affect users.
Set up your monitoring before you go live. Discovering you need it after an incident is too late.
— Dick Bassey | DevDick | 2023