Every system fails. Hardware fails. Networks partition. Third-party APIs go down. The laws of distributed systems do not make exceptions for well-intentioned engineers.
The patterns that compose resilient systems: circuit breakers (stop calling a failing downstream before cascading), retry with exponential backoff and jitter, timeouts everywhere, graceful degradation, and bulkheads.
Afripay's payment gateway implements circuit breakers across all 18+ provider integrations. When a provider starts failing, the circuit opens within 5 failed requests, traffic routes to the next provider, the circuit attempts to close after 60 seconds.
Build the failure handling before you need it. After is too late.
— Dick Bassey | DevDick | 2022