Design for Failure. Every System Will Fail.

·1 min read·

The question is not whether your system will fail. It is whether you designed it to fail safely.

Every system fails. Hardware fails. Networks partition. Third-party APIs go down. The laws of distributed systems do not make exceptions for well-intentioned engineers.

The patterns that compose resilient systems: circuit breakers (stop calling a failing downstream before cascading), retry with exponential backoff and jitter, timeouts everywhere, graceful degradation, and bulkheads.

Afripay's payment gateway implements circuit breakers across all 18+ provider integrations. When a provider starts failing, the circuit opens within 5 failed requests, traffic routes to the next provider, the circuit attempts to close after 60 seconds.

Build the failure handling before you need it. After is too late.

— Dick Bassey | DevDick | 2022

Share this article
← Back to all writing