Reliability & Resilience

Core: start here

• TimeoutsNever waiting forever on a network call.

Core: start here

• Rate LimitingLimiting how many requests a client can make.
• Retry with Exponential BackoffRetrying with growing delays so you don't hammer a failing service.

Core: start here

• Circuit BreakerStopping calls to a failing service so it can recover.
• Cascading FailureOne failure triggering failures in the systems that depend on it.
• Blast RadiusHow much breaks when something fails.

• Chaos EngineeringInjecting failures on purpose to find weaknesses.

• Rate LimitingLimiting how many requests a client can make.

• TimeoutsNever waiting forever on a network call.
• BackupsCopies of data for restoring, and why untested backups don't count.

• Rate LimitingLimiting how many requests a client can make.
• Single Point of FailureOne component whose failure takes everything down.
• RedundancyExtra components so a single failure isn't fatal.
• ReliabilityA system doing what it should, even when things fail.
• AvailabilityThe share of time a system is usable, often measured in "nines".
• Nines (99.9%, 99.99%)How much downtime each level of availability allows.
• Retry with Exponential BackoffRetrying with growing delays so you don't hammer a failing service.
• Handling Dependency FailuresDeciding what happens when something you call is down.
• Restore TestingActually restoring backups regularly to prove they work.
• Liveness vs Readiness"Am I running?" vs "Can I take traffic?"

• FailoverSwitching to a standby when the primary fails.
• Fault ToleranceContinuing to work when components fail.
• Circuit BreakerStopping calls to a failing service so it can recover.
• RPO and RTOHow much data you can afford to lose, and how long you can be down.
• Disaster RecoveryRestoring service after a catastrophic failure.
• BackpressureA slow consumer signalling a fast producer to slow down.
• Compensating TransactionUndoing a completed step with a new action, since you can't roll it back.
• Cascading FailureOne failure triggering failures in the systems that depend on it.
• Deadline PropagationPassing the remaining time budget down a chain of calls.
• JitterRandomizing retry delays so clients don't all retry in sync.
• BulkheadIsolating resources so one failure doesn't sink everything.
• Thread Pool ExhaustionAll workers stuck waiting on something slow, so nothing else gets served.
• Retry StormRetries multiplying the load on an already failing system.
• Degraded ModesServing reduced functionality when dependencies fail.
• Blast RadiusHow much breaks when something fails.
• Load SheddingRejecting some requests on purpose to protect the rest.
• Active-Active vs Active-PassiveAll nodes serving traffic vs standbys waiting.
• Self-Healing SystemsAutomatically restarting and replacing failed parts.

• Chaos EngineeringInjecting failures on purpose to find weaknesses.

• TimeoutsNever waiting forever on a network call.
• BackupsCopies of data for restoring, and why untested backups don't count.

• Rate LimitingLimiting how many requests a client can make.
• Single Point of FailureOne component whose failure takes everything down.
• RedundancyExtra components so a single failure isn't fatal.
• ReliabilityA system doing what it should, even when things fail.
• AvailabilityThe share of time a system is usable, often measured in "nines".
• Nines (99.9%, 99.99%)How much downtime each level of availability allows.
• Retry with Exponential BackoffRetrying with growing delays so you don't hammer a failing service.
• Handling Dependency FailuresDeciding what happens when something you call is down.

• Fault ToleranceContinuing to work when components fail.
• Circuit BreakerStopping calls to a failing service so it can recover.
• BackpressureA slow consumer signalling a fast producer to slow down.
• Cascading FailureOne failure triggering failures in the systems that depend on it.
• JitterRandomizing retry delays so clients don't all retry in sync.
• Retry StormRetries multiplying the load on an already failing system.
• Degraded ModesServing reduced functionality when dependencies fail.
• Blast RadiusHow much breaks when something fails.

• BackupsCopies of data for restoring, and why untested backups don't count.

• AvailabilityThe share of time a system is usable, often measured in "nines".

• Disaster RecoveryRestoring service after a catastrophic failure.