Reliability

Circuit Breaker

Problem

When a downstream service fails, retrying every request keeps hammering it and slows everything down. The system never recovers.

Pattern

Track failures per downstream. When failures exceed a threshold, open the circuit and fail fast for a cooldown period, then half-open to test recovery.

Implementation

circuit_breaker.pypython

import time

class CircuitBreaker:
    def __init__(self, threshold=5, cooldown=30.0):
        self.failures = 0
        self.threshold = threshold
        self.cooldown = cooldown
        self.opened_at = None

    def allow(self):
        if self.opened_at is None:
            return True
        if time.time() - self.opened_at > self.cooldown:
            # Half-open: allow one probe
            return True
        return False

    def record_success(self):
        self.failures = 0
        self.opened_at = None

    def record_failure(self):
        self.failures += 1
        if self.failures >= self.threshold:
            self.opened_at = time.time()

breaker = CircuitBreaker()

def call_protected(fn, *args):
    if not breaker.allow():
        raise CircuitOpenError()
    try:
        result = fn(*args)
        breaker.record_success()
        return result
    except Exception:
        breaker.record_failure()
        raise

Trade-offs

Prevents cascade failures
Fails fast instead of hanging
Can block traffic during transient blips if cooldown too long
Half-open probe can fail and re-open circuit

Production checklist

Tune threshold and cooldown per service
Half-open with single probe
Observability on circuit state
Fallback when circuit open
Manual override to force-close