Reliability

Retry with Exponential Backoff

Problem

LLM APIs and external services fail transiently — rate limits, network blips, provider hiccups. Naive retry storms the failing service and makes things worse.

Pattern

Retry failed requests with exponentially increasing delay, full jitter (randomised), a max attempt count, and only for idempotent / retryable status codes.

Implementation

exponential_backoff.pypython

import random
import time
from functools import wraps

RETRYABLE = {429, 500, 502, 503, 504}

def with_retry(max_attempts=4, base_delay=0.5, max_delay=30.0):
    def deco(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            attempt = 0
            while True:
                try:
                    return fn(*args, **kwargs)
                except Exception as e:
                    attempt += 1
                    status = getattr(e, "status_code", None)
                    retryable = status in RETRYABLE or status is None
                    if attempt >= max_attempts or not retryable:
                        raise
                    # Full jitter: random between 0 and exponential delay
                    delay = min(max_delay, base_delay * (2 ** attempt))
                    delay = random.uniform(0, delay)
                    time.sleep(delay)
        return wrapper
    return deco

@with_retry(max_attempts=4)
def call_llm(messages, model="gpt-4o-mini"):
    return client.chat.completions.create(model=model, messages=messages)

Trade-offs

Increases total latency on failures (by design)
Full jitter reduces thundering herd but adds variance
Retrying non-idempotent operations can cause duplicate side effects
Retry budget must respect provider rate limits, not fight them

Production checklist

Only retry idempotent operations
Respect Retry-After header on 429s
Cap total retry time (deadline propagation)
Log every retry with attempt number and delay
Surface final error to caller, not intermediate ones
Combine with circuit breaker for sustained outages