Reliability

Fallback Model Chain

Problem

A single LLM provider has outages, rate limits or degradations. Your product should not go down with it.

Pattern

Define an ordered fallback chain of models. On failure (after retries), fall through to the next model in the chain. Each model has its own retry policy.

Implementation

fallback_chain.pypython

from dataclasses import dataclass

@dataclass
class ModelConfig:
    provider: str
    model: str
    max_tokens: int
    temperature: float

FALLBACK_CHAIN = [
    ModelConfig("openai", "gpt-4o", 2048, 0.2),
    ModelConfig("anthropic", "claude-3-5-sonnet", 2048, 0.2),
    ModelConfig("openai", "gpt-4o-mini", 2048, 0.2),  # cheap last resort
]

def call_with_fallback(messages):
    last_error = None
    for cfg in FALLBACK_CHAIN:
        try:
            return call_model(cfg, messages)
        except Exception as e:
            last_error = e
            log.warning(f"Model {cfg.provider}/{cfg.model} failed: {e}")
            continue
    raise RuntimeError(f"All models failed: {last_error}")

Trade-offs

Resilience to provider outages
Behavioural differences across models (temperature, style, capability)
Structured output schemas may not be supported uniformly
Cost variance across the chain

Production checklist

Pin model versions, not 'latest'
Normalise response shapes across providers
Alert when fallback rate exceeds threshold
Track per-model latency and cost separately
Test fallback path in CI