Cost & Reliability
Caching, model routing, fallbacks, circuit breakers — the production reliability stack.
Cost & reliability
Cost and reliability are two sides of production AI. Cost: cache, route to cheap models, compress prompts, set budgets. Reliability: retry, fall back across providers, circuit-break failing services, degrade gracefully. Together they keep your feature affordable AND up.
Why this matters
Without cost controls, an AI feature can 10x its budget overnight (traffic spike, model swap, prompt bloat). Without reliability patterns, a single provider outage takes your product down. Both are mandatory for production. The good news: the patterns compose and compound.
Cost + reliability stack
Request → rate limit → semantic cache (hit = return) → model router (cheap model first) → retry with backoff → fallback chain (provider A → B → C) → circuit breaker per provider → cost budget enforcement → response. Each layer is independently useful and composable.
The full reliability stack
import hashlib, time, random
from dataclasses import dataclass
# --- Semantic cache ---
@dataclass
class CacheEntry:
response: str
query_embedding: list[float]
model: str
created_at: float
cache: dict[str, CacheEntry] = {}
def cache_get(messages, model, threshold=0.95):
q = str(messages)
qhash = hashlib.sha256(q.encode()).hexdigest()
if qhash in cache:
return cache[qhash].response
# Semantic check
qemb = embed(q)
for entry in cache.values():
if entry.model != model: continue
if cosine(qemb, entry.query_embedding) >= threshold:
return entry.response
return None
def cache_set(messages, response, model):
q = str(messages)
qhash = hashlib.sha256(q.encode()).hexdigest()
cache[qhash] = CacheEntry(response, embed(q), model, time.time())
# --- Model router ---
def route_model(query: str) -> str:
if len(query) < 80 and "compare" not in query.lower():
return "gpt-4o-mini"
return "gpt-4o"
# --- Fallback chain with retry + circuit breaker ---
RETRYABLE = {429, 500, 502, 503, 504}
FALLBACK = ["gpt-4o", "claude-3-5-sonnet", "gpt-4o-mini"]
breaker_failures = {m: 0 for m in FALLBACK}
def call_with_full_stack(messages):
# Cache
model = route_model(messages[0]["content"])
if cached := cache_get(messages, model):
return cached, "cache"
for model in [model] + [m for m in FALLBACK if m != model]:
if breaker_failures[model] >= 5:
continue # circuit open
for attempt in range(3):
try:
resp = call_llm(messages, model=model)
cache_set(messages, resp, model)
breaker_failures[model] = 0
return resp, model
except Exception as e:
status = getattr(e, "status_code", None)
if status not in RETRYABLE and status is not None:
breaker_failures[model] += 1
break # non-retryable, try next model
delay = min(30, 0.5 * (2 ** attempt))
time.sleep(random.uniform(0, delay))
breaker_failures[model] += 1
raise RuntimeError("All models failed")Experiment: cost + reliability levers
Toggle each lever. See cumulative effect on cost and uptime.
What to observe
The levers compound. Cache alone is -60% cost. Routing alone is -75%. Together: -90% (they multiply, not add). Fallback + breaker take uptime from 99% to 99.95%. Stack all four for production-grade cost AND reliability. None is sufficient alone.
Production cost + reliability
Production stack: semantic cache (tune similarity threshold), model routing (monitor accuracy), fallback chain (pin versions, normalise response shapes), circuit breaker (tune threshold + cooldown), per-user cost budgets, cost dashboard, anomaly alerts, and graceful degradation (return cached/partial on failure).
Challenge
Your feature costs $200/day in week 1 and $4,800/day in week 2. Traffic 3x'd, cost 24x'd. Three things changed — identify all three. (See the 'Cost Blowup After Launch' challenge.)
Production checklist
Production checklist
0 of 9 checked
Knowledge check
Cache reduces cost 60%, routing reduces 75%. Both on. Total reduction?
Complete
You can now build the cost + reliability stack. Next: security and guardrails — the final production layer.
Mark this chapter as complete
Track your progress and unlock the next chapter.