Cost

Model Routing

Problem

Using a frontier model for every query is expensive and slow. Most queries are simple and a small model is sufficient.

Pattern

Classify query complexity and route to the cheapest model that can handle it, with fallback to stronger models on low confidence.

Implementation

router.pypython

def route_model(query: str, history: list) -> str:
    # Rule-based fast path
    if len(history) == 0 and len(query) < 80:
        # Simple first turn — try small model
        return "gpt-4o-mini"

    # Keyword signals for complexity
    complex_signals = ["analyse", "compare", "design", "architect", "debug"]
    if any(sig in query.lower() for sig in complex_signals):
        return "gpt-4o"

    # Default to mid-tier
    return "gpt-4o-mini"

def call_with_routing(query, history):
    model = route_model(query, history)
    try:
        return call_llm(query, model=model), model
    except LowConfidenceError:
        # Escalate to frontier
        return call_llm(query, model="gpt-4o"), "gpt-4o"

Trade-offs

60-80% cost reduction on mixed traffic
Classification errors route to wrong model
Behavioural inconsistency across models
Requires per-model evals

Production checklist

Monitor routing accuracy
Per-model cost and quality dashboards
Fallback on low confidence
A/B test routing rules
Pin model versions