All patterns
Cost & Reliability
Model Router
Route requests to the cheapest model that can handle the query, falling back to stronger models as needed.
clientservicemodeldatabaseagentexternal
Explanation
A lightweight classifier scores query complexity. Simple queries go to a cheap fast model. Complex queries go to a frontier model. If the cheap model's confidence is low, fall back to the expensive model.
Components
Query classifier (rules or small LLM)Model registryConfidence scorerFallback chainCost telemetry
When to use
- High-volume production traffic
- Mixed query complexity
- Cost-sensitive products
When NOT to use
- Low volume — savings don't justify complexity
- Uniform query difficulty — just pick one model
Failure modes
- Classifier mis-routes complex queries to weak model
- Fallback storms during outages
- Behavioural inconsistency across models
- Cold start on new model versions
Production checklist
- Classifier accuracy monitored
- Per-model cost and quality dashboards
- Graceful fallback with circuit breaker
- A/B testing framework for routing rules
- Model version pinning