LLM Gateways & Routing
Put one secure, observable entry point in front of every model, and route each request to the cheapest model that is good enough.
LLM gateways & routing
An LLM gateway is a single service that sits between your applications and every model they use, hosted or self-hosted. Applications call one API; the gateway handles the rest:
Routing is the part that saves the most money. Many requests do not need the most capable model; sending each one to the cheapest model that meets its quality bar can cut costs substantially without users noticing.
Why it matters
Without a gateway, every team integrates every provider separately. Provider keys spread across repositories, nobody can say what AI costs per product, controls differ from team to team, and switching providers means changing dozens of codebases.
A gateway makes the organisation's AI usage governable. Security gets one place to enforce data and region rules, finance gets cost by team, and platform engineers can swap a deprecated model for its successor once, for everyone.
It also creates leverage. Once traffic flows through one place, improvements such as caching, routing or a better fallback benefit every application at the same time.
How a gateway handles a request
1. Authenticate the caller and load its policy: allowed models, regions, data classes and budget.
2. Check quotas and rate limits; reject or queue when exceeded, with a clear error.
3. Apply input policy: redact or block data the caller may not send to that destination.
4. Choose a route. Common strategies:
5. Call the model, streaming the response straight through to the caller without buffering.
6. On timeout or error, fall back along a defined chain, never to a destination the policy forbids.
7. Log the request with its attribution tags, model, tokens, cost, latency and outcome.
Build or buy: open-source gateways (for example LiteLLM or Envoy AI Gateway) and cloud provider gateways cover most of this. Build only the parts that are specific to you, such as your policy rules and attribution.
Routing with budgets and fallbacks
from dataclasses import dataclass
@dataclass
class Route:
model: str
price_per_mtok: float # blended input/output price, USD per million tokens
region: str
ROUTES = { # per use case: preferred first, then fallbacks
"ticket-classification": [Route("small-model", 0.3, "au"), Route("medium-model", 2.0, "au")],
"contract-review": [Route("large-model", 12.0, "au"), Route("large-model-alt", 10.0, "au")],
}
MONTHLY_BUDGET_USD = {"support-team": 2_000, "legal-team": 8_000}
class BudgetExceeded(Exception): ...
def handle(request: dict, spend_usd: dict, call_model, log) -> dict:
team, use_case = request["team"], request["use_case"]
if spend_usd.get(team, 0.0) >= MONTHLY_BUDGET_USD[team]:
raise BudgetExceeded(f"{team} has used its monthly AI budget")
last_error = None
for route in ROUTES[use_case]:
if route.region not in request["allowed_regions"]:
continue # policy: never fall back to a forbidden region
try:
response = call_model(route.model, request["messages"], timeout=30)
except Exception as e: # timeout, rate limit, provider outage
last_error = e
continue
cost = response["total_tokens"] / 1e6 * route.price_per_mtok
spend_usd[team] = spend_usd.get(team, 0.0) + cost
log({"team": team, "use_case": use_case, "model": route.model,
"tokens": response["total_tokens"], "cost_usd": round(cost, 6)})
return response
raise RuntimeError(f"All routes failed for {use_case}") from last_errorExperiment: routing strategy
Choose a routing strategy and a traffic mix. The effects are illustrative; measure quality with your own evaluation sets before changing routes.
What to observe
Routing only saves money safely when quality is measured per route. The right strategy depends on your traffic mix, and the mix changes as teams adopt new use cases, so review routes with evaluation results, not once at launch.
Gateways in production
The gateway becomes critical infrastructure: if it is down, every AI feature is down. Run it with multiple instances across zones, keep it stateless, and keep it thin. Business logic belongs in applications.
Do not break streaming. Buffering a whole response before forwarding it destroys time to first token for interactive applications.
Handle model deprecations centrally: keep a calendar of provider retirement dates, test successors against each use case's evaluation set, and switch routes before the deadline.
Caching needs care. Exact-match caching is safe; caching by meaning (semantic caching) can return an answer to a subtly different question, and must never serve one user's data to another. Scope caches by tenant and permission.
Rotate provider keys in the gateway only, and give each application its own gateway key so it can be revoked without affecting others.
Challenge
Fifteen teams call three model providers directly with their own keys. Finance wants AI cost by product, security wants all personal data kept in Australia, and one provider retires a model in 90 days. Design the gateway rollout: what you build or buy, how teams migrate, and what you ship first.
Production checklist
Production checklist
0 of 9 checked
Knowledge check
A preferred model times out. The only healthy fallback is in a region the caller's policy forbids. What should the gateway do?
Complete
You can now design a gateway that gives every team one secure entry point, with routing, budgets and attribution. Next: release prompt and model changes safely with evaluation gates.
Mark this chapter as complete
Track your progress and unlock the next chapter.