LLM Gateways: What They Do and How to Choose One

An LLM gateway is a service that every model call passes through. It handles identity, routing, budgets and rate limits, caching, logging and policy in one place, so applications do not each rebuild them. You need one when several teams or products call models and you must control cost, access or data handling centrally. Choose by measuring the overhead it adds, how it behaves when it fails and what it stores, not by feature count.

Without a gateway, each application talks to a model provider with its own client library and its own key. With one, applications talk to the gateway using a common interface, and the gateway talks to providers. The application does not need to know which model or provider served the request.

The pattern is covered in the LLM gateways and routing chapter and in the model router pattern. For where it sits in the whole stack, see the Enterprise AI Engineering Blueprint.

Not every gateway does everything, and you may not need every job on day one. These are the responsibilities worth knowing, with what good and bad look like.

A gateway is infrastructure to run, so adding it too early is a cost of its own.

The cost blowup challenge starts from a bill that rose far faster than traffic, and finding the cause depends on knowing where the spend came from. Even a very small gateway that only records caller, model and tokens gives you that attribution.

Gateways come as open-source software you run, managed services and features inside larger platforms. This post does not rank products, because the right choice depends on your constraints and any product list goes stale quickly. Decide the approach first.

Whichever route you take, evaluate the candidates against the same criteria. Run these against your own traffic where you can, because published numbers describe someone else's workload.

A gateway's configuration is where policy becomes real. This is an illustrative shape, not any product's schema.

routes:
  - name: support-assistant
    callers: ["team:support-eng"]
    allowed_models: ["model-small", "model-large"]
    data_class: internal
    redact: ["email", "phone"]
    budget:
      monthly_usd: 3000
      alert_at_percent: 80
      hard_stop: true
    fallback: ["model-large", "model-small"]
    log: { prompts: false, metadata: true }

Note the last line. Logging full prompts is useful for debugging and risky for privacy, so make it a deliberate choice per route, with retention you can defend.

• It becomes a single point of failure. Everything depends on it, so run it redundantly, test failure behaviour and decide per route whether to fail open (allow calls through) or closed (block them).
• It adds latency. Measure it, especially for streaming, where time to first token matters.
• It breaks features. Streaming, tool calling and structured outputs vary between providers, so test the ones you rely on.
• It stores sensitive data. Prompts and responses may be retained. Decide what is logged and for how long.
• It can be bypassed. If code can still call a provider directly, the controls are optional. Remove direct provider keys from applications so the gateway is the only route.

Cost levers such as caching and routing to cheaper models come after you can see where the money goes. How to reduce LLM costs covers them, and the AI Cost Calculator lets you model the effect with your own numbers.

What is an LLM gateway? A service that every model call passes through. It handles authentication, routing and fallbacks, budgets and rate limits, caching, logging and policy in one place, so each application does not rebuild them.

Do I need an LLM gateway? You probably do once several teams or products call models and you need central control of cost, access or data handling. With one team and one provider, a thin internal wrapper around provider calls is usually enough until that changes.

What is the difference between an LLM gateway and a model router? Routing is one job a gateway can do: choosing which model serves a request. A gateway also covers identity, budgets, caching, logging and policy. A router can exist inside a gateway or on its own.

Should I build or buy a gateway? Build when your needs are unusual and small, and you can own latency, upgrades and on-call. Adopt an existing one when you need breadth and speed, but check its data handling, roadmap and exit cost first.

Does a gateway slow my application down? It adds a hop, so it adds some latency. How much depends on the gateway and your setup, so measure it on your own requests, including time to first token for streaming, rather than relying on a published number.

What happens if the gateway goes down? Everything that depends on it is affected unless you planned for it. Run it redundantly, test failure behaviour, and decide per route whether it should fail open or closed.

Does a gateway store my prompts? It may. Check what is logged, for how long and where, and make prompt logging a deliberate per-route choice with a retention period you can defend.

• The Enterprise AI Engineering Blueprint
• How to reduce LLM costs by 90 percent without hurting quality
• Enterprise AI guardrails: a ten-layer defense in depth
• AI platform engineering topic hub
• AI Platform Engineer roadmap