How to Reduce LLM Costs by 90% Without Hurting Quality

LLM cost is tokens multiplied by price per token. Three levers attack that product from different sides. A semantic cache removes requests entirely. A model router lowers the price per token on the requests that remain. Context compression lowers the token count on the requests that reach an expensive model. Because each lever acts on what the previous one left behind, the savings multiply. With moderate assumptions, a cache that serves half of requests, a router that sends most of the rest to a cheap model, and compression that halves the remaining prompts, the arithmetic lands around a 90 percent reduction. The numbers in this article are a worked model with stated assumptions, not a measurement from one system. Yours will differ, and you should measure them.

Before touching anything, instrument. Per request, record the model, input tokens, output tokens, and the feature or user that generated it. Aggregate daily. Most teams that do this for the first time discover two things: a small number of features account for most of the spend, and prompts are far longer than anyone remembers writing because system prompts, few-shot examples and conversation history accumulate.

The Cost Blowup challenge walks through a case where traffic grew 3x and cost grew 24x. Three changes compounded: the cache was removed, the default model was moved to a larger one, and full conversation history was added to every prompt, growing the average prompt from 450 to 8,200 tokens. Finding the leak depends on measuring cost per request.

Optimising without this data means guessing. Measure first, for a week if you can.

What it does

A conventional cache returns a stored response when the exact same request arrives again. In LLM systems exact repeats are rarer than you would hope, because users phrase the same question differently. A semantic cache embeds the incoming prompt, searches for stored prompts with a similar embedding, and returns the stored response if the similarity clears a threshold.

How to implement it

• Embed the user-facing part of the prompt, not the whole assembled prompt with system instructions, or every request looks alike.
• Store the embedding, the original prompt, the response, the model that produced it, and a timestamp.
• On lookup, take the nearest neighbour and check the similarity against a threshold. Values around 0.95 are a common starting point for cosine similarity; the right value depends on the embedding model and how much paraphrase you can tolerate. Tune it on real traffic.
• Apply a time-to-live. Answers about anything that changes go stale.
• Scope the cache. A cached answer that includes one user's data must never be served to another. Key by tenant, and by user where the answer is personal.

Where it goes wrong

The threshold is too loose and two different questions get the same answer. The cache serves an answer produced by a model you have since replaced. A personalised answer leaks across users. Each of these is a correctness bug dressed as a cost saving, so add cache hits to your evaluation sampling. A hit that fails faithfulness is a bug.

What to expect

Hit rates depend entirely on how repetitive your traffic is. Customer support and internal knowledge assistants often see high repetition. Open-ended creative tools see almost none. Measure before you count on it.

What it does

Not every request needs the most capable model. A router classifies each request and sends simple ones to a smaller, cheaper model, reserving the frontier model for requests that need it. Price differences between model tiers are commonly an order of magnitude, so routing even a majority of traffic to the cheap tier changes the bill materially.

How to classify

Three approaches, from cheapest to most accurate.

1. Rules. Request length, feature, presence of code, keywords. Free to run, brittle to maintain. 2. A small classifier. Train or prompt a small model to label requests by complexity. Costs a little per request, generalises better. 3. Try the cheap model first. Send the request to the small model and ask it to report confidence or to flag when the task is beyond it. Escalate on low confidence. This costs an extra call on escalated requests but needs no separate classifier.

Most production routers combine the first and third.

Where it goes wrong

The small model is confidently wrong and never escalates. Quality on a category of requests drops and nobody notices because the golden set was built when everything went to the big model. Fix this by segmenting your evaluation by route and tracking each segment. Routing without per-route evaluation is a quality regression waiting to be discovered by a user.

Confidence-based fallback

Whatever the classifier decides, the cheap model's own answer should be checked. A verifier step, a schema validation, a refusal detector, or a self-reported confidence score can trigger a retry on the frontier model. The retry costs money, but only on the fraction that needed it.

What it does

Prompts grow. A system prompt, a handful of examples, retrieved documents, and the entire conversation so far. Every turn re-sends all of it. A conversation that starts at a few hundred tokens can be several thousand by the tenth turn, and the cost per turn grows with it. Compression keeps the information and drops the tokens.

Techniques

• Summarise old turns. Keep the last few turns verbatim and replace older ones with a running summary. Regenerate the summary every few turns, not every turn.
• Trim retrieval. Retrieve fewer, better chunks. Rerank and keep the top three rather than stuffing ten into context. This usually improves answers as well as cost, because irrelevant chunks distract the model.
• Cut the examples. Few-shot examples were often added during prototyping and never revisited. Test whether the model still needs them.
• Use prompt caching where the provider offers it. Several providers discount tokens that repeat exactly from a previous request, which makes a stable system prompt prefix much cheaper. Structure prompts so the stable part comes first.

Where it goes wrong

Summaries lose the detail the user mentioned eight turns ago and the assistant forgets a constraint. Evaluate long conversations specifically; short golden-set questions will not catch it.

The levers multiply because each one acts on what the previous one left behind. A worked model with stated assumptions:

• The cache serves 50 percent of requests. Half the traffic remains.
• The router sends 80 percent of the remaining traffic to a model that costs one tenth as much. The cost of that traffic falls by roughly 72 percent, leaving about 28 percent of the pre-routing cost.
• Compression halves the tokens on everything that still reaches a model, leaving 14 percent of the post-cache cost.

Half of the traffic, at 14 percent of its former cost, is about 7 percent of the original bill. A 93 percent reduction under these assumptions. Loosen every assumption, a 40 percent cache hit rate, 60 percent routed, 30 percent compression, and you land near 75 percent. Somewhere between those is realistic for a repetitive, assistant-style workload. A workload with no repetition and uniformly hard requests will get less from the cache and the router and should lean on compression and prompt caching.

Run your own numbers with your own measured traffic. The point of the model is that the levers compound, not that any particular figure is guaranteed.

Caching last is deliberate. It is the lever with the most ways to serve a wrong answer, and it should be added to a system whose evaluation is already in place.

• Per-request cost recorded with model, tokens, feature and tenant.
• Daily cost dashboard with an anomaly alert.
• Per-user or per-tenant budgets with a hard stop.
• Evaluation segmented by route and including cache hits.
• Cache scoped by tenant with a time-to-live.
• Stable prompt prefix ordered for provider prompt caching.
• Long-conversation cases in the golden set.

Will caching hurt answer quality? Only if the similarity threshold is too loose, the cache is not scoped, or entries outlive the facts they contain. Evaluate cache hits like any other output.

Which requests should go to the small model? Whichever ones your evaluation shows it handles as well as the large one. Decide by measured quality per route, not by intuition. Are the 90 percent figures real? They are a worked model with stated assumptions, shown above. Real reductions depend on how repetitive your traffic is and how much of it is genuinely simple. Measure your own.

What about fine-tuning a small model instead? That can lower cost further for a narrow, stable task, but it adds a training and maintenance burden. Exhaust routing and compression first; they need no training.

What should I measure first to find where my LLM spend goes? Per request, record the model, input tokens, output tokens, and the feature or user that generated it, then aggregate daily. A few features usually account for most of the spend, and prompts are often far longer than anyone remembers.

What similarity threshold should a semantic cache use? Values around 0.95 for cosine similarity are a common starting point, but the right value depends on the embedding model and how much paraphrase you can tolerate. Tune it on real traffic and evaluate cache hits for faithfulness.

How do I stop a cheap model being confidently wrong? Check its answer with a verifier step, schema validation, a refusal detector or a self-reported confidence score, and retry on the frontier model when the check fails. Segment your evaluation by route so a quality drop in one category is visible.

Does provider prompt caching help? Several providers discount tokens that repeat exactly from a previous request, so a stable system prompt prefix becomes much cheaper. Structure prompts so the stable part comes first, and check your provider's current terms.

• Cost & Reliability chapter
• Model Router architecture
• Cost Blowup challenge
• LLM Evaluation Fundamentals for Production (segmented evaluation is what makes routing safe)
• The Enterprise AI Engineering Blueprint (where cost metering fits in the AI gateway layer)
• The 90-Day Enterprise AI Adoption Playbook