Serving Models at Scale
What drives the latency, throughput and cost of model serving, when to self-host, and how to measure and scale an inference service.
Serving models at scale
Serving a language model means turning requests into tokens quickly, reliably and at a known cost. You either call a hosted API, where the provider runs the hardware, or run models on your own GPUs with an inference server.
Four numbers describe a serving system:
LLM serving is unlike serving a normal web service. Generating each token is limited by memory bandwidth, not compute, and every active request holds a key-value (KV) cache that grows with its context length. Fitting more requests into memory, and batching them together, is how throughput is won.
Why it matters
Serving choices often decide whether an AI product is affordable. The same model can cost several times more per token depending on batching, quantisation, hardware and utilisation, and an idle GPU costs the same as a busy one.
They also decide the experience. A chat assistant with a two-second TTFT feels slow however good its answers, while a nightly summarisation job does not care about TTFT at all and should be optimised purely for throughput. Platform teams that serve every workload the same way overpay for one and frustrate the other.
Finally, self-hosting is sometimes required rather than chosen: data that cannot leave the organisation, an air-gapped site, or a fine-tuned model that only you can run.
How an inference service works
Inference servers:
Levers that change cost and speed:
Scaling:
Load-test any OpenAI-compatible endpoint
import asyncio, json, os, statistics, time
import httpx
URL = os.environ.get("BASE_URL", "http://localhost:8000") + "/v1/chat/completions"
MODEL = os.environ.get("MODEL", "my-model")
PROMPT = "Summarise the main risks of migrating a monolith to microservices in five bullet points."
async def one_request(client: httpx.AsyncClient) -> dict:
body = {"model": MODEL, "stream": True, "max_tokens": 256,
"messages": [{"role": "user", "content": PROMPT}]}
start = time.perf_counter()
first, chunks = None, 0
async with client.stream("POST", URL, json=body, timeout=120) as resp:
async for line in resp.aiter_lines():
if not line.startswith("data: ") or line.endswith("[DONE]"):
continue
delta = json.loads(line[6:])["choices"][0]["delta"].get("content")
if delta:
first = first or time.perf_counter()
chunks += 1 # roughly one token per chunk
end = time.perf_counter()
return {"ttft": (first or end) - start, "tokens": chunks, "total": end - start}
async def run(concurrency: int, requests: int) -> None:
sem = asyncio.Semaphore(concurrency)
async with httpx.AsyncClient() as client:
async def guarded():
async with sem:
return await one_request(client)
t0 = time.perf_counter()
results = await asyncio.gather(*[guarded() for _ in range(requests)])
wall = time.perf_counter() - t0
ttft = sorted(r["ttft"] for r in results)
p95 = ttft[int(0.95 * (len(ttft) - 1))]
print(f"concurrency={concurrency:>3} ttft p50={statistics.median(ttft):.2f}s p95={p95:.2f}s "
f"throughput={sum(r['tokens'] for r in results) / wall:.0f} tokens/s")
if __name__ == "__main__":
for c in (1, 8, 32, 64): # find where throughput stops rising and p95 TTFT climbs
asyncio.run(run(concurrency=c, requests=c * 4))Experiment: hosted or self-hosted?
Pick a workload and a serving option. The guidance is illustrative; run the load test and the cost tools on your own numbers before deciding.
What to observe
Utilisation decides self-hosting economics. A GPU's hourly cost divided by the tokens it actually serves in that hour is your real price per token; at low utilisation it is often far higher than a hosted API. Interactive traffic makes high utilisation hard, while batch and steady workloads make it easy.
Serving in production
Capacity: load-test to find the concurrency where p95 TTFT exceeds your target, and plan capacity below that point with headroom for a failed node. Track GPU availability as a risk; it can take weeks to obtain in some regions.
Monitoring: watch TTFT and output speed percentiles, queue depth, KV-cache usage, preemptions or evictions, error rates and tokens per second per GPU. Alert on queueing before users notice latency.
Resilience: keep a fallback path, such as a hosted API for non-sensitive traffic or a smaller model, for when capacity runs out. Pin model and server versions, and roll out upgrades with the same evaluation gates as code.
Cost: report cost per million tokens for each pool, including idle time, so teams can compare self-hosting with hosted APIs honestly.
Challenge
An internal assistant will handle 2 million requests a day, averaging 1,500 input and 300 output tokens, mostly during business hours in one time zone. Build a cost model for a hosted API and for self-hosting an open model on rented GPUs, including idle time and a spare node, and state which you would choose and what would change your mind.
Production checklist
Production checklist
0 of 8 checked
Knowledge check
Why autoscale an inference service on queue depth or KV-cache usage rather than CPU usage?
Complete
You can now measure, size and scale a model-serving system. Next: put a gateway in front of every model so teams share one secure, observable entry point.
Mark this chapter as complete
Track your progress and unlock the next chapter.