Latency

Streaming Responses

Problem

Generating a full response before sending it to the user creates high perceived latency. Users abandon slow pages.

Pattern

Stream tokens from the LLM to the client via Server-Sent Events as they are generated, so the user sees the first token in < 500ms.

Implementation

stream.pypython

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

@app.post("/api/chat")
def chat(req: ChatRequest):
    def generate():
        stream = client.chat.completions.create(
            model=req.model,
            messages=req.messages,
            stream=True,
        )
        for chunk in stream:
            delta = chunk.choices[0].delta.content
            if delta:
                yield f"data: {json.dumps({'token': delta})}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")

Trade-offs

Great perceived latency (TTFT < 500ms)
More complex error handling mid-stream
Tool calls split across chunks need an incremental parser
Observability requires capturing the full stream server-side

Production checklist

Server-side full-response logging
Reconnection / resume
Heartbeat keepalive
Cancellation API
Backpressure handling
Graceful error mid-stream