Reliability

Streaming Pipeline

Stream tokens to the UI with server-side processing of structured outputs, function calls and tool execution.

clientservicemodeldatabaseagentexternal

Explanation

The API opens a streaming connection to the LLM. A parser incrementally tokenises the stream, detecting tool call boundaries. Text tokens stream to the UI; tool calls trigger execution and feed observations back into the model.

Components

Streaming transport (SSE/WebSocket)Incremental JSON parserTool call detectorBackpressure handlerReconnection logic

When to use

Any user-facing generation
Long outputs
Real-time agents

When NOT to use

When you need the full output before processing
Backend batch jobs

Failure modes

Dropped connection mid-stream
Malformed partial JSON breaks parser
Backpressure causes memory growth
Tool call split across chunks

Production checklist

Reconnection with resume
Partial JSON parser (not full-document)
Backpressure handling
Server-side full-response logging
Cancellation API
Heartbeat / keepalive