Observability & Tracing
OpenTelemetry, spans, and seeing your LLM system in production.
Observability for LLM systems
Observability is the ability to see inside your system as it runs. For LLM systems, that means traces: every LLM call, tool call, retrieval and generation, with latency, token count, cost, input and output. Without traces, production LLM systems are black boxes — you can't debug, can't optimise cost, can't detect regressions.
Why observability is non-negotiable
LLM systems are non-deterministic, multi-step, and expensive. Without traces: a slow query is a mystery, a cost spike is unattributable, a bad answer can't be traced to its retrieval. Observability turns a black box into a debuggable, optimisable, trustworthy system. It's the #1 production readiness gap.
Tracing architecture
Request → root span → child spans (retrieve, embed, search, rerank, llm, tool) → each with attributes (model, token_count, cost, latency) → export to backend (Jaeger, LangSmith, Honeycomb) → dashboards + alerts. Spans nest to show the full call tree.
Instrumented LLM app
def handle_query(query: str) -> str:
docs = retrieve(query)
answer = llm.generate(query, docs)
return answer
# Black box — no idea what happened inside. Slow? Why? Cost? Unknown.Experiment: trace inspection
See what a trace reveals about a slow query.
What to observe
Traces attribute latency, cost and quality to specific stages. A slow query is no longer a mystery — you see exactly which span is slow. A cost spike is attributable to a specific call. Bad answers can be traced to retrieval scores or finish_reason. Observability is the foundation of all production debugging.
Production observability
Production tracing: OpenTelemetry spans for every stage, attributes for queryable fields (model, tokens, cost, scores), sampling for high-volume traffic (keep 100% of errors, 1% of success), dashboards on latency/cost/quality, and alerts on anomalies. Export to a backend that supports trace queries (LangSmith, Honeycomb, Jaeger).
Challenge
P95 latency jumped from 1.2s to 4.8s overnight. No code deploy. Design a trace-based investigation to find the cause in under 30 minutes.
Production checklist
Production checklist
0 of 8 checked
Knowledge check
Your LLM app is a black box in production. What's the FIRST thing to add?
Complete
You can now instrument an LLM app with traces. Next: cost and reliability — caching, routing, fallbacks, circuit breakers.
Mark this chapter as complete
Track your progress and unlock the next chapter.