LLM Observability: What to Trace and Why

Observability for an LLM application means being able to reconstruct any request: what the user asked, what was retrieved, what prompt was assembled, which model and tools were used, what each step cost and how long it took, and what came back. Record that as a trace with one span per stage. Log identifiers and metrics by default, capture full prompt and response text deliberately and with care for privacy, sample live traffic for evaluation, and alert on changes in cost, latency, refusals and retrieval quality. The aim is that "why was this answer wrong?" takes minutes to answer.

Conventional monitoring tells you whether a service is up, how fast it responds and how often it errors. An LLM application can pass all of those checks and still give wrong answers, because the failure is in the content, not the infrastructure. A fast, error-free response that cites the wrong policy looks perfect on a normal dashboard.

The why good retrieval still produces bad answers post shows why this matters: you need to know which documents were retrieved and which actually reached the prompt before you can tell where an answer went wrong.

Think of one request as a tree of steps. A trace is the record of that tree, and each step is a span with its own timing and attributes.

Add cost to each model span by multiplying tokens by the price you pay, and roll it up per request, per feature and per team. Note which version of each prompt, model and index served the request, because without versions you cannot compare before and after a change.

You do not need to invent attribute names. OpenTelemetry, an open standard for traces and metrics, maintains semantic conventions for generative AI covering spans, metrics and events, in a dedicated GenAI conventions repository (read on 11 October 2026). The repository's README does not state a stability level, and its content has moved before, so check the current status and attribute names before you build on them, and plan to update when they change. Following a standard means your traces work with more tools and your team does not maintain a private vocabulary.

The most useful data is also the most sensitive. A full prompt may contain customer data and a full response may repeat it. Decide deliberately what to keep.

Treat trace storage with the same permissions and retention rules as the underlying data. If you cannot store full text, store ids and hashes and keep a reproducible way to rebuild the prompt from versioned inputs.

Traces are also the raw material for evaluation. Sample a small share of live requests, score them with the same checks and judge you use offline and review the failures by hand. Failures worth fixing become new test cases. LLM evaluation fundamentals explains the offline half, and the online evaluation chapter covers sampling and drift. For agents, record every step so trajectory checks are possible.

A sudden change in any of these can mean a provider changed behaviour, an index went stale, a prompt was edited or someone is probing the system. The cost blowup challenge starts from a bill that rose far faster than traffic, and finding the leak depends on per-request cost data.

Do not wait for a perfect setup. A request id, versions and cost per call already answer many debugging questions, and you can add richer detail as you learn what you need. Platform teams usually own this shared plumbing, as described in what an AI platform engineer actually owns, so product teams get tracing by default.

What is LLM observability? The ability to reconstruct and understand any request to an LLM application: the input, retrieved context, assembled prompt, model and tool calls, output, tokens, cost and latency. It goes beyond uptime and error rates to the content and quality of what the system did.

What should I trace in an LLM application? One span per stage: receive, retrieve, assemble the prompt, call the model, use tools and respond. Record identifiers, versions, token counts, cost, latency and the ids and scores of retrieved documents, and capture full prompt and response text only deliberately.

Should I log full prompts and responses? Only with care. They are very useful for debugging but may contain personal or confidential data. Capture them on a sample or opt-in basis, redact sensitive fields, set a retention period and restrict access.

Is there a standard for tracing LLM calls? OpenTelemetry maintains semantic conventions for generative AI covering spans, metrics and events. Check the current status and attribute names before building on them, since they may change.

How is LLM monitoring different from normal application monitoring? Normal monitoring shows whether the service is up, fast and error-free. An LLM application can pass those checks and still give wrong answers, so you also need visibility into retrieval, prompts, model behaviour, cost and output quality.

What should I alert on? Sharp changes in cost per request, latency by stage, refusal and error rates, retrieval scores and guardrail denials. Such changes can mean a provider changed behaviour, an index went stale, a prompt was edited or the system is being probed.

How do traces help with evaluation? Sample a small share of live traces, score them with the same checks and judge you use offline, review the failures and add the worthwhile ones to your evaluation set. Over time the set grows to match real usage.

• Why good retrieval still produces bad answers
• LLM evaluation fundamentals for production
• What an AI platform engineer actually owns
• AI platform engineering topic hub
• AI Platform Engineer roadmap

• OpenTelemetry semantic conventions for generative AI (OpenTelemetry), read on 11 October 2026: the repository defines conventions for GenAI spans, metrics and events and does not state a stability level in its README