Advanced50 minProduction AI Systems

Add Tracing & Observability

Instrument an LLM app with OpenTelemetry traces for every LLM call, tool call and retrieval. See your system in production.

Scenario

Your LLM app is a black box in production. Add tracing so you can see every step, latency and token cost.

Objective

Instrument call_llm, retrieve and tool_call with spans. Export to a trace backend.

Starter code

Implement the TODOs to complete the lab.

tracing.pypython

from opentelemetry import trace

tracer = trace.get_tracer("llm-app")

@tracer.start_as_current_span("call_llm")
def call_llm(messages, model="gpt-4o-mini"):
    # TODO: add attributes (model, token_count, latency)
    # TODO: add event on error
    pass

@tracer.start_as_current_span("retrieve")
def retrieve(query, k=5):
    # TODO: add attributes (query, k, scores)
    pass

Solution hints

1Use span.set_attribute for queryable fields
2Add token_count and cost as attributes
3Use span.add_event for errors
4Record exception on failure

Validation steps

Your implementation should pass these checks.

Trace shows nested spans for call_llm, retrieve, tool
Each span has model, latency, token_count attributes
Errors are recorded as events
Trace exports to backend (Jaeger/LangSmith)

Run validation

This is a simulated validation environment. In production, this would run your code against the validation steps.