Evaluating Agents
Score agents on outcomes, the steps they take and what they cost, with sandboxed tasks, repeated runs and evaluation gates in CI.
Evaluating agents
Evaluating a single LLM call asks one question: is this output good? Evaluating an agent asks several, because an agent takes many steps, uses tools, changes things in the world and behaves differently from run to run.
Score agents at three levels:
An agent that reaches the right answer by a dangerous or wasteful route has not passed.
Why it matters
Final-answer checks alone miss the failures that hurt most in production. An agent can succeed by luck, succeed while skipping an approval step, or succeed at three times last week's cost. Model upgrades often keep the success rate steady while changing behaviour underneath: more tool calls, different tool choices, a habit of retrying forever.
Agents also cannot be tested casually against live systems. A test that sends real emails or edits real records is not repeatable and can do harm. Good agent evaluation needs a controlled environment, recorded tasks and checks on the steps as well as the end state.
An agent evaluation harness
1. Task suite: 30–200 realistic tasks, each with a starting state, an instruction and a checker for the end state. Include easy tasks, hard ones, tasks that should be refused, and attack tasks containing injected instructions.
2. Sandboxed tools: the same tool interfaces the agent uses in production, backed by fakes or recorded responses with deterministic fixtures. Side effects land in the sandbox state that checkers inspect.
3. Runner: run every task several times, because agents are not deterministic. Record the full trace: every model call, tool call, argument, result, token count and timing.
4. Graders: code checks on the end state for outcomes; rules on the trace for trajectories (forbidden tools, step budgets, approval before high-risk actions, no repeated identical calls); and an LLM judge with a written rubric only for what code cannot check, calibrated against human labels.
5. Report and gate: success rate, consistency across runs, rule violations, and cost and steps per task, compared with the previous version. Fail the build when any of them crosses a threshold.
6. Feedback loop: sample production traces, label failures and turn each new failure into a task in the suite.
A minimal harness
from dataclasses import dataclass, field
from statistics import mean
from typing import Callable
HIGH_RISK = {"send_email", "issue_refund", "delete_record"}
@dataclass
class Task:
id: str
instruction: str
initial_state: dict
check_outcome: Callable[[dict], bool] # inspects the sandbox end state
max_steps: int = 12
@dataclass
class Run:
success: bool
violations: list[str] = field(default_factory=list)
steps: int = 0
cost_usd: float = 0.0
def trajectory_violations(trace: list[dict], task: Task) -> list[str]:
problems, approved = [], False
for i, step in enumerate(trace):
if step["tool"] == "request_approval":
approved = True
if step["tool"] in HIGH_RISK and not approved:
problems.append(f"step {i}: {step['tool']} without approval")
if i > 0 and step == trace[i - 1]:
problems.append(f"step {i}: repeated identical call")
if len(trace) > task.max_steps:
problems.append(f"{len(trace)} steps exceeds budget of {task.max_steps}")
return problems
def run_task(agent, sandbox_factory, task: Task) -> Run:
sandbox = sandbox_factory(task.initial_state) # fake tools, no real side effects
trace, cost = agent.run(task.instruction, tools=sandbox.tools)
violations = trajectory_violations(trace, task)
return Run(success=task.check_outcome(sandbox.state) and not violations,
violations=violations, steps=len(trace), cost_usd=cost)
def evaluate(agent, sandbox_factory, tasks: list[Task], k: int = 5) -> dict:
per_task = {t.id: [run_task(agent, sandbox_factory, t) for _ in range(k)] for t in tasks}
runs = [r for rs in per_task.values() for r in rs]
return {
"pass_rate": mean(r.success for r in runs), # average over all runs
"pass_all_k": mean(all(r.success for r in rs) for rs in per_task.values()), # pass^k: reliable every time
"violations": sum(len(r.violations) for r in runs),
"mean_steps": mean(r.steps for r in runs),
"mean_cost_usd": mean(r.cost_usd for r in runs),
}
# In CI: compare evaluate(new_agent, ...) with the stored baseline and fail the
# build if pass_all_k drops, any violation appears, or cost rises beyond budget.Experiment: what does your evaluation catch?
Pick a change to the agent and how thoroughly you evaluate it. The outcomes are illustrative of common failure patterns.
What to observe
Final-answer checks catch the least. Repeated runs reveal unreliability, and trajectory rules reveal unsafe or wasteful behaviour that leaves the final answer unchanged. Both are cheap to add once tasks run in a sandbox.
Agent evaluation in production
Keep the sandbox faithful: when a real API changes its responses or errors, update the fakes, or evaluation and production drift apart. Re-record responses periodically, and include error and timeout cases, not just happy paths.
Track the cost of evaluation itself. Running 100 tasks five times is 500 agent runs, so run a fast subset on every change and the full suite before releases and model upgrades.
Flaky tasks undermine trust in the gate. When a task passes only some of the time with no code change, decide whether the agent is unreliable (a real finding) or the task's checker is ambiguous (fix the task).
In production, sample traces every day, have a person label a small number, and add every new failure mode to the suite. Report pass^k, the share of tasks that succeed on every run, alongside the average pass rate: users experience the unreliable runs, not the average.
Challenge
Your scheduling agent books meetings across calendars. After a model upgrade, the meeting-booked rate is unchanged, but your cloud bill for the agent doubled and a customer reports an invite sent without their approval. Design the task suite, sandbox, trajectory rules and CI gate that would have caught both before release.
Production checklist
Production checklist
0 of 8 checked
Knowledge check
Why report pass^k (every one of k runs succeeds) as well as the average pass rate?
Complete
You can now evaluate agents on outcomes, trajectories and cost. This completes the LLM Evaluation series. Next: put it into practice with the Build an Evaluation Pipeline lab, or apply it to always-on agents.
Mark this chapter as complete
Track your progress and unlock the next chapter.