How to Evaluate AI Agents: Beyond Final-Answer Accuracy

An agent is a process, not an answer, so evaluating it means judging the whole run. Measure five things for every task: whether the outcome is right, whether the trajectory was sensible, whether it chose and used the right tools, whether it followed policy, and what it cost. Build a golden set of tasks (not questions) with machine-checkable success conditions, run each task several times because agents vary, and rerun the whole set whenever the model, prompt or tools change.

A question-answering system takes an input and produces an output, so you can compare the output with a reference. An agent takes a goal and then acts: it calls tools, reads results, changes its plan and finally stops. The same final answer can come from a clean three-step run or from a chaotic forty-step one, and only one of those is safe to ship.

If you have not yet built the basics for answer quality, start with LLM evaluation fundamentals for production. Everything there still applies to the answers an agent produces. This post covers what is added when the system also acts.

Give every task a score on each dimension, and keep them separate. A single blended score hides which part broke.

Outcome comes first because nothing else matters if the task failed. But a run that passes the outcome check and fails the others is a regression waiting to happen, which is why they are scored together.

A golden set for an agent is a list of tasks. Each task describes a starting state, a goal in plain language, and a way to check success automatically.

Prefer success checks that inspect the end state (the ticket was created with the right fields) over checks that read the agent's description of what it did. Agents describe their work optimistically.

Cover the cases that hurt in production: a tool that returns an error, a tool that times out, an ambiguous request, an instruction hidden in a document the agent reads, and a request the agent should decline. These are the tasks that reveal policy and recovery problems that easy tasks never will. The injection via retrieved documents challenge presents a RAG system derailed by an uploaded document, which is the kind of case worth keeping in the set.

Many trajectory problems can be caught with plain code. Record every step of a run (the tool, its arguments, the result and the cost), then assert on the record.

def check_run(run, task) -> list[str]:
    """Return a list of failures for one agent run; empty means it passed."""
    failures = []
    calls = [step.tool for step in run.steps if step.tool]
    signatures = [(step.tool, repr(step.args)) for step in run.steps if step.tool]

    if not task.success(run.final_state):
        failures.append("outcome: end state does not match the task")
    for tool in task.must_call:
        if tool not in calls:
            failures.append(f"tool use: never called {tool}")
    for tool in task.must_not_call:
        if tool in calls:
            failures.append(f"policy: called forbidden tool {tool}")
    if len(run.steps) > task.max_steps:
        failures.append(f"trajectory: {len(run.steps)} steps, limit {task.max_steps}")
    if len(signatures) != len(set(signatures)):
        failures.append("trajectory: repeated an identical tool call")
    if run.cost_usd > task.max_cost_usd:
        failures.append(f"cost: ${run.cost_usd:.2f}, budget ${task.max_cost_usd:.2f}")
    return failures

Assertions are cheap, deterministic and easy to explain, so use them for everything you can state as a rule. Keep a model judge for the soft questions, such as whether a plan was sensible or an explanation was honest. If you use a judge, give it a written rubric, a different model family from the agent, and calibrate it against human grades, as described in the evaluation fundamentals post.

Run the same task twice and an agent may take different paths. A single pass or fail on one run tells you little. Run each task several times and report the pass rate per task and overall, along with the spread of steps and cost.

Two practical points. First, pick the number of runs from your own variance: if results barely change between runs, fewer is fine; if they swing, you need more before you trust a comparison. Second, decide in advance what counts as a regression. A task that passed every time and now passes half the time is a regression even though the average may barely move.

The agent loops article covers the limits that keep a bad run bounded, and those same limits (step cap, cost budget, duplicate-call refusal) give you per-task thresholds to assert against.

Agents change underneath you: a vendor updates a model, someone edits a prompt, a tool's response format shifts. Treat each of these as a release that needs the golden set.

The model-upgrade regression challenge starts from an upgrade where production metrics looked fine but evaluation scores had dropped, which is the situation a rerun of the golden set is there to catch. Segment your results by task type, so a drop in one category cannot hide behind gains in another.

The golden set tests what you anticipated. Live traffic shows the rest. Sample a small share of real runs, apply the same assertions and judge, and review the failures by hand. Every failure worth fixing becomes a new task in the golden set, with its start state and success check. Over weeks the set grows to match what users actually ask. The online evaluation chapter covers sampling and drift.

If policy limits what you may store, decide that early: sampled online evaluation needs a retained, access-controlled copy of at least a sample of traces.

Start small and real. Twenty tasks with strong success checks teach you more than two hundred vague ones.

How do you evaluate an AI agent? Judge the whole run, not only the final answer. Score the outcome, the trajectory, tool use, policy adherence and cost for each task in a golden set, run each task several times because agents vary, and rerun the set whenever the model, prompt or tools change.

What is trajectory evaluation? It is checking the path an agent took, not just where it ended up: how many steps it used, whether it repeated calls or looped, whether it chose the right tools and whether it asked for approval at the right moments. Much of it can be done with simple assertions over a recorded run.

How is agent evaluation different from evaluating a chatbot or a RAG system? A chatbot or RAG system produces an answer you can compare with a reference. An agent takes actions in an environment, so you also check the resulting state, the steps taken, the tools used and the cost, none of which a final-answer comparison can see.

How big should an agent golden set be? Start with roughly twenty to thirty real tasks that each have a checkable success condition, and grow it from production failures. Coverage of task types and hard cases matters more than raw count, and the right size depends on your own variance.

Do I need an LLM judge for agents? Not for everything. Use code for anything you can state as a rule, such as the end state, forbidden tools, step limits and cost. Use a calibrated judge only for soft questions like whether a plan was sensible, with a written rubric and checks against human grades.

Why run each task more than once? Agents are nondeterministic, so one run can pass by luck or fail by bad luck. A pass rate across several runs shows whether a task is reliably solved, and a drop in that rate is a regression even when the average moves little.

What should I do when a new model version changes agent behaviour? Rerun the whole golden set with the same limits, compare per task rather than only on average, read the traces of tasks that flipped, and decide whether to ship, fix or roll back. Keep the baseline so the next change has something to compare against.

• LLM evaluation fundamentals for production
• AI agent loops: stopping conditions and guardrails
• When should an AI agent ask permission?
• Evaluation topic hub
• Agentic AI Engineer roadmap