Start Agent Evals Small: 20 Tasks, Trials, pass@k vs pass^k

Start with 20 to 50 tasks taken from real failures, run each one several times, and read the transcripts. Report two numbers: pass@k, the chance that at least one of k attempts succeeds, and pass^k, the chance that all k succeed. Use pass@k when one success is enough, and pass^k when users need consistency. The definitions and the 20 to 50 task guidance come from Anthropic's "Demystifying evals for AI agents", published in January 2026.

Agent evals are easier to discuss once everyone uses the same terms. The Anthropic guide defines them like this: a task is a single test with inputs and success criteria; a trial is one attempt at a task; a grader is logic that scores some aspect of performance; a transcript is the full record of a trial, including outputs, tool calls and reasoning; and an outcome is the final state of the environment.

The outcome definition is the one people skip. The guide's example is a booking agent: the question is whether the reservation actually exists in the database, not whether the agent says it does. Grade the state of the world wherever you can.

For the broader metrics (trajectory, tool use, policy adherence, cost), see How to Evaluate AI Agents: Beyond Final-Answer Accuracy. This article is about how to start, and how to read the results.

You will usually combine three kinds of grader. The same guide describes them as follows.

A sensible default is code graders for anything checkable (did the file change, did the test pass, does the row exist), a model grader for qualities like tone or completeness, and a small human sample to check the model grader.

Agents are nondeterministic, so a single pass or fail is one sample from a distribution. Running k trials per task turns that into a rate, and the choice of how to aggregate the trials changes the story.

• pass@k is the likelihood of at least one correct solution in k attempts. It rises as k increases. It suits tasks where a single success is what matters, such as generating candidate fixes that a person will pick from.
• pass^k is the probability that all k trials succeed. It falls as k increases. It suits customer-facing agents that must behave consistently.

The same agent looks excellent on pass@3 and mediocre on pass^3. Neither number is wrong. Decide which one matches how the agent is used, and say so in the report. If a person reviews every output, pass@k may be fine. If the agent acts alone on a customer's account, pass^k is the honest measure.

Here is a small script that computes both from raw trial results, assuming each task has k independent trials recorded as booleans. It computes simple empirical rates over the trials you ran.

def pass_at_k_empirical(trials: list[bool]) -> bool:
    """At least one trial succeeded."""
    return any(trials)

def pass_hat_k_empirical(trials: list[bool]) -> bool:
    """Every trial succeeded."""
    return all(trials)

def estimate(results: dict[str, list[bool]]) -> dict[str, float]:
    n = len(results)
    return {
        "pass@k": sum(pass_at_k_empirical(t) for t in results.values()) / n,
        "pass^k": sum(pass_hat_k_empirical(t) for t in results.values()) / n,
    }

# Per-task probability form, if you know the per-trial rate p:
def pass_hat_k_from_rate(p: float, k: int) -> float:
    return p ** k

def pass_at_k_from_rate(p: float, k: int) -> float:
    return 1 - (1 - p) ** k

With very few trials per task these empirical numbers are noisy, so treat them as a signal to investigate, and increase trials where the decision matters.

Do not wait for a large benchmark. The Anthropic guide says teams often delay evals until they have hundreds of tasks, and that "20-50 simple tasks drawn from real failures is a great start." It gives the reason: early in development, changes tend to have large effects, so small samples can detect them. It also recommends starting from manual checks you already run and converting user-reported failures into test cases.

Two details from the guide are worth copying. Tasks should be unambiguous enough that two domain experts would give the same pass or fail verdict. And a task that fails at 0% across many trials usually points to a broken task or grader, not an incapable agent.

Numbers hide the cause. The guide says you cannot tell whether graders work well without reading transcripts and grades from many trials, and that a score should not be taken at face value until someone has done so. Reading shows whether a failure was a real agent mistake or a grader rejecting a valid answer, and a fair failure should make clear what the agent got wrong.

Pick one agent you already run. Write down the twenty most recent things that went wrong or that you check by hand. Turn each into a task with a success condition, run each five times, and report pass@5 and pass^5 side by side. Then pick the one metric that matches how the agent is used, and put it in your release checklist. The Build an eval pipeline lab takes the same approach into a repeatable harness, and the LLM evaluation fundamentals article covers the wider discipline.

What is the difference between pass@k and pass^k? pass@k is the probability that at least one of k attempts succeeds, so it rises as k grows. pass^k is the probability that all k attempts succeed, so it falls as k grows. Use the first when one success is enough and the second when consistency matters.

How many tasks do I need to start evaluating an agent? Anthropic's guide says 20 to 50 simple tasks drawn from real failures is a great start. Add more as the agent matures and failures become subtler.

How many trials per task should I run? The guide does not fix a number. Run enough to see variation, for example several per task, and increase the count where a decision depends on a small difference.

Should I use an LLM as a grader? Model-based graders are flexible and scale well but are nondeterministic and need calibration against human judgement. Use code graders where a result is checkable, and sample by hand to confirm the model grader.

What does a 0% pass rate mean? According to the guide, a consistent 0% across many trials usually indicates a broken task or grader rather than an incapable agent, so inspect the task before changing the agent.

Why grade outcomes instead of what the agent says? Because an agent can claim success without having changed anything. Checking the final state, such as whether a record exists, measures what actually happened.