Evals from Customer Data
Build an evaluation set with the customer's experts so quality is a number both sides trust.
Evals from Customer Data
An evaluation set is a collection of real tasks with expert-approved answers, used to measure the system before and after every change. Built with the customer, it replaces arguments about whether the AI is good enough with a shared measurement.
A good customer evaluation set has:
Why it matters
Without an evaluation, every model, prompt or data change is a gamble, and quality complaints become debates between anecdotes. One bad answer in front of an executive can outweigh a thousand good ones.
With an evaluation, you can show a number moving, agree a launch bar in advance, and catch regressions before users do. Hosted models change underneath you; an evaluation is how you notice.
Building it with the customer matters as much as having it. When their experts wrote the expected answers, they trust the score, and they own the system's quality after you leave.
How to do it
Collect 50–200 real tasks, weighted towards common and costly cases. Pull them from logs, tickets or documents, not from memory.
Have customer experts write or approve the expected answer and its source. Budget their time explicitly; it is the scarcest input.
Mark a critical subset: cases where a wrong answer causes harm, cost or embarrassment.
Add cases the system should refuse or hand to a person.
Choose scoring: exact match where possible, a rubric for open answers, and a model-as-judge only with spot checks against human scores.
Agree the launch bar before the first run.
Run on every change, share results in the weekly update, and hand ownership to the customer at the end.
An evaluation runner with a launch bar
import statistics
def run_eval(cases, answer_fn, judge_fn):
"""cases: [{"id", "question", "expected", "critical": bool, "should_decline": bool}]"""
results = []
for c in cases:
got = answer_fn(c["question"])
if c.get("should_decline"):
passed = got.get("declined", False) # declining is the correct behaviour
else:
passed = judge_fn(got["answer"], c["expected"])
results.append({"id": c["id"], "critical": c["critical"], "pass": passed})
overall = statistics.mean(r["pass"] for r in results)
critical = [r["pass"] for r in results if r["critical"]]
return {"overall": round(overall, 3),
"critical": round(statistics.mean(critical), 3) if critical else None,
"failures": [r["id"] for r in results if not r["pass"]]}
LAUNCH_BAR = {"overall": 0.95, "critical": 1.0} # agreed before the first run
def meets_bar(report: dict) -> bool:
return report["overall"] >= LAUNCH_BAR["overall"] and (report["critical"] or 0) >= LAUNCH_BAR["critical"]Worked example: when experts disagree
Building a policy assistant's evaluation set, two senior underwriters disagree on 15% of the expected answers.
Rather than averaging or picking one, the team treats disagreement as information:
The launch bar is then applied to the 85% of cases with an agreed answer, and the system is required to escalate the rest.
In a real engagement
Agree the launch bar before seeing results: for example 95% overall and 100% on the critical subset. A bar set afterwards moves to fit whatever the results were.
Re-run the evaluation whenever a vendor updates the model underneath you; hosted models change without notice, sometimes for the better on average and worse on your critical cases.
Keep the evaluation set separate from anything used to build or tune the system, or the score stops meaning anything.
Grow the set from production: every confirmed failure after launch becomes a new case, approved by an expert.
Challenge
The customer's experts disagree on the correct answer for 15% of cases. How do you handle those cases in the evaluation and in the launch decision?
Checklist
Production checklist
0 of 9 checked
Knowledge check
When should the launch bar for an evaluation be agreed?
Complete
You have finished “Evals from Customer Data”. Continue to the next chapter or practise in the FDE simulator.
Mark this chapter as complete
Track your progress and unlock the next chapter.