LLM Evaluation Fundamentals for Production Systems

A production LLM system needs three evaluation layers. A golden set built from real traffic gives you a fixed exam the system must pass. Offline metrics for retrieval and generation, scored partly by a calibrated LLM judge, turn "feels better" into numbers. A CI gate blocks releases that regress those numbers, and online sampling catches the regressions your golden set never anticipated. None of this requires a large team. It requires that you start with fifty real questions and refuse to ship without them.

Every team that ships an LLM feature without evaluation ends up in the same place within a quarter. A prompt change that felt better in three manual tests regresses a case nobody tried. A model upgrade that the vendor calls an improvement changes the tone or the refusal rate. A retrieval index rebuild silently drops a document category. Nobody notices until a user, or worse a customer's customer, does.

Evaluation is not a quality nicety layered on afterwards. It is the thing that lets you change the system at all. Without it, every change is a gamble and the rational move is to stop changing anything, which is how promising systems freeze at "good enough for the demo."

A golden set is a fixed collection of inputs with known-good outputs, or at least known-good judgements. It is your exam. The system takes it before every release.

Where the questions come from

From real traffic, not from imagination. Questions engineers invent are cleaner, shorter and more sensible than what users type. Pull them from production logs, support tickets, the pilot's chat history, or the customer's own FAQ. If the system is not live yet, get the domain expert to write down the last fifty questions they were actually asked.

How many

Fifty is enough to start. A hundred is comfortable. Beyond a few hundred you get diminishing returns until you start segmenting by category. The number matters less than the coverage: every major question type, every source system, the ambiguous cases, the out-of-scope cases where the right answer is a refusal, and the adversarial cases where someone is trying to make it misbehave.

What "known-good" means

For factual questions, the expert writes the reference answer and the source that supports it. For open-ended questions, the expert writes a rubric: what must be present, what must not be present, what would make it wrong. For refusals, the expected output is "declines and explains why."

Versioning and refresh

The golden set is code. It lives in the repository, it has a version, and changes to it go through review. Refresh it on a schedule from new production queries, and keep the old version so you can compare like with like across releases. When you add a question because a bug was found, that question stays forever. It is a regression test.

Split the system into the parts you can measure separately. For retrieval-augmented systems that means retrieval and generation. For agents it also means tool selection and task completion.

Retrieval metrics

Retrieval is the easiest part to measure and the most common source of bad answers, so measure it first.

• Recall at k. Of the documents that should have been retrieved, what fraction appeared in the top k? Low recall means the answer cannot be right because the information never reached the model.
• Precision at k. Of the top k retrieved, what fraction are relevant? Low precision means the context is polluted with noise that distracts the model or crowds out the right chunk.
• Mean reciprocal rank. How high does the first relevant result appear? This matters when the model attends more to the start of the context.

These need labelled relevance judgements per question, which is the same expert effort as the golden set. Do them together.

Generation metrics

• Faithfulness. Is every claim in the answer supported by the retrieved context? This is the hallucination metric. It is usually scored by an LLM judge that decomposes the answer into claims and checks each one against the context.
• Answer relevance. Does the answer address the question asked, rather than a nearby question or a general lecture on the topic?
• Correctness against the reference. Does the answer match the expert's reference in substance? Partial credit is normal here; define what partial means.
• Refusal accuracy. Did the system refuse when it should have, and answer when it should have? Both directions are failures.

Agent metrics

For agents, add task completion rate, number of steps to completion, tool selection accuracy, and cost per completed task. An agent that completes the task in forty steps and fifteen dollars has not succeeded. The agent loops article covers why.

Human grading does not scale to every release. An LLM judge scores outputs against a rubric so you can run the golden set on every pull request. It works, with four conditions.

Use a different model from the one being evaluated. Models rate their own outputs more favourably. A judge from a different family removes the self-preference bias.

Temperature zero and an explicit rubric. The judge should see the question, the answer, the reference or the context, and a written rubric with named grades. Ask for the grade and a one-line reason. Do not ask "is this good?"

Calibrate against humans before trusting it. Have the expert grade fifty outputs. Have the judge grade the same fifty. Compare. If agreement is poor, fix the rubric, not the threshold. Re-check calibration whenever the judge model changes.

Judge the parts, not the whole. A single "quality" score hides everything. Separate scores for faithfulness, relevance and completeness tell you what to fix.

The judge is a measurement instrument. Like any instrument it drifts and needs recalibration. Put a date on the last calibration and treat an old one as suspect.

Evaluation only changes behaviour if it can block a release. Wire the golden set into continuous integration.

• Run the full golden set on every pull request that touches a prompt, a retrieval setting, a model version or the index build.
• Compare each metric against the main branch, not against an absolute number. The question is "did this change make it worse," not "is it above 80 percent."
• Block the merge when a metric drops beyond a threshold you chose in advance. Five percentage points is a common starting point. Make the threshold tighter for faithfulness than for style.
• Print the individual failures in the CI output. A metric that dropped is a headline; the three questions that now fail are the actual bug report.
• Cache judge results per question and per output so unchanged cases do not cost money on every run.

A release blocked by evaluation is the system working. Celebrate the first one.

The golden set tests what you anticipated. Production tests everything else. New question types appear, documents change, a vendor updates a model silently, a user finds a phrasing nobody tried.

Sample a small fraction of live traffic, one to five percent, and run the judge on it asynchronously. Track the same metrics over time. Alert when a rolling average moves. Feed the interesting failures back into the golden set, which is how the golden set grows to match reality.

If policy prevents storing prompts and responses, decide that now, because online evaluation is impossible without at least a sampled, retained copy. This is a discovery question, not a late surprise.

If you have nothing today, this is one week of work for one engineer and a few hours of an expert's time.

Everything after that is refinement.

How big does the golden set need to be? Fifty real questions to start, a hundred to be comfortable, segmented by category as it grows. Coverage matters more than count.

Can I use the same model as judge and as system? You can, but you should not. Self-preference bias is well documented. Use a different model family and calibrate against human grades.

What threshold should block a release? Start at five percentage points relative to the main branch for correctness and relevance, and tighter for faithfulness. Adjust once you have seen a few releases.

What if my domain expert has no time? Then you cannot evaluate, and you will be judged on anecdotes. Booking expert hours is a project prerequisite, not a favour. Put it in the scope document.

Is evaluation different for agents? The layers are the same. The metrics add task completion, step count, tool selection accuracy and cost per task, and the golden set contains tasks rather than questions.

What is faithfulness in LLM evaluation? Faithfulness asks whether every claim in the answer is supported by the retrieved context. It is the hallucination metric and is usually scored by an LLM judge that splits the answer into claims and checks each against the context.

What is the difference between recall at k and precision at k? Recall at k is the fraction of the documents that should have been retrieved that appear in the top k. Precision at k is the fraction of the top k that are relevant. Low recall means the answer cannot be right; low precision means noise crowds out the right chunk.

Where do golden-set questions come from? From real traffic: production logs, support tickets, pilot chat history or the customer's FAQ. If the system is not live, ask the domain expert to write down the last fifty questions they were actually asked.

• LLM Evaluation series
• Evaluation Pipeline architecture
• LLM Evaluation topic hub
• Why Good Retrieval Still Produces Bad Answers
• The AI-Native SDLC (where the CI eval gate fits in the delivery lifecycle)
• The 90-Day Enterprise AI Adoption Playbook