LLM Evaluation
Measure retrieval, generation, agents and production quality
Build evaluation systems for LLM applications: retrieval metrics, generation quality, agent trajectory eval, regression testing, online evaluation and eval-driven development.
Who this series is for
This series is for engineers who are changing prompts, models or retrieval settings and cannot say with confidence whether the change made things better. It assumes you have built at least a basic LLM or RAG application. Evaluation is what turns LLM development from trial and error into engineering, and it is the skill most often missing from teams shipping AI features.
What each chapter covers
- Evaluation Fundamentals covers building a small, representative test set, choosing metrics that match what users care about, and separating retrieval quality from generation quality.
- LLM-as-Judge covers using a model to grade outputs, its known biases (position, length, self-preference), and how to calibrate a judge against human labels before trusting it.
- Online Eval & Regression covers running evaluations in CI so a prompt change cannot silently break quality, and sampling production traffic to catch problems your test set missed.
How to approach it
Start with twenty to fifty hand-labelled examples from your own application. A small set you trust is worth more than a large generated one you do not. Re-run it after every change you make during the rest of the curriculum. The series itself is around seven hours.
Skills you'll gain
Prerequisites
Chapters
Why eval, golden sets, metrics, and the eval-driven development loop.
Using a model to score outputs at scale. Bias, calibration, and reliability.
CI eval gates, online sampling, drift detection and the eval-driven deploy loop.
Score agents on outcomes, the steps they take and what they cost, with sandboxed tasks, repeated runs and evaluation gates in CI.