All patterns
Quality
Evaluation Pipeline
Continuous evaluation of LLM outputs in CI and production to catch regressions before users do.
clientservicemodeldatabaseagentexternal
Explanation
Every code change runs an eval against a golden dataset. An LLM judge scores outputs on faithfulness, relevance and correctness. Reports gate the deploy. In production, a sample of traffic is scored online to detect drift.
Components
Golden datasetEval runnerLLM-as-judge scorerRegression gateOnline sampling + scoringDashboards
When to use
- Any production LLM system
- Before prompt or model changes
- RAG systems with measurable retrieval
When NOT to use
- Early prototyping with no golden data
- Pure novelty generation (hard to score)
Failure modes
- Golden set drifts from real traffic
- LLM judge aligned with the system under test (circular eval)
- Eval passes but UX regresses
- Eval too slow — devs skip it
Production checklist
- Golden set curated and versioned
- Judge model different from system model
- Human review of judge disagreements
- Eval runs in < 5 minutes
- Online eval on sampled traffic
- Regression threshold gates deploy