Online Eval & Regression
CI eval gates, online sampling, drift detection and the eval-driven deploy loop.
Online eval & regression
CI eval catches regressions before deploy by running the golden set on every PR. Online eval catches drift in production by sampling real traffic, scoring it, and alerting on score drops. Together they form the eval-driven deploy loop: change → CI eval → gate → deploy → online eval → alert → fix.
Why both CI and online
CI eval catches known regressions (golden set). Online eval catches UNKNOWN regressions — production queries the golden set doesn't cover, model provider silent changes, data drift. You need both: CI for known, online for unknown.
Eval-driven deploy loop
PR opened → CI runs eval → if regression > threshold, block merge → if pass, deploy → production serves traffic → sample 1% → score with judge → if score drops > threshold, alert + auto-rollback → fix → repeat. The loop runs continuously.
CI eval gate + online sampling
# .github/workflows/eval.yml
# name: Eval Gate
# on: pull_request
# jobs:
# eval:
# runs-on: ubuntu-latest
# steps:
# - uses: actions/checkout@v4
# - run: pip install -r requirements.txt
# - run: python -m eval.run --gate --threshold 0.05
# eval/run.py
import sys
from eval import run_eval, gate_deploy, load_golden, load_baseline
def main():
golden = load_golden("eval/golden_v3.json")
baseline = load_baseline("eval/baseline_v3.json")
candidate = run_eval(golden, prompt_version="candidate")
print(f"Candidate: faith={candidate.faithfulness:.2f} rel={candidate.answer_relevance:.2f}")
if "--gate" in sys.argv:
threshold = float(sys.argv[sys.argv.index("--threshold") + 1])
if not gate_deploy(baseline, candidate, threshold):
print("DEPLOY BLOCKED — regression detected")
sys.exit(1)
print("Deploy approved")
# Online eval — samples production traffic
from collections import deque
import random
recent_scores = deque(maxlen=1000)
ALERT_THRESHOLD = 0.08 # 8% drop from baseline
async def score_online_sample(query, answer, context):
"""Score 1% of production traffic."""
if random.random() > 0.01:
return
score = await judge_faithfulness(answer, context)
recent_scores.append(score.score)
# Drift detection
if len(recent_scores) >= 100:
avg = sum(recent_scores) / len(recent_scores)
baseline = 0.82
if baseline - avg > ALERT_THRESHOLD:
await alert(f"Eval drift: {baseline:.2f} → {avg:.2f}")
await trigger_rollback()Experiment: eval-driven loop
See how the eval-driven loop catches different failure types.
What to observe
CI eval catches known regressions early (at PR). Online eval catches unknown regressions late (post-deploy) but covers failures CI can't see: new queries, silent model changes, data drift. You need BOTH. CI-only misses the unknown; online-only lets known regressions reach users.
Production eval loop
Production eval loop: CI gate on every PR (blocks known regressions), online eval on 1% of traffic (catches unknown), drift alerts with auto-rollback, weekly golden-set refresh from real traffic, and quarterly judge recalibration. The loop runs forever — eval is not a one-time setup.
Challenge
Your eval passes in CI but production quality drops 10% over a month. The model hasn't changed, your code hasn't changed. What changed, and how do you detect it earlier?
Production checklist
Production checklist
0 of 8 checked
Knowledge check
Your CI eval passes but production quality drops. What failed?
Complete
You can now build an eval-driven deploy loop with CI + online eval. This completes the LLM Evaluation series.
Mark this chapter as complete
Track your progress and unlock the next chapter.