EvaluationAdvanced

Eval Regression After Model Upgrade

You upgraded gpt-4o to gpt-4o-2025-01. Production metrics look fine but eval scores dropped 15%. Users haven't complained yet but you're worried.

Symptoms

Eval faithfulness dropped 0.81 → 0.69
Production latency improved 200ms
Users haven't complained
Some answers are now longer and more confident-sounding

Evidence

Eval diff

faithfulness: 0.81 → 0.69
answer_relevance: 0.78 → 0.76
context_relevance: 0.90 → 0.91
length: 180 → 340 tokens avg

Sample regression

Q: 'Can I export my data?'
Old: 'Yes, via Settings > Export. CSV and JSON supported.' (faithful)
New: 'Yes, you can absolutely export your data! Navigate to Settings, then Export. We support CSV, JSON, and PDF formats, and you can schedule recurring exports too.' (PDF and recurring exports are NOT supported — hallucinated)

Tasks

1Explain why faithfulness dropped
2Decide: roll back or fix forward?
3Improve the eval to catch this earlier

Your diagnosis