How do you evaluate a RAG system? What metrics matter?
Think first
Think about: retrieval metrics vs generation metrics, golden sets, LLM-as-judge, online eval.
Strong answer
RAG eval splits into retrieval and generation. Retrieval metrics: recall@k (did the relevant chunk appear in top-k?), precision@k (how many of top-k are relevant?), MRR (mean reciprocal rank). Generation metrics: faithfulness (is every claim supported by context? — use LLM-as-judge), answer_relevance (does the answer address the question?), context_relevance (is the retrieved context relevant?). Build a golden set from real traffic (curated, versioned). Run eval in CI on every PR — gate deploy if any metric drops > 5%. Use a DIFFERENT judge model from the system model (avoids self-preference). Add online eval: sample 1% of production traffic, score with the judge, alert on drift. The judge must be calibrated against human ratings (correlation > 0.7).
Weak-answer indicators
- Only mentions 'accuracy'
- Doesn't distinguish retrieval vs generation metrics
- Skips LLM-as-judge bias mitigation
- No mention of golden set or CI gating
Likely follow-up questions
- What is LLM-as-judge and what are its biases?
- How do you handle golden set drift?
- What's the difference between CI eval and online eval?