RAGIntermediate

Good Retrieval, Bad Answers

Your RAG system over internal docs has excellent retrieval scores (recall@5 = 0.91) but users complain the answers are wrong, vague or cite the wrong source. Diagnose the problem.

Symptoms

Retrieval recall@5 is 0.91 — top chunks are relevant
Users report answers 'miss the point'
Citations sometimes point to the wrong chunk
Answer quality drops for long questions

Evidence

Query

What is our policy for handling customer data subject access requests under GDPR?

Retrieved chunks (top 3)

Chunk A (score 0.89): 'Customer data is stored encrypted at rest...'
Chunk B (score 0.86): 'GDPR grants users the right to access their data...'
Chunk C (score 0.84): 'Our data retention policy is 24 months...'

Generated answer

Customer data is encrypted at rest and our retention policy is 24 months. Users have the right to access their data.

Latency

embed: 80ms · search: 40ms · generate: 1200ms · total: 1320ms

Eval scores

faithfulness: 0.42 · answer_relevance: 0.55 · context_relevance: 0.88

Tasks

1Identify the root cause of the answer quality gap
2Explain why retrieval is good but generation is poor
3Propose two concrete fixes

Your diagnosis