Why Good Retrieval Still Produces Bad Answers

A retrieval result is not the prompt, and the prompt is not the answer. Measure three things per question: which IDs retrieval returned, which IDs actually reached the final prompt, and whether each claim in the answer is supported by that evidence. The first boundary where evidence disappears tells you what to fix, and you change one boundary at a time.

A retrieval result is not the prompt. A prompt is not the answer. If your dashboard measures only whether a relevant document appeared in the search results, it cannot tell you whether that document survived context assembly or whether the answer used its evidence correctly.

Consider an internal support assistant. A fictional policy changed the return period from 14 days to 30 days. Search retrieves both versions, but an answer still says 14 days. There are several possible failures: the index contains only the old version, the correct version is ranked below the context cutoff, or both versions reach the model without an effective-date label. Each requires a different repair. Adding a reranker without tracing the evidence can leave the underlying error untouched.

This guide uses synthetic records, not customer data or a production benchmark. You can run the exercise without an API key. Its purpose is to make pipeline boundaries observable before spending money on model experiments.

Record the IDs returned by retrieval, the IDs actually placed into the final prompt, and the evidence supporting each factual claim in the answer. Keep document versions and chunk IDs stable so the trace can be reconstructed.

• Retrieval recall: relevant reference IDs retrieved divided by all relevant reference IDs. If there are no relevant references, report it as not applicable rather than perfect recall.
• Evidence retention: relevant reference IDs present in the final context divided by all relevant reference IDs. This catches truncation, filtering and context construction losses.
• Claim support: supported factual claims divided by all factual claims. A response with no claims gets no support score; a refusal should be evaluated separately.

For a real dataset, define the reference unit consistently. A document-level label can hide a missing paragraph. If two chunks must be combined to answer a question, annotate both and inspect whether the context contains the complete procedure.

Claim support is narrower than correctness. An answer can faithfully repeat an outdated document. It can also make a true statement that is absent from the provided context. Ragas documents faithfulness as consistency with retrieved context. Check current-policy correctness and usefulness separately.

Download the complete Python exercise, save it as rag-pipeline-audit.py, and run python3 rag-pipeline-audit.py with Python 3.9 or later. It uses only the standard library and makes no network requests. The script includes six cases and assertions for their expected diagnoses.

The core diagnostic function is:

def diagnose(case):
    gold = case["gold"]
    recall = len(gold & set(case["retrieved"])) / len(gold) if gold else None
    retained = len(gold & set(case["context"])) / len(gold) if gold else None
    claims = case["claims"]
    support = sum(claims) / len(claims) if claims else None
    if not case["answerable"]:
        stage = "correct refusal" if case["refused"] else "missing refusal"
    elif recall < 1:
        stage = "retrieval"
    elif retained < 1:
        stage = "context assembly"
    elif case["refused"]:
        stage = "unnecessary refusal"
    elif support is None or support < 1:
        stage = "generation / evidence"
    else:
        stage = "supported; check correctness separately"
    return recall, retained, support, stage

Expected output:

retrieval-miss: recall=0.0, retained=0.0, support=0.0 -> retrieval
context-dropped: recall=1.0, retained=0.0, support=0.0 -> context assembly
unsupported-answer: recall=1.0, retained=1.0, support=0.5 -> generation / evidence
supported-answer: recall=1.0, retained=1.0, support=1.0 -> supported; check correctness separately
correct-refusal: recall=None, retained=None, support=None -> correct refusal
unnecessary-refusal: recall=1.0, retained=1.0, support=None -> unnecessary refusal

The second case is the important counterexample: retrieval recall is perfect, yet the final prompt has none of the required evidence. An aggregate retrieval score would label this a success. Logging the context boundary reveals where to investigate.

The Boolean claim labels are supplied by hand. The program does not determine whether arbitrary natural-language claims are supported. For a real assistant, have reviewers label claim/evidence pairs, or evaluate a model-based judge against a human-labeled sample before trusting it. The script also reports the first failing boundary; several failures can coexist in one trace.

For a retrieval miss, check ingestion completeness, version filters, permissions and the reference labels first. Then compare lexical, vector or hybrid retrieval using the same questions. A search system cannot retrieve a paragraph that was never indexed. Do not weaken access controls to improve a retrieval score.

For a context-assembly failure, inspect the exact serialized prompt rather than the pre-template document list. Check your token budget, truncation order, deduplication and effective-date handling. Preserve complete procedures when splitting documents. Try a different chunk boundary only after identifying which information was lost; there is no universally correct chunk size or overlap.

For unsupported claims, retain citations at the claim level and test an instruction that asks the model to abstain when evidence is missing. A self-reported confidence number is not a calibrated probability. Evaluate refusals on answerable and unanswerable questions so a model that refuses everything cannot appear successful.

A cross-encoder can score query/document pairs after an initial retrieval step. Sentence Transformers describes this retrieve-and-rerank pattern. Measure quality and latency on your own corpus; this article does not claim a universal percentage improvement. Reranking also cannot recover a document outside its candidate set.

Start with a small reviewed set of questions covering common tasks, missing answers, outdated documents, multi-step procedures and permission boundaries. Separate a development set from a held-out set before tuning prompts. Keep the question IDs stable between runs.

For every change, record the model identifier, prompt version, document snapshot, retrieval configuration and full pipeline trace. Compare the same questions before and after the change. Review individual failures as well as averages, because a change that improves frequent questions may break rare but important ones.

Report refusal accuracy separately from claim support. Record retrieval and generation latency, request failures and token usage alongside quality. Decide acceptable thresholds based on the consequences of a wrong answer and the reviewed baseline, rather than copying an arbitrary faithfulness score from a tutorial.

Store traces with the same permissions and retention controls as the underlying documents. Prefer stable IDs and redacted excerpts when full prompts contain confidential material. An evaluation log should not become a second uncontrolled document store.

• Move policy-v2 into the second case's context. The first failure should move from context assembly to generation because the unsupported claim is still present.
• Change both claim labels in the third case to True. The result should become supported, but the script still cannot establish whether the source is current.
• Set refused=False on the unanswerable case. It should become a missing-refusal failure rather than receiving perfect recall for an empty reference set.
• Add a case with two required references and retrieve only one. Recall should become 0.5, making an incomplete multi-document answer visible.

These exercises test the diagnostic logic, not the performance of any language model. The next step is to replace the synthetic records with reviewed traces from your own application and investigate failures at the first boundary where evidence is lost.

Why does my RAG system retrieve the right document but give a wrong answer? Retrieval is only the first boundary. The right chunk can be dropped by truncation or filtering before the prompt is built, or reach the model and still be ignored or contradicted. Log the retrieved IDs, the IDs in the final prompt and claim support separately to see which boundary lost the evidence.

What is evidence retention? It is the share of relevant reference IDs that are present in the final context, out of all relevant reference IDs. It catches truncation, filtering and context-construction losses that a retrieval score hides.

What is claim support, and is it the same as correctness? Claim support is the share of factual claims in the answer that the provided evidence supports. It is not correctness: an answer can faithfully repeat an outdated document, or state something true that is absent from the context. Check correctness and usefulness separately.

Will adding a reranker fix this? Not necessarily. A cross-encoder can reorder a candidate set, but it cannot recover a document outside that set, and it does not fix context-assembly or generation failures. Measure quality and latency on your own corpus rather than assuming a gain.

How do I stop a model that refuses everything from looking good? Evaluate refusals on both answerable and unanswerable questions, and report refusal accuracy separately from claim support.

How do I make this a regression check? Keep a small reviewed question set with stable IDs, split a development set from a held-out set, and record the model, prompt version, document snapshot and retrieval configuration for every run so you can compare the same questions before and after a change.

• Production RAG series
• RAG system architecture
• Good Retrieval, Bad Answers challenge
• RAG topic hub
• Inside the AI-Augmented Developer Workflow (a retrieval-filter ticket that hits this exact class of failure)

September 11, 2026: Replaced unsupported universal precision-gain claims and scoring thresholds with a reproducible synthetic exercise. The earlier 0.91 recall and 0.42 faithfulness figures were illustrative, not measurements from a reported production system.