Production RAG Architecture: A Complete Guide

Production RAG is not "embed, search, generate". It is a pipeline of eight stages (ingestion, chunking, embedding, hybrid retrieval, reranking, grounded generation, evaluation and observability) wrapped in access control and cost management. Most failures come from the first four stages, not from the model, and you cannot improve what you do not measure, so build the evaluation set before you tune anything. The numbers in this guide (chunk sizes, top-k, thresholds) are starting points to test on your own data, not universal constants.

Retrieval-augmented generation (RAG) answers a question by first searching your own documents, then asking a language model to answer using only what was found. The production version adds everything a demo skips: repeatable ingestion, versioned embeddings, hybrid retrieval, reranking, citations, refusal when the context is insufficient, evaluation, tracing, permissions and cost control. The reason to build it is grounding: answers that come from your current documents, with sources a user can check, instead of from whatever the model memorised.

Data Sources → Ingestion → Chunking → Embedding → Vector Store + Keyword Index
                                                          ↓
User Query → Query Rewrite → Hybrid Search (Vector + BM25) → RRF Fusion → Reranking
                                  → Context Construction → LLM → Answer + Citations
                                                          ↓
                                            Observability + Evaluation

Ingestion decides what the system can ever know, so treat it as a data pipeline, not a script. Extract text from every format you have (PDF, HTML, Markdown, spreadsheets), strip boilerplate such as navigation and footers, and deduplicate near-identical documents so one policy does not occupy five of your top five results. Attach metadata to every document at this point: source, section, a permission group, and an updated date. You cannot add it later without re-reading everything.

Make ingestion idempotent. Store a content hash per document, and only re-chunk and re-embed what changed. That makes re-runs cheap, lets you detect deletions, and keeps the index from drifting away from the source of truth. Two failure modes are worth testing early: scanned PDFs that extract as empty text, and tables that flatten into unreadable lines.

The document ingestion chapter covers extraction and cleaning in depth.

Chunking is usually the highest-leverage decision in a RAG system, because a chunk is the unit that gets retrieved, ranked and shown to the model. Too large and the relevant sentence is buried in noise. Too small and the answer is split across pieces that never appear together.

Split by document structure first (headings, then paragraphs, then sentences) and by size only as a fallback. A common starting point is a few hundred tokens per chunk with a small overlap; 512 tokens with 64 overlap is a reasonable first experiment, not a rule. Keep the heading path with each chunk ("Refund policy > Digital goods") so a chunk is understandable on its own, and consider storing a small child chunk for retrieval but returning its larger parent for context.

Test your choice rather than trusting a default: change the chunk size, re-run your golden set, and compare retrieval recall. The chunking strategies chapter and the document chunking lab let you do exactly that.

An embedding model maps each chunk to a vector so that similar meanings land near each other. Three practices matter in production. First, pin the embedding model version and record it with every vector: queries must be embedded with the same model as the documents, and swapping models means re-embedding the whole corpus. Second, store vectors alongside the metadata you will filter on, so a query can restrict by permission group, date or document type. Third, use an approximate nearest neighbour index such as HNSW for speed, and know that it trades a little recall for latency, so measure recall on your own data.

For many teams, a Postgres database with the pgvector extension is the right starting point: one system, SQL filters, and transactional consistency with your metadata. Move to a dedicated vector database when scale or latency demands it, not before. The embeddings and vector search chapter and lab go deeper, and the embedding model drift challenge presents a search-quality drop after an embedding model upgrade.

Vector search finds paraphrases: "how do I get my money back" matches "refund policy". It is weak on exact strings such as policy numbers, error codes and product names. Keyword search (BM25) is the opposite. Running both and fusing the results catches both kinds of query.

A simple, robust way to fuse is Reciprocal Rank Fusion (RRF): give each document a score of one over (a constant plus its rank) in each list, and add the scores. It needs no score normalisation, which is the hard part of combining two different scoring systems.

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    """Fuse several ranked lists of document ids. k=60 is a common default."""
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

candidates = rrf([vector_search(query, top=50), bm25_search(query, top=50)])[:20]

Rewriting the query before retrieval helps with conversational follow-ups ("what about for digital goods?"): turn the chat history plus the new message into one self-contained search query. Keep the original wording as well, since a rewrite can drop a rare keyword. The hybrid search pattern shows the full design.

First-stage retrieval is fast but coarse. A cross-encoder reranker reads the query and each candidate together and scores how well they match, which is more accurate and also slower, so you apply it only to a short list. The usual shape is to retrieve a wide set (for example 20 to 50), rerank, and pass the top few (often three to five) to the model.

Reranking commonly improves precision, but how much depends on your data, so measure it: compare recall and answer quality with and without it on your golden set, and weigh the added latency and cost. Rerankers are available as hosted APIs and as open-source models. The retrieval and reranking chapter and reranking lab walk through it.

Generation is where retrieval quality turns into a user-visible answer, and where the prompt has to enforce honesty. Put the instruction that matters most first and be explicit: answer only from the text inside the context tags, say "I don't know" when it is not there, and never invent a source.

Do not ask the model to write its own citations. Give every chunk an id, ask for the ids it used in a structured output field, and build the citation from your own metadata. That makes citations verifiable and lets you test them automatically. Treat retrieved text as data, not instructions: a document that says "ignore previous instructions" is a prompt-injection attempt, and the prompt injection via RAG challenge shows how it happens.

You cannot tune a RAG system by reading outputs. Build a golden set first: a few dozen to a few hundred real questions, each with the document or passage that should answer it, sampled from real traffic or from the people who will use the system. Then measure the pipeline in two parts.

Wire the golden set into CI so a change that lowers a score fails the build. Choose thresholds from your own baseline; something like "fail if a metric drops more than 5 percent" is a sensible starting rule, and a faithfulness target such as 0.85 is a goal to calibrate, not a standard. The evaluation pipeline lab builds this end to end, and LLM evaluation fundamentals explains the metrics.

Production RAG fails quietly, so make it loud. Trace every request through every stage (rewrite, retrieval, rerank, generation) with OpenTelemetry or your tracing tool of choice. Log latency per stage, token cost, the retrieved chunk ids and their scores. Then sample a small share of live traffic, for example one percent, for online evaluation, and alert when retrieval scores or refusal rates drift. When an answer is wrong, a trace should tell you in a minute whether retrieval or generation was to blame. The observability lab shows how.

Access control. Filter by permission group inside the retrieval query, so restricted chunks are never retrieved, never reranked and never shown to the model. Filtering after generation is too late, because the text has already influenced the answer.

Cost. Track tokens and spend per request and per user, set budgets, and cache. Semantic caching can answer repeated questions without a model call, at the risk of stale answers, so key it on the document version. How to reduce LLM costs covers the levers.

Freshness. Decide how quickly a changed document must reach the index, and test deletion: a removed document must stop appearing.

• Hybrid search (vector and BM25 with RRF fusion)
• Cross-encoder reranking, measured against no reranking
• Structure-aware chunking, with chunk size tested on your data
• Pinned embedding model version stored with every vector
• Access control enforced in the retrieval query
• Structural citations built from metadata, not model-generated
• "I don't know" refusal when context is insufficient
• Golden set in CI with regression gates calibrated to your baseline
• Tracing on every stage
• Cost monitoring and per-user budgets
• Semantic caching keyed on document version
• Online evaluation sampling

What is production RAG architecture? It is a multi-stage pipeline: ingestion, chunking, embedding, hybrid retrieval, reranking, grounded generation, evaluation and observability, plus access control and cost management. The goal is answers that come from your current documents, with citations a user can verify.

What is the best chunk size for RAG? There is no universal best. A few hundred tokens with a small overlap, split along headings and paragraphs, is a sensible starting point. Test several sizes against a golden set and keep the one with the best retrieval recall for your documents.

Do I need hybrid search, or is vector search enough? Vector search alone misses exact identifiers such as policy numbers and error codes. Combining it with keyword search (BM25) and fusing the results with Reciprocal Rank Fusion catches both paraphrased and exact queries, which is why hybrid is the common production choice.

Is reranking worth the extra latency? Often, but measure it. Reranking applies a more accurate cross-encoder to a short candidate list, which commonly improves precision. Compare answer quality and latency with and without it on your own golden set before committing.

How do I stop a RAG system from hallucinating? Instruct the model to answer only from the supplied context and to say it does not know otherwise, build citations from chunk metadata rather than model output, and measure faithfulness on a golden set. Better retrieval removes the most common cause, which is missing or wrong context.

How do I evaluate a RAG system? Build a golden set of real questions with the passages that should answer them. Measure retrieval (recall at k, context relevance) and generation (faithfulness, answer relevance) separately, run them in CI, and sample live traffic for online evaluation.

How should I handle permissions in RAG? Enforce them in the retrieval query with a metadata filter on a permission group, so restricted content is never retrieved or shown to the model. Filtering the answer afterwards is too late.

When should I move from pgvector to a dedicated vector database? When measured scale, latency or filtering needs exceed what your Postgres setup delivers. Until then, one system with SQL filters and transactional consistency is simpler to operate.

• Production RAG series
• RAG System architecture
• Hybrid Search architecture
• RAG topic hub
• Why good retrieval still produces bad answers
• The Enterprise AI Engineering Blueprint (how RAG fits into the full enterprise stack)
• Inside the AI-Augmented Developer Workflow (a retrieval-filter ticket walked through end to end)