Retrieval

RAG System

Retrieval-augmented generation: ground the LLM in your private data by retrieving relevant chunks and injecting them into the prompt.

clientservicemodeldatabaseagentexternal

Explanation

The query is embedded and used to search a vector database for relevant chunks. Candidates are reranked by a cross-encoder, the top results become the LLM context, and the model generates a grounded answer with citations.

Components

Embedding modelVector database (pgvector/Pinecone)Reranker (Cohere/sentence-transformers)LLMIngestion pipelineChunking strategy

When to use

Knowledge base Q&A over private documents
Customer support assistants
Internal policy and procedure search
Codebase assistants

When NOT to use

When the answer must be exact (use a lookup, not generation)
When data is highly structured (use Text-to-SQL)
When latency budget is < 500ms

Failure modes

Poor chunking loses context
Embedding model mismatch between ingestion and query
Retrieval returns semantically similar but irrelevant chunks
Context window overflow from greedy top-k
Stale data not refreshed
No reranking — top-k by cosine similarity is often wrong

Production checklist

Evaluated retrieval (recall@k, precision@k)
Evaluated generation (faithfulness, answer relevance)
Reranking stage
Citation and source attribution
Stale data detection and refresh pipeline
Hybrid search (vector + keyword/BM25)
Query rewriting / HyDE
Observability on retrieval scores
Fallback when no relevant context retrieved