RAG (Retrieval Augmented Generation)
A pattern that grounds LLM responses in retrieved documents, reducing hallucination and enabling citations.
Simple explanation
RAG means: before asking the LLM a question, you first search your documents for relevant chunks, then include those chunks in the prompt as context. The LLM then answers using that context. This lets the model answer questions about your private data without being trained on it.
Engineering explanation
RAG embeds the query, performs approximate nearest neighbour search against a vector index of chunked documents, optionally reranks candidates with a cross-encoder, injects the top-k into the prompt, and generates with citation metadata. Production RAG adds hybrid search (BM25 + vector), access-control metadata filtering at the SQL level, caching, query rewriting and online evaluation.
Example
User asks: 'What is our refund policy?' → embed query → search pgvector for policy chunks → retrieve top 5 → rerank to top 3 → inject into prompt → LLM answers with citations to the source chunks.
When to use it
When you need the LLM to answer from private, frequently-updated, or domain-specific data. When you need citations. When fine-tuning is too slow or expensive to update.
Common misconception
RAG does NOT make the model 'smarter' — it grounds the model in your data. A bad embedding model or poor chunking will produce bad RAG regardless of the LLM.