Why RAG?
Why retrieval-augmented generation beats fine-tuning and long-context for grounding LLMs in your data.
What is RAG?
Retrieval-Augmented Generation (RAG) grounds an LLM in your private data by retrieving relevant text chunks at query time and injecting them into the model's prompt. Instead of training the model on your data, you retrieve on-demand. The model reads the retrieved context and generates an answer — ideally with citations.
The alternative approaches are fine-tuning (expensive, slow to update, no citations) and stuffing everything into a long context window (expensive per query, doesn't scale, latency grows). RAG is the production default because it's cheap, updatable, auditable and grounds answers in citable sources.
Why RAG matters
Without RAG, an LLM only knows what it was trained on. It cannot answer questions about your private documents, your codebase, your policies. It will also confidently hallucinate.
RAG turns a general-purpose LLM into a system that answers from YOUR data, with citations, that you can update by re-indexing (minutes) rather than re-training (weeks). It is the single most impactful pattern in production generative AI.
Where RAG fits
A RAG system sits between your data stores and the LLM:
Data sources → Ingestion pipeline → Chunks → Embeddings → Vector store
↓
User query → Query embed → Vector search → Rerank → Top-k context → LLM → Answer + citations
The ingestion pipeline runs offline (or on update). The query path runs per-request. Production RAG adds: query rewriting, hybrid search, reranking, caching, RBAC filtering, observability and eval.
Minimal RAG
from openai import OpenAI
import numpy as np
client = OpenAI()
# --- Indexing (once, or on update) ---
documents = [
"Customers can cancel within 14 days for a full refund.",
"Refunds are processed within 5-7 business days.",
"Premium plan includes priority support and unlimited seats.",
]
def embed(text: str) -> list[float]:
r = client.embeddings.create(model="text-embedding-3-small", input=text)
return r.data[0].embedding
doc_embeddings = [embed(d) for d in documents]
# --- Query (per request) ---
def rag_answer(question: str) -> str:
q_emb = embed(question)
# Cosine similarity
scores = [np.dot(q_emb, d) / (np.linalg.norm(q_emb) * np.linalg.norm(d))
for d in doc_embeddings]
# Top 2
top = sorted(zip(scores, documents), reverse=True)[:2]
context = "\n\n".join(d for _, d in top)
prompt = f"Answer using only this context. Cite sources.\n\nContext:\n{context}\n\nQuestion: {question}"
r = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
)
return r.choices[0].message.content
print(rag_answer("How long do refunds take?"))
# → "Refunds are processed within 5-7 business days. [Source 2]"Experiment: retrieval vs no retrieval
Compare answering with and without retrieved context. See how retrieval grounds the answer.
What to observe
Notice that retrieval grounds answers in YOUR data with citations. Without retrieval, the model guesses from training data — fine for world knowledge, dangerous for your business policies. The value of RAG is grounding + citation + updatability, not 'making the model smarter'.
Production considerations
The naive implementation above is not production-ready. In production you need: hybrid search (vector + keyword), reranking, metadata filtering for access control, chunking strategy, caching, query rewriting, observability on retrieval scores, and evaluation. Each is covered in subsequent chapters.
Challenge
Your RAG system returns relevant chunks but the answer is wrong. Where would you look first — embeddings, chunking, reranking, or the prompt? (Hint: chunking is the #1 cause of bad RAG.) See the 'Good Retrieval, Bad Answers' challenge for a worked example.
Production checklist
Production checklist
0 of 9 checked
Knowledge check
Your team wants the LLM to know about a new product launch. Which is the fastest way to make it answerable?
Complete
You now understand why RAG is the production default for grounding LLMs in private data. Next: document ingestion — the first step of any RAG pipeline.
Mark this chapter as complete
Track your progress and unlock the next chapter.