Retrieval & Reranking
Hybrid search, reranking, and the production retrieval stack that produces precise context.
Retrieval & reranking
Retrieval finds candidate chunks relevant to a query. Reranking re-scores those candidates with a more expensive, more accurate model to pick the best top-k.
Two-stage retrieval is the production default: (1) fast approximate search (vector + BM25) returns top-20 candidates; (2) a cross-encoder reranker scores each candidate against the query jointly and returns top-5. This gives near-brute-force quality at a fraction of the cost.
Why two stages
Vector search alone (cosine similarity) is fast but imprecise — it ranks semantically-similar-but-irrelevant chunks highly. A cross-encoder reranker (which reads query AND document together) is far more accurate but too slow to run over the whole index. Two-stage: fast retrieval for candidates, slow reranker for the shortlist. Typical precision gain: 15-30%.
Production retrieval stack
Query → (optional) query rewrite → parallel: vector search (top-20) + BM25 (top-20) → RRF fusion → reranker (cross-encoder, top-5) → context for LLM. Add: metadata filter for RBAC at every stage, caching of embeddings and reranker results, observability on scores at each stage.
Hybrid search + reranking
import psycopg2
from pgvector.psycopg import register_vector
import cohere
co = cohere.Client()
conn = psycopg2.connect("dbname=rag")
def vector_search(query_embedding: list[float], k: int = 20, filters: dict = None) -> list[dict]:
"""ANN vector search with metadata filter at SQL level."""
register_vector(conn)
cur = conn.cursor()
# Filter is in the WHERE clause — RBAC enforced at query, not display
where = "WHERE permission_group = ANY(%s)" if filters else ""
params = [filters["groups"]] if filters else []
params.append(query_embedding)
cur.execute(f"""
SELECT id, text, source, section, embedding <=> %s AS distance
FROM chunks
{where}
ORDER BY embedding <=> %s
LIMIT %s
""", [query_embedding, query_embedding, k])
return [{"id": r[0], "text": r[1], "source": r[2], "section": r[3], "score": 1 - r[4]} for r in cur.fetchall()]
def bm25_search(query: str, k: int = 20, filters: dict = None) -> list[dict]:
"""Full-text search using PostgreSQL tsvector."""
cur = conn.cursor()
where = "AND permission_group = ANY(%s)" if filters else ""
params = [query] + ([filters["groups"]] if filters else []) + [k]
cur.execute(f"""
SELECT id, text, source, section,
ts_rank(tsv, plainto_tsquery(%s)) AS rank
FROM chunks
WHERE tsv @@ plainto_tsquery(%s) {where}
ORDER BY rank DESC LIMIT %s
""", params)
return [{"id": r[0], "text": r[1], "source": r[2], "section": r[3], "score": r[4]} for r in cur.fetchall()]
def rrf_fuse(vector_results: list[dict], bm25_results: list[dict], k: int = 60) -> list[dict]:
"""Reciprocal Rank Fusion — combine rankings without score calibration."""
scores: dict[str, float] = {}
for rank, r in enumerate(vector_results):
scores[r["id"]] = scores.get(r["id"], 0) + 1 / (k + rank + 1)
for rank, r in enumerate(bm25_results):
scores[r["id"]] = scores.get(r["id"], 0) + 1 / (k + rank + 1)
return sorted(scores.items(), key=lambda x: -x[1])
def rerank(query: str, candidates: list[dict], top_k: int = 5) -> list[dict]:
"""Cross-encoder rerank — reads query + doc jointly."""
docs = [c["text"] for c in candidates]
results = co.rerank(model="rerank-english-v3.0", query=query, documents=docs, top_n=top_k)
return [{"text": docs[r.index], "score": r.relevance_score} for r in results.results]
def retrieve(query: str, query_embedding: list[float], filters: dict = None) -> list[dict]:
v = vector_search(query_embedding, k=20, filters=filters)
b = bm25_search(query, k=20, filters=filters)
fused = rrf_fuse(v, b)
candidates = [{"id": cid, "text": next(c["text"] for c in v+b if c["id"]==cid)} for cid, _ in fused[:20]]
return rerank(query, candidates, top_k=5)Experiment: retrieval strategy
Compare vector-only, BM25-only, and hybrid+rerank on different query types.
What to observe
Vector search wins on semantic/paraphrased queries. BM25 wins on exact identifiers (policy numbers, code names, article numbers). Hybrid + rerank is robust across all query types — that's why it's the production default. Never ship vector-only over mixed content.
Production retrieval
Production retrieval: hybrid (vector + BM25) with RRF fusion, cross-encoder reranking to top-5, metadata filtering for RBAC at SQL level (not display), query rewriting for multi-turn conversations, caching of embeddings and rerank results, and observability on scores at every stage. Online eval samples retrieval quality in production.
Challenge
A user searches for 'POL-4412 cancellation window'. Vector search returns chunks about cancellation but misses the policy number. BM25 returns the policy but ranks an outdated version first. Design a retrieval system that returns the current version of POL-4412 with its cancellation window.
Production checklist
Production checklist
0 of 8 checked
Knowledge check
Why is the metadata (RBAC) filter applied in the SQL WHERE clause, not after retrieval?
Complete
You now have the full production retrieval stack. This completes the Production RAG series. Next step: build the RAG Assistant project, or explore evaluation.
Mark this chapter as complete
Track your progress and unlock the next chapter.