Embeddings & Vector Search
What embeddings capture and miss, how vector indexes trade recall for speed, and how to choose and change an embedding model safely.
Embeddings & vector search
An embedding model turns a piece of text into a list of numbers, a vector, so that texts with similar meaning land close together. "How do I get my money back?" and "refund process" share no words, but their vectors sit near each other. Vector search finds the stored vectors nearest to a query's vector, usually by cosine similarity.
Embeddings are good at meaning, paraphrase and topic. They are weak at exact identifiers (POL-4412, fetchUserById), numbers and dates, negation ("not covered" sits close to "covered"), rare jargon the model never saw, and anything that depends on the precise wording. That is why production retrieval combines vector search with keyword search rather than replacing it.
Many modern embedding models are asymmetric: a short query and a long passage are embedded differently, often with an instruction or prefix such as "query:" and "passage:". Using the wrong prefix, or a different model for queries than for documents, quietly ruins retrieval.
Why it matters
Retrieval sets the ceiling for a RAG system. If the right passage is not in the top results, no prompt or model can answer correctly. Many "the model hallucinated" bugs are retrieval misses.
Public leaderboards are a starting point, not an answer: a model that ranks well on general benchmarks can do poorly on your contracts, tickets or code. The only reliable way to choose is to measure recall on a few hundred labelled queries from your own domain. The same applies to index settings, which trade recall for speed and memory in ways that only show up at your data size.
The pipeline and the index
Write path: chunk → embed in batches → store the vector with its text, metadata (source, permissions, date) and the embedding model's name and version → build or update the index.
Read path: embed the query with the same model, version and query prefix → approximate nearest-neighbour (ANN) search for the top 20–50 → fuse with keyword results → rerank to the top 5.
Index choices:
Two details cause many production bugs. First, the distance function must match the model: if vectors are normalised, cosine similarity and dot product rank the same way, but mixing metrics does not. Second, metadata filters such as permissions interact with ANN: filtering after the search can return too few results, so use an index that filters during the search, or over-fetch and check that enough results remain.
Embed, index and measure recall
from dataclasses import dataclass
import numpy as np
from sentence_transformers import SentenceTransformer
# Example open model; swap for any embedding model you are evaluating.
MODEL_NAME = "BAAI/bge-small-en-v1.5"
MODEL_VERSION = f"{MODEL_NAME}@2026-10"
model = SentenceTransformer(MODEL_NAME)
def embed_passages(texts: list[str]) -> np.ndarray:
# normalize_embeddings=True makes cosine similarity equal to a dot product.
return model.encode(texts, batch_size=64, normalize_embeddings=True)
def embed_query(text: str) -> np.ndarray:
# This model family expects an instruction prefix on queries only.
prefix = "Represent this sentence for searching relevant passages: "
return model.encode([prefix + text], normalize_embeddings=True)[0]
# Store vectors with the model version so a future re-embedding is traceable:
# CREATE TABLE chunks (id text PRIMARY KEY, text text, source text,
# permission_group text, embedding vector(384), embedding_model text);
# CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops)
# WITH (m = 16, ef_construction = 64);
# SET hnsw.ef_search = 100; -- raise for recall, lower for speed
@dataclass
class LabelledQuery:
query: str
relevant_ids: set[str] # chunk ids a human marked as answering the query
def recall_at_k(search, queries: list[LabelledQuery], k: int = 10) -> float:
"""Share of queries where at least one relevant chunk is in the top k."""
hits = 0
for q in queries:
found = {r["id"] for r in search(q.query, k)}
hits += bool(found & q.relevant_ids)
return hits / len(queries)
def compare(searches: dict, queries: list[LabelledQuery]) -> None:
for name, search in searches.items():
print(f"{name:<24} recall@10 = {recall_at_k(search, queries):.2f}")
# compare({"exact": exact_search, "hnsw ef_search=40": hnsw_40,
# "hnsw ef_search=200": hnsw_200, "hybrid + rerank": hybrid}, labelled)Experiment: index settings and scale
See how the index type and data size change the trade-off between recall, speed and memory. The figures are illustrative ranges; measure your own data before deciding.
What to observe
There is no free setting: every index trades recall, latency and memory against each other, and the right point depends on your size and on how costly a missed passage is. Always keep exact search available offline as the ground truth for measuring an approximate index.
Embeddings in production
Changing embedding models is a migration, not a config change: old and new vectors are not comparable. Run it like a database migration. Write new vectors to a separate column or index (tagged with the model version), backfill in batches, compare recall on your labelled set, switch reads only when the new model wins, and keep the old index until you are sure.
Watch retrieval health continuously: the share of queries whose best match scores below a threshold, the share returning no results after permission filters, and the distribution of similarity scores over time. A sudden shift usually means a model, prefix or preprocessing change somewhere.
Budget for embedding cost: a one-off backfill plus every new or edited document. Reduce storage with smaller dimensions (some models support shortening vectors) or quantisation, and cache query embeddings for repeated questions.
Challenge
Your knowledge base has 20 million chunks embedded with last year's model. A newer model scores 8 points higher on recall@10 in your offline test. Plan a migration with no downtime: how you store both sets of vectors, how you backfill, what evaluation gates the switch, how you roll back, and what it costs.
Production checklist
Production checklist
0 of 9 checked
Knowledge check
You upgrade to a better embedding model for new documents only, keeping old vectors. What happens?
Complete
You can now choose, measure and operate embeddings and vector indexes. Next: combine vector search with keyword search and a reranker in Retrieval & Reranking, or practise in the Embeddings & Vector Search lab.
Mark this chapter as complete
Track your progress and unlock the next chapter.