Document Ingestion
Build a robust ingestion pipeline: multi-format extraction, cleaning, metadata, and idempotent updates.
What is ingestion?
Ingestion is the pipeline that turns raw documents (PDFs, HTML, Markdown, CSV, Word) into clean, structured, embeddable text with metadata. It runs once at index-build and incrementally on updates.
A good ingestion pipeline is: format-aware, idempotent (re-running doesn't duplicate), metadata-preserving (source, section, permission group, updated_at), and observable (tracks what was ingested, failed, skipped).
Why ingestion matters
Garbage in, garbage out. The #1 cause of bad RAG is not the embedding model or the LLM — it's poor ingestion. Tables extracted as gibberish, headers lost, multi-column PDFs read in the wrong order, metadata stripped. If your ingestion is bad, no amount of reranking will save you.
Ingestion architecture
Source → Extractor (format-specific) → Cleaner (normalize whitespace, fix encoding) → Structurer (sections, headers, tables) → Metadata enrich → Chunker → Embedder → Vector store (+ metadata).
Each stage is idempotent: re-ingesting the same document updates rather than duplicates. Track a content hash per document to detect changes.
Multi-format ingestion
import hashlib
from pathlib import Path
def content_hash(text: str) -> str:
return hashlib.sha256(text.encode()).hexdigest()[:16]
def ingest_pdf(path: str) -> list[dict]:
"""Extract text per page with PyMuPDF, preserving reading order."""
import fitz # PyMuPDF
doc = fitz.open(path)
pages = []
for i, page in enumerate(doc):
text = page.get_text("text") # respects reading order
pages.append({
"source": path,
"page": i + 1,
"text": text.strip(),
"hash": content_hash(text),
})
return pages
def ingest_html(path: str) -> list[dict]:
"""Extract main content from HTML, strip nav/footer."""
from bs4 import BeautifulSoup
html = Path(path).read_text()
soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style", "nav", "footer", "header"]):
tag.decompose()
text = soup.get_text(separator="\n", strip=True)
return [{
"source": path,
"page": 1,
"text": text,
"hash": content_hash(text),
}]
def ingest(path: str) -> list[dict]:
ext = Path(path).suffix.lower()
if ext == ".pdf":
return ingest_pdf(path)
if ext in (".html", ".htm"):
return ingest_html(path)
if ext == ".md":
return [{"source": path, "page": 1, "text": Path(path).read_text(), "hash": content_hash(Path(path).read_text())}]
raise ValueError(f"Unsupported format: {ext}")Experiment: extraction quality
Different extractors produce different text from the same PDF. See how the choice affects downstream retrieval.
What to observe
There is no universal PDF extractor. Choose based on document type: digital text → PyMuPDF; tables → pdfplumber; scanned → OCR. A production pipeline detects document type and routes to the right extractor. Never assume one extractor fits all.
Production ingestion
Production ingestion needs: content hashing for idempotency (re-ingest updates, doesn't duplicate), metadata extraction (source, section, permission group, updated_at), incremental updates (only re-embed changed docs), failure isolation (one bad doc doesn't break the batch), and observability (track ingested, failed, skipped counts).
Challenge
Your ingestion works on test docs but fails on 5% of production PDFs with cryptic errors. Design a pipeline that isolates failures, retries, and surfaces a report of what failed and why — without blocking the rest of the batch.
Production checklist
Production checklist
0 of 8 checked
Knowledge check
You re-run ingestion on an unchanged document. What should happen?
Complete
You can now ingest multi-format documents with idempotency and metadata. Next: chunking — turning extracted text into retrieval-friendly units.
Mark this chapter as complete
Track your progress and unlock the next chapter.