Design a RAG system for a bank with 60,000 internal documents and 18,000 employees with role-based access control.
Think first
Think about: ingestion, chunking, embeddings, access control, retrieval, reranking, citation, audit, cost, rollout.
Strong answer
Ingestion: separate pipelines for PDF (OCR + table extraction), Confluence (HTML), tickets (CSV). Section-aware chunking, 512 tokens, 64 overlap. Embed with 1024-dim model, store in pgvector with metadata (source, permission_group, updated_at, doc_id). HNSW index. Retrieval: hybrid (vector + BM25 with RRF fusion), rerank top-20 to top-5 with Cohere. Access control: CRITICAL — filter at the SQL WHERE clause by user's permission groups, NOT at display time. The model must never see restricted content. Citation: structural (return chunk IDs, render citations from metadata) — never model-generated. Confidence: low-confidence escalation to human reviewer. Audit: append-only log of every query + answer + retrieved chunks + model + cost, 7-year retention. Cost: semantic cache (0.95 threshold) + model routing (small model default) + per-user budget → target $0.03/query. Observability: trace every query, online eval sample. Rollout: shadow mode → pilot one team → expand team-by-team with rollback runbook.
Weak-answer indicators
- Filters access control at display time (model already saw restricted content)
- Skips audit logging
- No mention of rollout strategy
- Uses model-generated citations
- Doesn't address cost ceiling
Likely follow-up questions
- How do you handle policy updates — how fast must new content be searchable?
- What's your SLA for answer latency?
- How do you measure user satisfaction?