Designing RAG Systems

Given a RAG requirement, produce five deliverables in order:

1. RAG Architecture — end-to-end diagram description (ingestion → chunking → embedding → index → retrieval → rerank → generation)
2. Index Strategy — vector DB choice, embedding model, index type, metadata schema
3. Retrieval Flow — query transformation, top-k, filters, reranking, context assembly
4. Storage Design — collections/namespaces, versioning, update strategy
5. Optimization Plan — evaluation metrics, tuning levers, cost/latency tradeoffs

Example prompt: "Design RAG for a legal document Q&A system with 50k contracts, updated weekly, needs citation accuracy."

Progress:

• Clarify data type, volume, update frequency, latency/accuracy requirements
• Choose chunking strategy based on document structure
• Select embedding model (domain fit, dimension, cost)
• Select vector database (scale, filtering needs, hosting constraints)
• Design retrieval pipeline (query rewrite, hybrid search, rerank)
• Design storage schema (metadata, namespaces, TTL/versioning)
• Define evaluation metrics and optimization loop
• Output the five sections

1. Chunking decision

• Structured docs (contracts, manuals) → section/heading-based chunking, 300–800 tokens, 10–15% overlap
• Unstructured/conversational → semantic chunking (embed sentences, cluster) or sliding window with overlap
• Code → function/class-level chunking
• Always attach metadata: source, section title, page, timestamp

2. Embedding strategy

• General domain, cost-sensitive → text-embedding-3-small or open-source (bge-small, e5-small)
• High accuracy, domain-specific → text-embedding-3-large, or fine-tuned bge/e5
• Multilingual → multilingual-e5-large or bge-m3
• Match query and document embeddings with same model; consider asymmetric models for short query / long doc

3. Vector DB selection

• Prototype / small scale (<1M vectors) → Chroma, pgvector
• Production, managed → Pinecone, Weaviate, Qdrant Cloud
• Self-hosted at scale → Qdrant, Milvus
• Need hybrid (keyword + vector) + strong filtering → Weaviate, Qdrant, Elasticsearch with vector plugin

4. Retrieval flow design

• Query transformation: HyDE, multi-query expansion, or query decomposition for complex questions
• Hybrid search: combine BM25/keyword with vector similarity (weighted fusion or RRF)
• Retrieve top 20–50 candidates, rerank down to top 3–8 with cross-encoder (e.g., bge-reranker, Cohere Rerank)
• Assemble context with source citations; enforce token budget

5. Storage design

• Separate namespaces/collections per document type or tenant
• Store raw text + metadata alongside vectors (never vector-only)
• Versioning: soft-delete old chunks on document update, re-embed only changed sections
• Include updated_at, source_id, chunk_index in metadata for traceability

6. Optimization plan

• Metrics: retrieval recall@k, MRR, faithfulness, answer relevance (RAGAS or custom eval set)
• Tuning levers: chunk size/overlap, top-k, reranker threshold, embedding model swap
• Cost/latency: cache frequent queries, batch embed on ingestion, async reranking
• Set up a golden Q&A eval set (30–50 pairs) to regression-test changes

Example 1: Input: "RAG for internal HR policy chatbot, 200 PDF documents, updated monthly, employees ask natural language questions."

Output:

RAG Architecture:
PDF ingestion -> text extraction (unstructured.io) -> heading-based chunking
-> embedding (text-embedding-3-small) -> pgvector store -> hybrid retrieval
(BM25 + vector, RRF fusion) -> cross-encoder rerank -> top-5 context -> LLM answer with citations

Index Strategy:
- Chunk size: 500 tokens, 15% overlap, split on headings/subheadings
- Embedding: text-embedding-3-small (cost-efficient, sufficient for policy text)
- Vector DB: pgvector (fits existing Postgres infra, <1M vectors)
- Metadata: doc_title, section, page_number, effective_date, department

Retrieval Flow:
- Query rewrite: expand acronyms (HR-specific glossary)
- Hybrid search: BM25 (keyword) + vector, combined via RRF
- Retrieve top 20 -> rerank with bge-reranker-base -> top 5
- Context assembly with inline citations [doc_title, page]

Storage Design:
- Single collection "hr_policies" with metadata filters by department
- Monthly re-ingestion: diff-based, only re-embed changed sections
- Soft-delete superseded chunks, keep effective_date for point-in-time queries

Optimization Plan:
- Eval set: 40 real HR questions with verified answers
- Track recall@5 and faithfulness monthly
- If recall low: reduce chunk size or increase top-k before rerank
- Cache top 100 frequent queries (1hr TTL)