Building RAG Systems

Minimal RAG pipeline (LangChain + Qdrant):

Python
from langchain_community.vectorstores import Qdrant
from langchain_openai import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyPDFLoader

# 1. Load & chunk
docs = PyPDFLoader("report.pdf").load()
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)
chunks = splitter.split_documents(docs)

# 2. Embed & index
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
vectorstore = Qdrant.from_documents(chunks, embeddings, location=":memory:", collection_name="docs")

# 3. Retrieve + generate
retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
results = retriever.invoke("What are the key findings?")
context = "\n\n".join(f"[{i+1}] {r.page_content}" for i, r in enumerate(results))

prompt = f"""Answer using ONLY the context below. Cite sources as [N].
Context:
{context}

Question: What are the key findings?"""

Progress:

• Step 1: Ingest & parse documents (PDF, HTML, DB, API)
• Step 2: Chunk text (256–1024 tokens, 10–20% overlap)
• Step 3: Generate embeddings, load into vector DB
• Step 4: Design retrieval strategy (hybrid: dense + sparse)
• Step 5: Rerank results (cross-encoder)
• Step 6: Assemble prompt (context + question → LLM)
• Step 7: Post-process, attach citations, verify grounding

Step details

1. Ingestion & parsing

• Normalize all sources to clean text; preserve structure (headings, tables) as metadata.
• Keep source, page, section in metadata for every chunk — required for citations.

2. Chunking

• Default: 512 tokens, 15% overlap. Smaller (256) for dense FAQ-like content; larger (1024) for narrative/legal text.
• Prefer semantic/recursive splitting (by headers, then paragraphs, then sentences) over fixed-size blind splits.
• Attach parent-document reference for "small-to-big" retrieval when needed.

3. Embeddings & indexing

• Choose embedding model by language/domain: text-embedding-3-large (general, multilingual), bge-large-ru (Russian-heavy corpora), Cohere embed v3 (multilingual + reranking ecosystem).
• Store vectors with metadata filters (date, doc type, access level) in Milvus/Qdrant/Weaviate/Chroma/pgvector. Default choice: Qdrant for most projects (speed + filtering + easy ops), pgvector if already on Postgres.

4. Retrieval strategy

• Default to hybrid search: dense vector similarity + BM25/sparse keyword search, combined via reciprocal rank fusion (RRF).
• Retrieve top-20 candidates for reranking, not just top-5 — reranker recovers precision.
• Apply metadata filters before vector search when query has structured constraints (dates, doc type).

5. Reranking

• Always rerank before passing to LLM when corpus > few thousand chunks. Use bge-reranker-v2 (self-hosted) or Cohere Rerank (managed API).
• Keep top-3 to top-5 after reranking for the final context window.

6. Prompt assembly

• Structure: system instructions (grounding rules) → numbered context chunks → question.
• Explicit instruction: "If the answer is not in the context, say you don't know" — core anti-hallucination guardrail.
• Include source identifiers inline so LLM can cite them natively (e.g., [1], [2]).

7. Citation & post-processing

• Validate that every claim in the generated answer maps to a cited chunk; flag ungrounded sentences.
• Return citations with clickable source/page references to the user.
• Optional: run a second LLM pass ("groundedness check") comparing answer against retrieved context.