Architecting Vector Search Systems
For a typical semantic search / RAG use case, start with this baseline:
YAMLembedding_model: bge-m3 # multilingual, strong default; swap to text-embedding-3-large for English-heavy, high-budget cases vector_db: Qdrant # best default: fast HNSW, built-in hybrid search, easy self-host index: HNSW m: 16 ef_construction: 200 ef_search: 100 # tune up for recall, down for latency hybrid_search: dense_weight: 0.6 sparse_method: BM25 # or SPLADE for learned sparse fusion: RRF # Reciprocal Rank Fusion — robust default over weighted sum metadata_filtering: enabled # pre-filter before ANN when selective; post-filter when not
Benchmark before committing to a stack: load 10k representative vectors, measure recall@10 and p99 latency, then extrapolate to target scale.
Progress:
- [ ] Step 1: Select embedding model (task + language fit)
- [ ] Step 2: Design schema (metadata, indexes, partitioning)
- [ ] Step 3: Choose and configure ANN index (HNSW/IVF/PQ)
- [ ] Step 4: Set up hybrid search (dense + BM25/keyword)
- [ ] Step 5: Optimize (caching, sharding)
- [ ] Step 6: Benchmark (recall@k, latency, throughput)
Step 1: Embedding Model Selection
- Multilingual / mixed language:
bge-m3(dense+sparse+colbert in one model) orCohere embed-v3 - English, max quality, budget allows:
text-embedding-3-large - On-prem / cost-sensitive: open-source
bge-large,e5-large, self-hosted - Decision driver: dimensionality (higher = better recall, more storage/compute cost). 1024-1536 dims is the common sweet spot.
Step 2: Schema Design
- Separate payload/metadata (filterable fields: date, category, tenant_id) from the vector itself.
- Partition by tenant/language/time when data is naturally segmented — reduces search space and enables data lifecycle management (TTL, archiving).
- Index metadata fields that will be used for pre-filtering; avoid high-cardinality unindexed filters at query time.
Step 3: ANN Index Choice
| Index | When to use | Tradeoff |
|---|---|---|
| HNSW | Default for <100M vectors, need high recall + low latency | High memory (graph overhead) |
| IVF | Billion-scale, memory-constrained | Needs training (clustering), lower recall unless nprobe tuned high |
| IVF+PQ | Billion-scale + tight memory budget | Lossy compression, recall drop — mitigate with re-ranking on exact vectors |
| ScaNN | Google-scale, need best recall/speed tradeoff at scale | More complex ops, fewer DB integrations |
Rule of thumb: start HNSW, move to IVF+PQ only when memory becomes the bottleneck (usually >50-100M vectors on a single node).
Step 4: Hybrid Search Setup
- Run dense (embedding) and sparse (BM25/SPLADE) retrieval in parallel, fuse with Reciprocal Rank Fusion (RRF) — more robust than linear score blending across incompatible score scales.
- Keyword/BM25 catches exact matches (IDs, codes, rare terms) that embeddings blur — critical for technical/legal/product-code domains.
- Qdrant and Weaviate have native hybrid support; with pgvector, combine with Postgres full-text search (
tsvector) manually and fuse in application layer.
Step 5: Optimization
- Caching: cache embeddings for frequent/repeated queries; cache hot vector results at the application layer.
- Sharding: shard by tenant or hash of ID for horizontal scale; keep replica count sufficient for read throughput (reads dominate in search workloads).
- Quantization: scalar or product quantization to cut memory 4-32x — always validate recall impact before shipping.
Step 6: Benchmarking
Measure on representative production-like data and query distribution, not toy datasets:
- recall@k: compare ANN results vs brute-force ground truth on a sample set
- p50/p95/p99 latency under realistic concurrent load
- throughput (QPS) at target
ef_search/nprobesettings - Re-run benchmarks after every major index parameter change — recall/latency tradeoffs are non-linear.
Example 1: RAG system for internal docs, 2M chunks, multilingual
Input: Need semantic search over company docs in English/Russian, low-latency chat context retrieval.
Output: bge-m3 embeddings (1024-dim) → Qdrant with HNSW (m=16, ef_construction=200) → hybrid search with BM25 fusion via RRF → metadata filter by department/doc_type → benchmark shows recall@10=0.94, p99=45ms at 2M scale, no need for quantization yet.
Example 2: E-commerce product search, 500M+ products
Input: Billion-scale product catalog, needs fast filtering by category/price plus semantic similarity, tight memory budget.
Output: text-embedding-3-large truncated to 512 dims (Matryoshka) → Milvus with IVF+PQ (nlist=4096, PQ m=64) → pre-filter by category/price range before ANN search → re-rank top-200 candidates with exact distance → sharded by product category → recall@20 recovered to 0.91 after re-ranking step compensates PQ loss.
- Always benchmark recall vs ground truth (exact kNN) — never trust ANN defaults blindly.
- Prefer RRF over manual score-weighted fusion for hybrid search — avoids score-scale mismatch issues.
- Over-provision
ef_search/nprobeduring initial rollout, then tune down once latency budget is known. - Use pre-filtering when metadata filters are selective (<10% of data); post-filtering wastes ANN budget on filtered-out results.
- Re-embed and re-index when switching embedding models — never mix vectors from different model versions in one index.
- Ignoring recall degradation from quantization: PQ/scalar quantization silently drops recall — always validate with recall@k before production.
- Single giant index with no partitioning: makes scaling, multi-tenancy, and deletion/TTL painful; partition early.
- Pure vector search without keyword fallback: embeddings fail on exact-match needs (IDs, SKUs, rare proper nouns) — always pair with BM25/sparse for production search.
- Benchmarking on small/toy datasets: recall and latency behavior at 10k vectors doesn't predict behavior at 10M+; benchmark at realistic scale.
- Forgetting index rebuild cost: HNSW inserts are cheap but IVF retraining (updating centroids) at scale is expensive — plan reindexing cadence upfront.