Architecting Knowledge Corpora
YAML--- name: architecting-knowledge-corpora description: Determines whether sourced datasets/documents should be merged into a single unified knowledge corpus (e.g., one NotebookLM notebook or Gem knowledge base) or decomposed into specialized, segmented "expertise ledger" systems with routing. Use when organizing multiple data sources for retrieval-augmented systems, deciding notebook/Gem architecture, evaluating whether to consolidate or split knowledge bases, or auditing an existing corpus for retrieval quality and governance risk. --- Quick Start Given a set of sources grouped into candidate "themes" (by task type), score each theme on 8 weighted criteria (max 24 points), apply the veto rule, then assign one of five architectures: Merge, Split (ledger), Hybrid (router), Separate (reference-only), or Registry (benchmark catalog).
Theme: "security-guardrails" Rubric score: 15/24 → merge-band Veto check: refresh_cadence=0 AND access_tier=1 → VETO FIRES Final decision: SPLIT (ledger), despite merge-band score
Progress:
- Step 1: Define decision units (group sources by shared task type + output schema, not by file)
- Step 2: Score each unit on the 8-criteria rubric
- Step 3: Apply the veto rule (hard constraints override score)
- Step 4: Run or project the 5 empirical tests
- Step 5: Assign architecture pattern
- Step 6: Write a decision record per theme
- Step 7: Define re-check triggers
Step 1 — Define decision units
A merge candidate = sources sharing: same query type, same output schema, same owner, same refresh cadence. Any one of these differing is a potential split boundary. Decide per theme, never for the whole corpus in one shot.
Step 2 — Score the 8 criteria (0–2 each, weighted, max 24)
| Criterion | Weight | Merge signal (score 2) | Split signal (score 0) |
|---|---|---|---|
| Task homogeneity | 2 | Same query type across sources | Divergent query types |
| Schema compatibility | 2 | Fields map 1:1 into one schema | No shared schema |
| Retrieval collision (inverted) | 2 | Top-k stays in-domain | Cross-theme false positives dominate |
| Volume vs. platform cap | 1.5 | Well under source/token limits | Exceeds cap (e.g., NotebookLM's 50-source limit) |
| Refresh cadence | 1 | All sources update together | Mixed static/continuous feeds |
| Access-control tier | 1.5 | Same audience/license/PII tier | Tiers differ |
| Ownership | 0.5 | One maintainer | Fragmented ownership |
| Cross-domain query share | 1.5 | >30% of queries span themes | <10%, purely single-theme |
Decision bands: ≥12 = merge-band, 9–11 = hybrid-band, ≤8 = split-band.
Step 3 — Apply the veto rule (critical — do not skip)
Any theme scoring 0 on refresh-cadence homogeneity OR access-control tier is forced to split, regardless of total score. A high aggregate must not override a hard compliance/governance constraint. This is the single most important rule in the framework — rubric totals alone will mislead on mixed-license or mixed-cadence corpora.
Separately: a schema-compatibility score of 0 disqualifies a theme from "unified corpus" status even if everything else is strong — route it to Registry (a task catalog) instead of a merged retrieval notebook. This is distinct from the veto rule and catches benchmark-style data (e.g., heterogeneous tabular sets) that should never be merged into one retrieval index.
Step 4 — Run (or project) the 5 empirical tests
Never decide on intuition. Build or estimate:
| Test | Procedure | Fail signal |
|---|---|---|
| Retrieval collision | 50–100 queries/theme against merged index; % top-k from wrong theme | >20% → split |
| Needle-in-haystack | Insert unique fact per source; measure recall merged vs. split | Merged recall drops >10pts → split |
| Schema join | Attempt one unified table; % fields mapping cleanly | <50% → split |
| Ablation faithfulness | Score answers merged vs. split against ground truth | Split wins by >10% → split |
| Cost/latency | Tokens + latency merged vs. split+router | Router overhead > retrieval gain → merge |
If no live index exists, label projected estimates explicitly (e.g., [Projected] collision 0.06, basis: shared field vocabulary) — never present untested numbers as measured.
Step 5 — Assign architecture
| Pattern | When to use |
|---|---|
| Monolith (merge) | Score ≥12, no veto, corpus <60% of platform cap |
| Federated router + ledgers (hybrid) | Score 9–11, or cross-domain queries 10–30% |
| Expertise ledger (split, per-domain, with manifest) | Veto fired, or access/cadence/ownership diverge |
| Reference notebook (separate) | Content is documentation/instructions, not queryable data (schema score = 0, but not retrieval data at all) |
| Registry (catalog + task artifacts) | Schema score = 0 but content is legitimate heterogeneous benchmark data |
Step 6 — Decision record (one per theme)
YAMLtheme: security-guardrails decision: split rubric_score: 15/24 veto_triggered: true # cadence=0, access=1 architecture: expertise_ledger ledger_ids: [static-academic-guardrails-v1, live-harvest-feed-v1] review_date: 2026-11-28
Step 7 — Define re-check triggers
State explicitly what measured result would flip the decision, e.g. "if collision measures >0.20, move from merge to hybrid."
Example 1: Input: Document-extraction theme — homogeneous invoice/field-extraction schema, static refresh, public access, but one dataset (Docmatix) is enormous and VQA-formatted. Output: Score 17/24, no veto. Decision: Merge, with a volume guard — route oversized/divergent-schema assets to a sibling notebook rather than forcing them in.
Example 2: Input: Security/guardrails theme — high task homogeneity and low cross-theme collision (score 15/24, merge-band) but mixed licenses (some gated) and one continuously-harvested feed vs. frozen academic sets. Output: Veto fires on cadence=0 and access=1. Decision: Split into ledger (frozen core + live-harvest feed), overriding the merge-band score.
Example 3: Input: General ML/UCI theme — each dataset has its own unrelated schema (Iris ≠ Adult ≠ Spambase), but task homogeneity and access are fine (score 14.5/24). Output: Schema compatibility = 0 → Registry, not a unified corpus: build a catalog/manifest of tasks, keep each dataset as a separate loaded artifact, never merge into one retrieval index.
- Decide per theme (shared task + schema), never per individual file or for an entire corpus at once.
- Treat the veto rule as non-negotiable: compliance/cadence mismatches are governance questions, not retrieval-quality questions — don't let a good score paper over them.
- Always give every ledger/notebook a machine-readable manifest (schema, provenance, owner, refresh cadence, eval metrics, routing keywords) — this is what makes decomposition auditable instead of a silent drift risk.
- Put a thin catalog/router in front of any split architecture; it should hold only metadata, never duplicate data.
- Label all untested "test" metrics as
[Projected]with stated basis; never present estimates as measured results. - Define explicit re-check triggers (e.g., "if X% grows past Y, re-run the rubric") since corpora evolve.
- Respect platform hard caps (e.g., source-count limits) as a forcing function, independent of the rubric score.
- Premature monolith: merging divergent schemas produces a notebook that answers every question vaguely — always run the schema-join test first.
- Over-fragmentation: more than ~8 notebooks with <10 sources each raises maintenance cost and hurts recall more than it helps; consolidate where the rubric supports it.
- Ignoring the veto rule: a high aggregate score does not excuse mixed license tiers or mismatched refresh cadence — these are hard constraints, not scoring inputs to be averaged away.
- Duplicating core docs (glossaries, policies, schemas) across every ledger instead of once in the catalog.
- Router without fallback: always define default/low-confidence routing behavior (escalate to a general notebook) rather than failing silently.
- Presenting projected test numbers as measured: always flag estimates and state their basis; conflating inference with verified data undermines the whole audit.