Scoring Corpus Architecture Decisions
YAML--- name: scoring-corpus-architecture-decisions description: Applies the 8-criteria rubric, veto rule, and 5 empirical tests from the architecting-knowledge-corpora framework to a specific real-world corpus, producing per-theme decision records with scores, veto checks, projected/measured test results, architecture assignments, and re-check triggers. Use when a user provides their own sources/themes and wants a concrete, auditable merge-vs-split decision rather than an explanation of the framework itself. ---
Given a framework (rubric + veto + architectures) and a real corpus description, produce one decision record per theme — never a generic restatement of the framework.
Theme: "security-guardrails"
Rubric: 15/24 (merge-band)
Veto: cadence=0, access=1 → FIRES
Decision: SPLIT (expertise ledger), overriding merge-band score
Trigger to revisit: if license tiers unify AND cadence synchronizes, re-score for merge
If the corpus description is thin, state minimal assumptions per theme, mark affected scores provisional, and proceed anyway — do not stall on clarifying questions.
Progress:
- Step 1: Build theme inventory table from the corpus description (group by task type + schema, not by file)
- Step 2: Flag boundary signals per theme (owner, cadence, access tier, query type differences)
- Step 3: Score each theme on the 8 criteria, show per-criterion numbers, sum to band
- Step 4: Apply veto rule before trusting the band
- Step 5: Produce test plan — measured where data exists,
[Projected]with stated basis otherwise - Step 6: Assign architecture, state the one driving factor
- Step 7: Write YAML decision record per theme
- Step 8: Write re-check triggers per theme
- Step 9: Add one-line business-impact statement per decision (maintenance cost / governance exposure / retrieval risk)
Step 1–2: Theme inventory
Output a table:
| Theme | Sources (count) | Shared task type | Shared schema? | Owner(s) | Cadence | Access tier | Boundary flags |
|---|
Never invent sources; if the user's description doesn't give source counts, write "unspecified — assumed single owner/cadence" and mark provisional.
Step 3: Scoring table
One row per criterion, per theme:
| Criterion | Weight | Score (0–2) | Weighted | Basis / assumption |
|---|---|---|---|---|
| Task homogeneity | 2 | |||
| Schema compatibility | 2 | |||
| Retrieval collision (inverted) | 2 | |||
| Volume vs. platform cap | 1.5 | |||
| Refresh cadence | 1 | |||
| Access-control tier | 1.5 | |||
| Ownership | 0.5 | |||
| Cross-domain query share | 1.5 | |||
| Total | /24 |
Band: ≥12 merge, 9–11 hybrid, ≤8 split. If any score is provisional, state the total as provisional too.
Step 4: Veto check (do before quoting the band as final)
Veto check: refresh_cadence=<score> AND access_tier=<score>
→ FIRES / does not fire
Schema check: schema_compatibility=<score>
→ disqualifies unified status (route Registry/Reference) / passes
State explicitly when a veto overrides a merge-band or hybrid-band score — this is the most common place totals mislead.
Step 5: Test plan
| Test | Measured or Projected | Value | Basis (if projected) |
|---|---|---|---|
| Retrieval collision | |||
| Needle-in-haystack | |||
| Schema join | |||
| Ablation faithfulness | |||
| Cost/latency |
Never state a projected number without [Projected] and a one-clause basis (e.g., "basis: shared field vocabulary across sources").
Step 6: Architecture assignment
State one line: Decision: <pattern> — driven by <veto | score band | schema=0 | cap exceeded>.
Step 7: YAML decision record (per theme)
YAMLtheme: <name> decision: <merge|split|hybrid|separate|registry> rubric_score: <x>/24 veto_triggered: <true|false> architecture: <pattern> ledger_ids: [<ids or "n/a">] review_date: <YYYY-MM-DD>
Step 8: Re-check triggers
One sentence per theme: the specific measured value or corpus change that flips the decision. E.g., "If cross-domain query share measured >30%, upgrade from ledger to federated router."
Step 9: Business impact
One or two sentences per theme tying the decision to maintenance cost, governance exposure, or retrieval risk — not just a technical justification.
Example 1: Input: "Invoice-extraction theme: 40 static PDFs, one owner, public, one huge VQA-formatted outlier dataset." Output: Score 17/24 (task=2, schema=1.5 [outlier causes partial mismatch], cadence=2, access=2, ownership=0.5, collision=2, volume=1.5 [outlier pushes near cap], cross-domain=1.5). No veto. Decision: Merge, with the outlier routed to a sibling notebook as a volume guard. Trigger: "if outlier + growth pushes source count >80% of platform cap, split out the outlier permanently."
Example 2: Input: "Guardrails theme: mix of frozen academic benchmark sets (public) and a continuously-harvested live feed (gated license)." Output: Score 15/24 but cadence=0, access=1 → veto fires. Decision: Split into expertise ledger (frozen-core-v1, live-harvest-v1). Business impact: avoids license exposure from merging gated content into a notebook that is also queried against public academic data.
- Always produce the theme inventory table first — it forces per-theme (not per-corpus) decisions.
- Show per-criterion scores, not just totals; totals without a breakdown can't be audited or corrected.
- Check veto and schema=0 conditions before presenting the band as the likely outcome — don't bury the override.
- Default to minimal, explicit assumptions when the corpus description is thin; mark every affected number
provisionalrather than refusing to score. - Separate measured from
[Projected]test results in every single row — no exceptions, even for "obvious" estimates. - Always close each theme with a concrete re-check trigger tied to a measurable threshold, not a vague "revisit later."
- Keep the business-impact line short and concrete (cost, exposure, risk) — this is what makes the record useful to a non-technical stakeholder.
- Explaining the framework instead of applying it: the user wants scored output for their corpus, not a restatement of the rubric.
- Averaging away a veto: a 15/24 score does not survive a cadence=0 or access=1 flag — say so explicitly, don't quietly fold it into "hybrid."
- Presenting assumptions as facts: if the user didn't specify owner/cadence/access, say "assumed" and mark provisional — don't silently guess and present as measured.
- Skipping the schema=0 → Registry/Reference distinction: this is a different exit path than the veto rule and is easy to conflate with a simple split.
- Omitting re-check triggers: a decision without a stated flip-condition isn't auditable and invites silent architecture drift.
- Citing platform limits or benchmark facts without flagging uncertainty: if a specific cap or product behavior can't be verified from the given input, say so rather than asserting a number.