Auditing Validation Frameworks
Given a framework (e.g., "7-step go/no-go process for X"), do not evaluate it holistically. Break it into atomic claims — one per tool, proxy, metric, or threshold — and fact-check each independently before synthesizing a verdict. A framework is only as sound as its weakest verifiable claim; one fabricated or misapplied tool invalidates any downstream scoring built on it.
Primary answer first, always: "This framework is [viable/unviable] because [the single biggest structural flaw]."
Progress:
- [ ] Step 1: Select mode(s) — Fact-Checking is primary when the framework's validity depends on tool/data capabilities; Value Determination is primary when the framework's logic is sound but its business utility is in question
- [ ] Step 2: Decompose the framework into discrete, testable claims (one per tool, proxy signal, data source, and threshold)
- [ ] Step 3: For each claim, determine: does this tool/source actually exist and do what's claimed? Label: True / Mostly True / Misleading / False / Unverified Assumption
- [ ] Step 4: Check for proxy mismatch — does the signal operate at the right level of granularity (macro vs. micro, aggregate vs. niche)?
- [ ] Step 5: Check for quantitative theater — do scoring thresholds have any empirical grounding, or are they arbitrary numbers applied to unverified inputs?
- [ ] Step 6: Synthesize into a single biggest risk, not a list of risks
- [ ] Step 7: Close with Core Finding, Confidence Level (with reason), Strategic Action
Decomposition discipline
List every tool, data source, and metric the framework references as a separate line item before analysis. This makes the audit trail checkable and prevents claims from being bundled or glossed over.
Labeling rules (fact-checking mode)
- True: Tool/claim verified as described, no material gap.
- Mostly True: Core capability exists but requires inferential leaps the framework doesn't acknowledge.
- Misleading: Claim is technically true but implies a capability or correlation that doesn't hold (classic proxy mismatch — macro data standing in for micro-level demand).
- False: Tool does not do what is claimed, verifiably (check the tool's actual stated purpose/product description).
- Unverified Assumption: No way to confirm from available information; state what evidence would resolve it (e.g., "a historical dataset mapping scores to outcomes would validate this threshold").
Proxy mismatch — the pattern to always check
The most common failure mode in business frameworks is substituting an adjacent, measurable signal for the actual signal of interest because the real one is hard to get (e.g., public company stock performance standing in for single-product demand; broad academic literature standing in for localized consumer intent). Flag this explicitly as "decoupled from the actual decision variable" even if each individual data point is factually accurate.
Scoring threshold scrutiny
Any numeric go/no-go threshold (e.g., "proceed at 10/15") is an Unverified Assumption by default unless the framework cites historical calibration data. Say so plainly: arbitrary weights applied to arbitrary inputs produce an illusion of rigor, not actual risk reduction.
Example 1: Input: A content-strategy framework recommends using "SocialPulse API" to measure engagement-to-conversion correlation, scoring candidate topics 1–10, proceeding only above 7. Output: Decompose into (1) does SocialPulse API exist and measure what's claimed, (2) is engagement a valid proxy for conversion, (3) is 7/10 empirically grounded. Verification shows SocialPulse is a scheduling tool with no conversion analytics → False on claim 1. This alone invalidates the scoring step, regardless of claims 2–3. Core finding: framework cannot be executed as described because its primary data source doesn't provide the required data.
Example 2: Input: A hiring framework uses LinkedIn follower count as a proxy for candidate domain authority, gated at 5,000+ followers. Output: Tool (LinkedIn) is real and the data point is real — Misleading, not False: follower count is verifiably measurable but is a weak, gameable proxy for domain authority, and the threshold (5,000) has no cited correlation to job performance → Unverified Assumption. Biggest risk: filtering out qualified candidates and admitting self-promoters.
- Always verify a tool's actual documented purpose before accepting a framework's characterization of it — frameworks often misattribute capabilities to tools that sound plausible.
- Separate "this data point is real" from "this data point measures what we need" — most framework failures are mismatch errors, not fabrication errors, though both occur.
- State explicitly what evidence would upgrade an Unverified Assumption to True or False — this makes the audit actionable, not just critical.
- Synthesize to one biggest risk in the close, not a bulleted list — decision-makers need a single prioritized action, not a risk inventory.
- When a framework names specific commercial tools/platforms, treat their described capability as a factual claim requiring the same verification rigor as any other.
- Don't evaluate a framework's internal logic in isolation from whether its inputs are real — a perfectly structured scoring matrix built on fictional data sources is worthless.
- Don't let one correct component (e.g., a real academic research tool) launder credibility onto the framework's other, false components.
- Don't present a numeric threshold as rigorous just because it's a number — demand calibration evidence or label it an assumption.
- Don't bury the single biggest risk among several minor ones — decision-makers acting on this analysis need one clear pivot point, not a diffuse list.
- Don't skip stating the domain inference and decomposition — an auditable trail is the point of this analysis, not a formality.