Evaluating Output Metrics
Evaluates AI response quality against predefined KPIs (accuracy, completeness, speed) while accounting for interaction effects between metrics, similar to measuring wind turbine efficiency accounting for wake effects. Use when assessing AI output quality, comparing model responses, building evaluation rubrics, or when optimizing one metric risks degrading another.
Quick Start
Evaluate an AI response against a weighted KPI scorecard, then check for "wake effects" — cases where optimizing one metric silently degrades another (e.g., high speed causing low completeness, like a downstream turbine losing wind to the one in front of it).
Input: AI response + task context
Output:
KPI Scorecard:
- Accuracy: 8.5/10 (weight 0.4) → 3.40
- Completeness: 6.0/10 (weight 0.3) → 1.80
- Speed: 9.5/10 (weight 0.3) → 2.85
--------------------------------------------
Raw Score: 8.05/10
Wake Effect Check:
⚠ High speed (9.5) correlates with low completeness (6.0)
→ Likely cause: response truncated key edge cases to respond fast
→ Adjusted Score: 7.1/10 (penalty applied for interaction effect)
Progress:
- Step 1: Define KPIs and weights for the task
- Step 2: Score each KPI independently (blind to the others)
- Step 3: Check for wake effects (metric interactions)
- Step 4: Apply adjustment and compute final score
- Step 5: Report with rationale and actionable gaps
Step 1 — Define KPIs and weights Standard triad (adapt per task):
- Accuracy: factual/logical correctness vs. ground truth or expert judgment
- Completeness: coverage of all required sub-tasks, edge cases, caveats
- Speed: latency or token-efficiency relative to task complexity
Assign weights summing to 1.0 based on task priority (e.g., customer support → speed-weighted; legal/medical → accuracy-weighted).
Step 2 — Score independently Score each KPI in isolation first, without letting one judgment bleed into another. Use a 0–10 scale with explicit rubric anchors (e.g., 10 = no errors, 7 = minor non-critical error, 4 = major factual error).
Step 3 — Check for wake effects This is the core analytical move, borrowed from the turbine analogy: one turbine's output can't be judged in isolation if it's sitting in another turbine's wake. Look for these interaction patterns:
- Speed↑ causing Completeness↓ (rushed, truncated answers)
- Completeness↑ causing Accuracy↓ (padding with unverified claims)
- Accuracy↑ causing Speed↓ (excessive hedging/over-verification)
Flag any pair where one metric's strength plausibly explains another's weakness.
Step 4 — Apply adjustment If a wake effect is detected, apply a penalty (typically 5–15% of raw score) to reflect that the high metric was "stolen" from the low one rather than being a genuine independent strength. Document the adjustment reasoning — never adjust silently.
Step 5 — Report Always include: raw weighted score, detected wake effects, adjusted score, and 1-2 concrete fixes.
Example 1: Input: Customer support bot answers a billing question in 0.8s with a one-line answer missing the refund policy caveat. Output:
Accuracy: 9/10 (0.3) → 2.70
Completeness: 4/10 (0.3) → 1.20
Speed: 10/10 (0.4) → 4.00
Raw: 7.90/10
Wake Effect: ⚠ Speed/Completeness inverse correlation detected
→ Fast response achieved by omitting refund caveat
Adjusted Score: 6.7/10
Fix: Add caveat-check step before send; target <1.2s (still fast, now complete)
Example 2: Input: Research summary is comprehensive (12 sources) but took 45s and included 2 outdated citations. Output:
Accuracy: 6/10 (0.4) → 2.40
Completeness: 9.5/10 (0.3) → 2.85
Speed: 5/10 (0.3) → 1.50
Raw: 6.75/10
Wake Effect: ⚠ Completeness/Accuracy inverse correlation detected
→ Breadth achieved by including unverified/stale sources
Adjusted Score: 5.9/10
Fix: Cap sources at 8, add recency filter (<2yr), re-verify before inclusion
- Score KPIs blind to each other first — don't let a strong speed score unconsciously inflate your completeness score.
- Always state the rubric anchor used for each score (not just the number).
- Treat wake-effect penalties as diagnostic, not punitive — the goal is to surface why the tradeoff happened, so it can be engineered away.
- Re-run the same rubric across multiple responses for comparability; don't invent ad hoc criteria per evaluation.
- When weights aren't specified, default to equal thirds and state that assumption explicitly.
- Scoring metrics independently without checking interactions — a 9/10/9 scorecard that ignores a hidden tradeoff is a false positive, just like crediting a turbine for high output without noting it's drafting off its neighbor.
- Treating the adjusted score as final truth — it's an estimate; always show the raw and adjusted scores together.
- Using vague scoring language ("pretty good," "mostly complete") instead of rubric-anchored numbers.
- Applying the same weights to every task type — a creative-writing task and a medical-diagnosis task should not share a KPI weighting.
- Ignoring sample size — one evaluation is an anecdote; base conclusions on multiple runs when stakes are high.