AI Skill Report Card
Analyzing Calculations for Bias
Quick Start13 / 15
Given a calculation document or a reasoning chain, run this two-pass check:
- Formal pass: verify arithmetic, units, formula references, cross-references, formatting standards (GOST/internal templates).
- Cognitive pass: scan the narrative/justification text for bias markers (confirmation bias, anchoring, overconfidence, sunk cost) and score them against protective factors (explicit assumptions, sensitivity analysis, independent review, alternative scenarios).
Output: a defect table (formal errors) + a Risk-Resilience balance score (bias load vs protective factors) + recommendation.
Recommendation▾
Add a third example showing a 'High risk' balance ≥4 outcome to cover all three risk tiers
Workflow14 / 15
Progress:
- Step 1: Parse document structure (inputs, formulas, outputs, conclusions)
- Step 2: Recompute key formulas; flag unit/dimension mismatches
- Step 3: Check formatting against the applicable standard/template
- Step 4: Extract reasoning/justification text (why these inputs, why this method)
- Step 5: Tag cognitive bias markers in the reasoning text
- Step 6: Tag protective factors present in the document
- Step 7: Compute Risk-Resilience balance, assign risk tier
- Step 8: Produce defect table + bias report + recommendations
Step 2-3: Formal Error Categories
- Arithmetic/formula mismatch (result ≠ recomputed value)
- Dimensional inconsistency (mixing m/mm, kg/tonnes, VAT in/out)
- Missing or outdated normative reference (SNiP/GOST/internal regs)
- Inconsistent rounding or significant-figure drift across pages
- Broken cross-reference (table/figure numbers, section links)
- Formatting deviation (missing signatures, stamps, revision log)
Step 5: Cognitive Bias Markers to Scan For
| Bias | Textual signal |
|---|---|
| Confirmation bias | Only supporting data cited; alternatives dismissed without analysis |
| Anchoring | First estimate repeated unchanged across revisions despite new data |
| Overconfidence | Absolute language ("will definitely", "no risk") without uncertainty range |
| Sunk cost | Justification references past investment rather than current merits |
| Availability bias | Risk assessment based on one memorable precedent, not statistics |
| Authority bias | "Approved by X" used instead of technical justification |
Step 6: Protective Factors to Scan For
- Explicit stated assumptions and their validity range
- Sensitivity/scenario analysis (best/worst/base case)
- Independent second reviewer or cross-check signature
- Documented alternatives considered and rejected with reasons
- Margin/safety factor explicitly justified (not just copied default)
Step 7: Risk-Resilience Balance Model
Score each bias marker found: +1 (minor), +2 (strong). Score each protective factor: -1 (present), -2 (rigorous/quantified).
Balance = Σ(bias scores) + Σ(protective scores)
- Balance ≤ 0 → Resilient: document can proceed, note minor comments
- Balance 1–3 → Watch: request clarification on flagged reasoning points
- Balance ≥ 4 → High risk: require rework/independent review before approval
Recommendation▾
Provide an English-language example alongside the Russian ones for broader applicability/clarity
Examples15 / 20
Example 1: Input: Смета на устройство фундамента; обоснование нагрузки ссылается на "аналогичный объект 2019 года", без расчета по актуальным нагрузкам; коэффициент запаса взят "по умолчанию 1.3" без обоснования. Output:
- Formal errors: none found in arithmetic; unit consistency OK.
- Bias markers: Availability bias (+2, опора на один прецедент), Anchoring (+1, коэффициент не пересчитан)
- Protective factors: none found (0)
- Balance = 3 → Watch. Рекомендация: запросить расчет нагрузок по текущим данным и обоснование коэффициента запаса.
Example 2: Input: Расчет нагрузки на балку с указанием диапазона (min/max), двумя независимыми проверяющими подписями, явным указанием допущений по материалу. Output:
- Formal errors: round-off inconsistency between page 3 and summary table (minor formatting note).
- Bias markers: none found (0)
- Protective factors: sensitivity analysis (-2), independent review (-2), explicit assumptions (-1)
- Balance = -5 → Resilient. Рекомендация: исправить округление, документ можно утверждать.
Recommendation▾
Include a concrete defect table template/schema (columns, format) rather than just prose description
Best Practices
- Always recompute, never trust stated totals — formal pass must be independent of the author's own check.
- Separate the formal defect report from the bias report; don't let bias scoring excuse arithmetic errors or vice versa.
- Quote the exact sentence that triggered a bias tag — never assert bias without textual evidence.
- Treat the Risk-Resilience score as a prioritization tool, not a pass/fail gate by itself.
- When self-auditing (AI checking its own prior output), explicitly list assumptions made earlier and re-evaluate them as if written by someone else.
Common Pitfalls
- Don't flag confident language as bias when it's backed by quantified sensitivity analysis — confidence ≠ overconfidence.
- Don't stop at formal errors and skip the cognitive pass; polished formatting can mask flawed reasoning.
- Don't double-count the same evidence as both a bias marker and a missing protective factor.
- Don't apply a fixed universal threshold — calibrate risk tiers to document criticality (load-bearing structure vs. cosmetic estimate).