Debugging
Debugging
Diagnose before you fix. A fix applied to an unconfirmed cause is a guess, not a solution. This skill treats every non-trivial failure as a hypothesis-testing problem: define the failure precisely, generate plausible causes grounded in actual evidence, run the cheapest experiment that can discriminate between them, and only then change code.
Decision rule for how much process to apply:
Is the cause already known with certainty (error message names the exact
line/config, or you just introduced this bug yourself in the last edit)?
→ YES: Skip to "Minimal Fix" (Step 5). Don't manufacture ceremony.
→ NO: Proceed through Steps 1–6 below.
Core loop for the non-trivial case:
1. DEFINE — expected vs actual, scope, repro steps, evidence on hand
2. HYPOTHESIZE — 2-4 plausible causes consistent with evidence + architecture
3. DISCRIMINATE — for each hypothesis: what evidence confirms/refutes it?
run the single cheapest check that distinguishes them
4. LOCATE — find the exact point where actual diverges from expected
5. FIX — smallest change addressing the confirmed cause only
6. VERIFY — reproduce-before/confirm-after, regression test, check blast radius
Never skip straight to step 5 when the cause is still a guess.
Step 1: Define the Failure
Capture, explicitly, before touching code:
- Expected behavior — what should happen, per spec/docs/reasonable assumption
- Actual behavior — what actually happens, verbatim error text/stack trace
- Scope — one user, all users, one environment, one input shape, one platform
- Reproduction — exact steps, or "could not reproduce" (see intermittent flow)
- Evidence available — logs, traces, screenshots, failing test, git diff of recent changes
Inspect only what's relevant to the symptom: recent commits touching the affected area, config/env differences, dependency versions, related tests. Do not assume the cause from the symptom's resemblance to a past bug — same symptom, different codebase, often different cause.
If it cannot be reproduced, go to Intermittent Failures below instead of guessing.
Progress:
- [ ] Expected vs actual behavior stated explicitly
- [ ] Scope and reproduction conditions established
- [ ] Relevant evidence collected (logs/traces/diff/tests)
- [ ] Confirmed this is NOT a known-cause trivial fix
Step 2: Form Hypotheses
List 2–4 plausible causes consistent with:
- the observed evidence (not a generic "it's probably a race condition")
- the actual architecture of this system (not a pattern borrowed from another project)
Discard hypotheses that don't explain all observed symptoms. If a hypothesis explains the error but not why it's intermittent, or explains the frontend symptom but ignores that the API payload is wrong, it's incomplete — refine it.
Step 3: Discriminate — Test Before You Fix
For each live hypothesis, before writing a fix:
| Question | Answer required |
|---|---|
| What evidence would confirm this? | specific log line, return value, breakpoint state |
| What evidence would refute it? | the opposite observation |
| Cheapest check that produces that evidence? | pick one: log/print, breakpoint, test, curl, diff, bisect |
| Result? | record actual outcome, not expected outcome |
Pick the lowest-cost, lowest-risk technique that can discriminate — don't reach for git bisect across 200 commits when a single breakpoint settles it, and don't guess at a breakpoint location when git log -p on the affected file shows the exact change.
Available techniques (pick what the evidence needs, not all of them):
- Minimal repeatable reproduction case
- Inspect inputs/outputs/variables/exceptions/call stack (debugger, breakpoints, watches)
- Trace data/execution across component boundaries
- Inspect API requests/responses, network tab, config, env vars, dependency versions
git log,git diff,git blame,git bisectagainst recent history- Write a targeted diagnostic test that isolates the suspected unit
- Logs/metrics/traces/profiler when they can discriminate between hypotheses specifically
Rule against speculative parallel edits: change one variable at a time. If you apply fix-candidate A and fix-candidate B simultaneously and the symptom disappears, you don't know which one worked, or whether either addressed the actual cause. Revert to isolate before committing to an explanation.
Correlation ≠ causation: "the error appeared after I changed X" is a lead, not proof. Confirm X causes the symptom (e.g., revert X alone, verify symptom disappears) before calling it the root cause.
Progress:
- [ ] Hypotheses listed, each explaining ALL observed symptoms
- [ ] For each: confirming/refuting evidence identified
- [ ] Cheapest discriminating check run, one variable at a time
- [ ] Hypothesis confirmed by actual (not assumed) result — or revised
Step 4: Locate the Failure Boundary
Find where actual behavior first diverges from expected — not merely where the error surfaces. A stack trace pointing at a null-check is often the consequence, not the cause (e.g., the null originated three calls upstream from a misconfigured API response).
Trace across the boundary that evidence implicates:
- input transformation / validation
- state updates
- API/contract boundary (request shape, response shape, status codes)
- database read/write, transaction boundaries
- async/ordering/race conditions
- configuration and environment differences
- external services / third-party APIs
If evidence shows the initial routing/assumption was wrong (e.g., "frontend bug" is actually a backend contract mismatch), stop speculative work immediately, update the diagnosis, and hand off to the correct specialist (see Specialist Coordination). Don't keep patching the wrong layer because you're already there.
Step 5: Implement or Coordinate the Fix
Apply the smallest change that addresses the confirmed cause:
- Preserve unrelated behavior, existing conventions, and the user's authorized scope
- No unrelated refactors, no new dependencies "while I'm in here"
- No blanket try/catch, no arbitrary retries/delays, no suppressing the symptom instead of correcting the cause
- If the fix belongs to a specialist domain (frontend/backend-data/architecture/performance/security/testing-quality) and that mechanism is actually available in this environment, hand off with the confirmed diagnosis, evidence, and failure boundary. If that mechanism is not available, say so explicitly and apply only the knowledge you actually have — never claim a specialist was invoked when it wasn't.
Incident exception: if the failure is actively causing damage (data loss, security exposure, production outage), apply safe containment first — feature flag off, rollback, rate-limit, circuit-break — before completing full root-cause analysis. Preserve logs/state needed for later diagnosis. Follow up with Steps 1–4 afterward; containment is not a substitute for diagnosis.
Step 6: Verify and Prevent Regression
Choose verification proportional to risk — a one-line typo fix doesn't need a stress test; a race condition fix does.
Where feasible:
- Reproduce the original failure (confirm it exists pre-fix)
- Confirm the failure no longer occurs post-fix, using the same repro steps
- Add/update a regression test targeting the specific defect, not a vague area
- Run existing tests covering the affected component and direct integrations
- Check for unintended side effects (did the fix change adjacent behavior?)
For intermittent defects: run repeated trials proportional to original failure frequency (if it failed 1-in-20, don't declare victory after 2 clean runs), or add instrumentation that would catch recurrence.
If reproduction/automated verification is impossible, state explicitly: what was inspected, what evidence supports the conclusion, and what remains unverified. Never claim a fix works just because the code changed or the app started.
- Pull all available logs/traces for actual past occurrences — timestamps, inputs, environment, concurrency level
- Look for a common factor: load, timing, specific input shape, specific environment/config, race between async operations
- Form a hypothesis about the triggering condition, not just "it's flaky"
- If justified by impact, add targeted instrumentation or a stress/concurrency test designed to surface the specific suspected condition — don't blast the whole test suite in a loop and hope
- Explicitly label the conclusion: "confirmed" only if the triggering condition was reproduced and the fix was shown to prevent it; otherwise "best available diagnosis, unconfirmed" — and say what evidence would confirm it later (e.g., "if this recurs, capture X")
Diagnose first. Hand off based on confirmed evidence, not initial assumption about where the bug "probably" lives.
| Domain | Hand off when evidence shows... |
|---|---|
| frontend | client-side rendering, interaction, state, or data-fetching logic is the actual cause |
| backend-data | API logic, persistence, transactions, or data integrity is the actual cause |
| architecture | the failure stems from structural boundaries/dependencies, not a local bug |
| performance | a measurable, systemic bottleneck (not a one-off slow call) |
| security | the failure is or enables an authz/authn/confidentiality/integrity issue |
| testing-quality | fix has meaningful integration surface requiring broader regression strategy |
Don't invoke multiple specialists speculatively. Diagnose, confirm the boundary, hand off to the one(s) the evidence actually implicates.
- Use only tools confirmed available in this environment (debugger, DevTools, terminal, git, test runner, docs). Verify availability before relying on a tool's output.
- Never fabricate command output, stack traces, or test results. If a command wasn't run, say so.
- Sanitize diagnostic output: never log/display credentials, tokens, passwords, or private user data. Redact before including in logs, reports, or test fixtures.
Example 1 — Deterministic bug, obvious cause
Input: TypeError: Cannot read property 'id' of undefined at user.profile.id, stack trace points directly at the line, and the preceding code shows no null check after an await fetchUser() that can return null on 404.
Output: Cause confirmed by stack trace + code inspection alone — no further hypothesis testing needed. Fix: add null-check / early return for the 404 case. Verify: add test for fetchUser returning null; confirm existing tests pass.
Example 2 — Runtime failure, cause unknown
Input: Checkout endpoint returns 500 intermittently under normal-looking load; error log shows "deadlock detected" in DB layer.
Output: Hypotheses: (a) two transactions updating orders and inventory in different lock order, (b) long-running report query holding locks, (c) connection pool exhaustion mimicking lock errors. Discriminator: inspect DB lock/wait logs during an occurrence — confirms (a) via lock-order mismatch between checkout() and restock() transactions. Fix: enforce consistent