AI Skill Report Card

Testing Quality

A93·Oct 15, 2026·Source: Extension-selection

Testing Quality

Testing proves specific claims about behavior with the smallest effective set of reliable tests. It is not "make the suite green." A passing suite with the wrong assertions, excessive mocking, or suppressed failures is worse than an honest report of what was and wasn't verified.

14 / 15

Before writing any test, answer three questions:

  1. What behavior must be proven? (the contract, not the implementation)
  2. What is the smallest test scope that proves it? (unit / integration / e2e)
  3. What evidence will demonstrate this is true when done?
Change: "Fix discount calculation to cap at 50%"
1. Behavior to prove: discount never exceeds 50%, regardless of input combination
2. Scope: unit test on the pricing function (pure logic, no I/O) — integration/e2e not needed
3. Evidence: new test reproduces the original bug (fails on old code), passes on fixed code;
   existing pricing tests still pass

Then: inspect existing tests/framework → write the minimal tests → run them for real → report actual results.

Recommendation▾
The final acceptance scenario (#6, security-sensitive change) is cut off mid-sentence — complete it or remove the fragment.
15 / 15
Progress:
- [ ] Identify behavior, acceptance criteria, and failure conditions to verify
- [ ] Inspect existing tests, framework, conventions, and available commands
- [ ] Scope investigation to the size/risk of the change
- [ ] Select test strategy (unit/integration/e2e/regression/contract/other)
- [ ] Write tests: clear names, real assertions, realistic data, isolation
- [ ] Run tests using actual project commands
- [ ] Diagnose any failures (defect vs. bad test vs. environment vs. flake)
- [ ] Fix root cause, not the symptom (never loosen assertions to pass)
- [ ] Re-run and confirm stability (repeat if flakiness suspected)
- [ ] Report: what ran, what passed/failed/skipped, what remains unverified

1. Understand what must be verified

  • Extract the requested behavior, acceptance criteria, and the failure modes that matter.
  • Read existing tests near the change, test config (package.json, pytest.ini, jest.config, CI YAML), and check dependency versions.
  • Identify what's affected: the component itself, its callers, and external boundaries (DB, API, filesystem, queue).
  • Match scope to risk: a one-line pure-function fix needs a handful of unit tests, not a full regression sweep. A shared-auth-middleware change needs much broader verification.
  • If requirements are ambiguous in a way that changes what "correct" means, state your assumption explicitly in the test names/comments or ask before proceeding.

2. Select test strategy

SituationUse
Isolated logic, pure functions, algorithms, boundary mathUnit
Behavior crosses a real boundary (DB, API, file, service)Integration
Critical user journey through the integrated systemEnd-to-end (few, high-value)
Two services/components must agree on an interfaceContract
A defect was observed and must never silently returnRegression
Workload/throughput/latency is the actual concernPerformance (coordinate with performance)
Access control / injection / crypto correctnessSecurity (coordinate with security)

Treat the test pyramid as a heuristic: many fast unit tests, fewer integration tests, a small number of e2e tests — but let the architecture and risk decide, not a fixed ratio. Never apply every category to every change.

3. Write reliable tests

  • Name tests after behavior + expected outcome: rejects_withdrawal_when_balance_insufficient, not test_withdraw_2.
  • Assert on actual outcomes (return values, state changes, emitted events, persisted rows) — not "no exception was thrown" as the sole check.
  • Cover: the normal case, boundary values (0, max, empty, null), invalid input, and at least one realistic failure path.
  • Mock only at real boundaries (network, time, randomness, third-party services). Don't mock the thing under test or mock so much that the test only checks that mocks were called.
  • Use realistic fixtures; avoid magic values with no connection to the domain.
  • Make tests order-independent and free of shared mutable state. Use fresh fixtures per test.
  • Replace arbitrary sleep() calls with condition/event-based waits (poll-until, done callbacks, fake timers).
  • Don't add redundant tests just to raise a coverage percentage — each test should assert something the others don't.

4. Handle failures and flakiness

When a test fails, diagnose before touching anything:

  1. Read the assertion message and the actual vs. expected values.
  2. Reproduce it in isolation (run just that test, then the file, then the suite).
  3. Classify the cause:
    • Product defect → do not touch the test; report/fix the code.
    • Wrong test expectation → fix the test, document why the original expectation was incorrect.
    • Stale fixture / test data → update data, note the change.
    • Environment/dependency issue → fix environment, don't blame the code.
    • Timing/nondeterminism → fix synchronization, don't add blind retries.
    • Unknown root cause → hand off to debugging; don't guess with code changes.
  4. Never: loosen an assertion, delete the test, inflate timeouts, or add retries as a substitute for understanding why it failed.
  5. If a test is suspected flaky, run it repeatedly (e.g., 20–50x or until the framework's repeat mode) and report the failure rate honestly — a later pass does not mean "fixed."
  6. When editing an existing test, preserve its original intent unless that intent is proven wrong or the requirement changed; say so explicitly in the diff/commit message.

5. Regression and integration risk

  • For a reported bug: write a test that fails against the old code and passes against the fix. This is the proof the bug is actually resolved, not just "looks fixed."
  • For cross-boundary changes: verify the contract (request/response shape, schema, status codes, error formats), not just the happy path. Check existing consumers aren't broken.
  • Expand scope when the change touches shared components, persistence layers, authorization, or anything with blast radius beyond the immediate caller.
  • For destructive, security-sensitive, or migration-related changes, coordinate with security or the relevant specialist and scale verification to the consequences (e.g., test both the dry-run and the real execution path on disposable data).

6. Test data and environments

  • Use isolated/ephemeral data: local DB instances, test containers, in-memory stores, seeded fixtures — never production data or live external systems.
  • Prefer fakes/stubs for speed and determinism, but include at least one real integration check when the actual dependency's behavior (serialization quirks, latency, error codes) matters to correctness.
  • Make setup/teardown explicit and symmetric — every created resource is cleaned up, even on failure.
  • If a failure might be environmental (network blip, container not ready, port conflict), investigate that before concluding it's an application bug.
  • Don't introduce a new test framework/library casually. If the repo has none, assess: is a small test feasible with stdlib/existing tooling? Is the change risky enough to justify adding a framework? State the trade-off and recommendation rather than silently adding dependencies.

7. Run and report

  • Use the project's actual test command (npm test, pytest, go test ./..., CI script) — don't invent a different invocation.
  • Scale what you run: focused test file(s) for a localized change; broader suite for cross-cutting changes; add targeted e2e only where it adds real signal.
  • Report explicitly:
    • Which commands were run.
    • Pass/fail/skip counts and which specific tests failed.
    • Whether the original requirement is actually verified by what ran.
    • Any flakiness observed, environment constraints, or things left unverified.
  • Never state tests passed without having run them. Never present "the build succeeded" or "coverage is 92%" as proof of correctness — they are partial signals, not verification of the specific behavior requested.

8. Coordinating with other specialists

  • debugging — unexplained failures, root-cause analysis beyond test diagnosis.
  • frontend / backend-data — implementing the behavior under test when it's missing or wrong.
  • architecture — structural issues that make something untestable.
  • security — vulnerability verification and access-control correctness.
  • performance — load/throughput/latency measurement and profiling.

Hand off rather than absorbing their workflow; testing-quality verifies, it doesn't redesign architecture or patch production code beyond what's needed to make a diagnosed fix testable.

Recommendation▾
Consider adding a brief 'bad example' showing an anti-pattern test (e.g., mock-heavy test that proves nothing) alongside the good examples for contrast.

1. Small function change (focused unit tests only) Change: pure function fix. Verify: normal case, boundaries, the specific bug input. Scope: unit tests on that function only — do not run the full suite or add integration/e2e tests. Avoid: inspecting the whole repo, adding unrelated coverage. Sufficient evidence: new/updated unit tests pass, cover the changed logic's edge cases, and existing unit tests for that module still pass.

2. Cross-component feature (targeted integration + minimal e2e) Change: new feature spanning API + DB. Verify: request/response contract, persistence correctness, error paths across the boundary. Scope: integration tests hitting a real/test-instance DB and the API layer; one e2e test only if a full user journey is materially at risk otherwise. Avoid: mocking the DB entirely (would prove nothing about the real contract); writing e2e tests for every branch. Sufficient evidence: integration tests pass against a real test dependency, contract matches spec, no existing consumer tests broke.

3. Bug fix regression test Change: defect reported in production/issue tracker. Verify: a test reproducing the exact defect fails on pre-fix code and passes on post-fix code. Scope: smallest test that isolates the defective behavior (usually unit, sometimes integration if the bug is cross-boundary). Avoid: fixing the code without a test that would have caught the original bug; writing a test so broad it could pass for unrelated reasons. Sufficient evidence: regression test demonstrably fails on old code, passes on new code, and is committed alongside the fix.

4. Flaky test (passes/fails nondeterministically on unchanged code) Verify: actual source of nondeterminism (timing, ordering, shared state, external dependency). Scope: diagnostic — run the test repeatedly (isolated and in full suite) to find the pattern; inspect for sleeps, shared fixtures, unmocked clocks/network. Avoid: adding retries, increasing timeouts, marking as skip/quarantine without root-causing, or treating one clean run as proof. Sufficient evidence: identified root cause, applied a deterministic fix (proper wait condition, isolated state, mocked time), and demonstrated N consecutive stable runs post-fix.

5. API/DB integration where mocks alone are insufficient Change: integration with external API or DB where version/schema drift is a real risk. Verify: actual request/response shape, status codes, serialization, and error formats against a real or realistic test instance (sandbox API, test container, contract test). Scope: integration or contract test using the real dependency (or a high-fidelity test double like a container), not a hand-rolled mock that encodes assumptions. Avoid: asserting only that a mock was called with expected arguments. Sufficient evidence: test runs against a real/representative instance of the dependency and passes; mock-based unit tests, if also present, are supplementary, not sole proof.

6. Security-sensitive change Change: modifies auth, access control,

0
Grade AAI Skill Framework
Scorecard
Criteria Breakdown
Quick Start
14/15
Workflow
15/15
Examples
19/20
Completeness
19/20
Format
15/15
Conciseness
13/15