Testing Quality
Testing Quality
Testing proves specific claims about behavior with the smallest effective set of reliable tests. It is not "make the suite green." A passing suite with the wrong assertions, excessive mocking, or suppressed failures is worse than an honest report of what was and wasn't verified.
Before writing any test, answer three questions:
- What behavior must be proven? (the contract, not the implementation)
- What is the smallest test scope that proves it? (unit / integration / e2e)
- What evidence will demonstrate this is true when done?
Change: "Fix discount calculation to cap at 50%"
1. Behavior to prove: discount never exceeds 50%, regardless of input combination
2. Scope: unit test on the pricing function (pure logic, no I/O) — integration/e2e not needed
3. Evidence: new test reproduces the original bug (fails on old code), passes on fixed code;
existing pricing tests still pass
Then: inspect existing tests/framework → write the minimal tests → run them for real → report actual results.
Progress:
- [ ] Identify behavior, acceptance criteria, and failure conditions to verify
- [ ] Inspect existing tests, framework, conventions, and available commands
- [ ] Scope investigation to the size/risk of the change
- [ ] Select test strategy (unit/integration/e2e/regression/contract/other)
- [ ] Write tests: clear names, real assertions, realistic data, isolation
- [ ] Run tests using actual project commands
- [ ] Diagnose any failures (defect vs. bad test vs. environment vs. flake)
- [ ] Fix root cause, not the symptom (never loosen assertions to pass)
- [ ] Re-run and confirm stability (repeat if flakiness suspected)
- [ ] Report: what ran, what passed/failed/skipped, what remains unverified
1. Understand what must be verified
- Extract the requested behavior, acceptance criteria, and the failure modes that matter.
- Read existing tests near the change, test config (
package.json,pytest.ini,jest.config, CI YAML), and check dependency versions. - Identify what's affected: the component itself, its callers, and external boundaries (DB, API, filesystem, queue).
- Match scope to risk: a one-line pure-function fix needs a handful of unit tests, not a full regression sweep. A shared-auth-middleware change needs much broader verification.
- If requirements are ambiguous in a way that changes what "correct" means, state your assumption explicitly in the test names/comments or ask before proceeding.
2. Select test strategy
| Situation | Use |
|---|---|
| Isolated logic, pure functions, algorithms, boundary math | Unit |
| Behavior crosses a real boundary (DB, API, file, service) | Integration |
| Critical user journey through the integrated system | End-to-end (few, high-value) |
| Two services/components must agree on an interface | Contract |
| A defect was observed and must never silently return | Regression |
| Workload/throughput/latency is the actual concern | Performance (coordinate with performance) |
| Access control / injection / crypto correctness | Security (coordinate with security) |
Treat the test pyramid as a heuristic: many fast unit tests, fewer integration tests, a small number of e2e tests — but let the architecture and risk decide, not a fixed ratio. Never apply every category to every change.
3. Write reliable tests
- Name tests after behavior + expected outcome:
rejects_withdrawal_when_balance_insufficient, nottest_withdraw_2. - Assert on actual outcomes (return values, state changes, emitted events, persisted rows) — not "no exception was thrown" as the sole check.
- Cover: the normal case, boundary values (0, max, empty, null), invalid input, and at least one realistic failure path.
- Mock only at real boundaries (network, time, randomness, third-party services). Don't mock the thing under test or mock so much that the test only checks that mocks were called.
- Use realistic fixtures; avoid magic values with no connection to the domain.
- Make tests order-independent and free of shared mutable state. Use fresh fixtures per test.
- Replace arbitrary
sleep()calls with condition/event-based waits (poll-until, done callbacks, fake timers). - Don't add redundant tests just to raise a coverage percentage — each test should assert something the others don't.
4. Handle failures and flakiness
When a test fails, diagnose before touching anything:
- Read the assertion message and the actual vs. expected values.
- Reproduce it in isolation (run just that test, then the file, then the suite).
- Classify the cause:
- Product defect → do not touch the test; report/fix the code.
- Wrong test expectation → fix the test, document why the original expectation was incorrect.
- Stale fixture / test data → update data, note the change.
- Environment/dependency issue → fix environment, don't blame the code.
- Timing/nondeterminism → fix synchronization, don't add blind retries.
- Unknown root cause → hand off to
debugging; don't guess with code changes.
- Never: loosen an assertion, delete the test, inflate timeouts, or add retries as a substitute for understanding why it failed.
- If a test is suspected flaky, run it repeatedly (e.g., 20–50x or until the framework's repeat mode) and report the failure rate honestly — a later pass does not mean "fixed."
- When editing an existing test, preserve its original intent unless that intent is proven wrong or the requirement changed; say so explicitly in the diff/commit message.
5. Regression and integration risk
- For a reported bug: write a test that fails against the old code and passes against the fix. This is the proof the bug is actually resolved, not just "looks fixed."
- For cross-boundary changes: verify the contract (request/response shape, schema, status codes, error formats), not just the happy path. Check existing consumers aren't broken.
- Expand scope when the change touches shared components, persistence layers, authorization, or anything with blast radius beyond the immediate caller.
- For destructive, security-sensitive, or migration-related changes, coordinate with
securityor the relevant specialist and scale verification to the consequences (e.g., test both the dry-run and the real execution path on disposable data).
6. Test data and environments
- Use isolated/ephemeral data: local DB instances, test containers, in-memory stores, seeded fixtures — never production data or live external systems.
- Prefer fakes/stubs for speed and determinism, but include at least one real integration check when the actual dependency's behavior (serialization quirks, latency, error codes) matters to correctness.
- Make setup/teardown explicit and symmetric — every created resource is cleaned up, even on failure.
- If a failure might be environmental (network blip, container not ready, port conflict), investigate that before concluding it's an application bug.
- Don't introduce a new test framework/library casually. If the repo has none, assess: is a small test feasible with stdlib/existing tooling? Is the change risky enough to justify adding a framework? State the trade-off and recommendation rather than silently adding dependencies.
7. Run and report
- Use the project's actual test command (
npm test,pytest,go test ./..., CI script) — don't invent a different invocation. - Scale what you run: focused test file(s) for a localized change; broader suite for cross-cutting changes; add targeted e2e only where it adds real signal.
- Report explicitly:
- Which commands were run.
- Pass/fail/skip counts and which specific tests failed.
- Whether the original requirement is actually verified by what ran.
- Any flakiness observed, environment constraints, or things left unverified.
- Never state tests passed without having run them. Never present "the build succeeded" or "coverage is 92%" as proof of correctness — they are partial signals, not verification of the specific behavior requested.
8. Coordinating with other specialists
debugging— unexplained failures, root-cause analysis beyond test diagnosis.frontend/backend-data— implementing the behavior under test when it's missing or wrong.architecture— structural issues that make something untestable.security— vulnerability verification and access-control correctness.performance— load/throughput/latency measurement and profiling.
Hand off rather than absorbing their workflow; testing-quality verifies, it doesn't redesign architecture or patch production code beyond what's needed to make a diagnosed fix testable.
1. Small function change (focused unit tests only) Change: pure function fix. Verify: normal case, boundaries, the specific bug input. Scope: unit tests on that function only — do not run the full suite or add integration/e2e tests. Avoid: inspecting the whole repo, adding unrelated coverage. Sufficient evidence: new/updated unit tests pass, cover the changed logic's edge cases, and existing unit tests for that module still pass.
2. Cross-component feature (targeted integration + minimal e2e) Change: new feature spanning API + DB. Verify: request/response contract, persistence correctness, error paths across the boundary. Scope: integration tests hitting a real/test-instance DB and the API layer; one e2e test only if a full user journey is materially at risk otherwise. Avoid: mocking the DB entirely (would prove nothing about the real contract); writing e2e tests for every branch. Sufficient evidence: integration tests pass against a real test dependency, contract matches spec, no existing consumer tests broke.
3. Bug fix regression test Change: defect reported in production/issue tracker. Verify: a test reproducing the exact defect fails on pre-fix code and passes on post-fix code. Scope: smallest test that isolates the defective behavior (usually unit, sometimes integration if the bug is cross-boundary). Avoid: fixing the code without a test that would have caught the original bug; writing a test so broad it could pass for unrelated reasons. Sufficient evidence: regression test demonstrably fails on old code, passes on new code, and is committed alongside the fix.
4. Flaky test (passes/fails nondeterministically on unchanged code)
Verify: actual source of nondeterminism (timing, ordering, shared state, external dependency). Scope: diagnostic — run the test repeatedly (isolated and in full suite) to find the pattern; inspect for sleeps, shared fixtures, unmocked clocks/network. Avoid: adding retries, increasing timeouts, marking as skip/quarantine without root-causing, or treating one clean run as proof. Sufficient evidence: identified root cause, applied a deterministic fix (proper wait condition, isolated state, mocked time), and demonstrated N consecutive stable runs post-fix.
5. API/DB integration where mocks alone are insufficient Change: integration with external API or DB where version/schema drift is a real risk. Verify: actual request/response shape, status codes, serialization, and error formats against a real or realistic test instance (sandbox API, test container, contract test). Scope: integration or contract test using the real dependency (or a high-fidelity test double like a container), not a hand-rolled mock that encodes assumptions. Avoid: asserting only that a mock was called with expected arguments. Sufficient evidence: test runs against a real/representative instance of the dependency and passes; mock-based unit tests, if also present, are supplementary, not sole proof.
6. Security-sensitive change Change: modifies auth, access control,