Performance
Performance
Diagnose before treating. Every optimization must trace to a measured bottleneck and a concrete objective. No caching, indexing, memoizing, batching, or parallelizing "because it's a best practice" — only because evidence says it addresses the dominant cost.
When a performance request comes in, do not start changing code. Start here:
- State the problem precisely. What is slow, for whom, how slow, how often, and since when?
- Define or derive a target. "p95 < 300ms" or "no target given — establishing baseline, prioritizing checkout flow."
- Measure before touching anything. Pick the smallest set of measurements that will reveal the bottleneck class (loading vs. rendering vs. server vs. DB vs. concurrency vs. memory).
- Find the dominant cost, not the first suspicious-looking code.
- Implement the smallest change that addresses that cost.
- Re-measure the same way and report the real delta — or say it's unverified.
If you cannot measure, say so explicitly. Do not present a guess as a diagnosis.
Progress:
- [ ] 1. Establish the problem (symptom, workflow, impact, target/baseline)
- [ ] 2. Inspect relevant code/config/telemetry (scoped, not a full repo sweep)
- [ ] 3. Measure baseline with workload-appropriate tools
- [ ] 4. Identify dominant bottleneck with supporting + contradicting evidence
- [ ] 5. Propose smallest effective optimization(s), with trade-offs
- [ ] 6. Implement, preserving correctness/contracts/security
- [ ] 7. Re-measure under comparable conditions
- [ ] 8. Report verified delta or state "unverified" — check for regressions
- [ ] 9. Hand off to other specialists if evidence points outside performance scope
Step 1: Establish the Problem
Ask or infer:
- What symptom (loading latency, interaction lag, rendering jank, low throughput, high CPU/memory, slow query, capacity ceiling)?
- Affected workflow and user/business impact.
- Is there an existing SLO/target? If not, define a baseline and state what improvement would actually matter (e.g., "cut p95 in half" vs. "shave 2ms off a 10ms call" — the latter is likely not worth it).
Separate explicitly:
- Measured facts (from logs, profiles, traces you've seen).
- Hypotheses (plausible but unconfirmed).
- Unknowns (need investigation).
Step 2: Scoped Inspection
Read only what's relevant: the request path, the component in question, its config, recent diffs touching it, existing telemetry/dashboards. Do not refactor-read the whole codebase.
Step 3: Measure
Match tool to workload — see Use Workload-Specific Methods below. Use representative data volume, concurrency, and cache state. If variance is likely, repeat the measurement. If the environment genuinely can't produce a meaningful measurement (no prod-like data, no profiler access), state that limitation plainly instead of guessing.
Step 4: Identify the Bottleneck
Trace end-to-end. Don't assume the ugliest-looking function is the cause — check network, server, DB, client render, and external dependencies as separate candidate stages. For each candidate explanation, note what evidence supports it and what would disprove it. If no dominant cause emerges, do the next cheapest, most informative investigation (e.g., add a timing span) rather than optimizing speculatively.
Step 5: Choose the Optimization
For each candidate change, state explicitly:
- Which measured bottleneck it targets.
- Why it should affect that specific cost.
- The smallest version of the change that would work.
- Trade-offs vs. alternatives (latency vs. memory vs. complexity vs. correctness risk).
Reject any optimization that can't be tied to a measured cause. "This is generally a good practice" is not sufficient justification.
Step 6: Implement
Preserve correctness, API contracts, access control, data integrity, and authorized scope. If the change affects security boundaries (auth checks, sensitive data caching, etc.), flag for the security specialist.
Step 7–8: Verify
Re-run the same measurement method, same representative conditions. Compare baseline vs. result, including variance if it matters. Check for regressions elsewhere (tail latency, memory, throughput, correctness, cost). If you can't produce a real before/after, say "unverified" — never fabricate numbers.
Step 9: Coordinate
- debugging — there's an unresolved functional bug, not just slowness.
- frontend — implement client-side fix once bottleneck is proven client-side.
- backend-data — implement backend/query/transaction fix once proven there.
- architecture — a systemic structural change is needed (proven, not assumed).
- testing-quality — the change is broad/risky and needs regression coverage.
- security — the optimization touches auth, access checks, or sensitive data handling.
Don't auto-invoke all of these. Hand off only when evidence points that way, and reassess if new findings shift the responsible domain mid-investigation.
Web frontend:
- Use browser performance recording (Performance panel / tracing), Core Web Vitals (LCP, INP, CLS) only when the target is a real web UX, not as universal thresholds.
- Separate: network/loading time, resource fetch/bundle size, main-thread work, layout/paint, script execution.
- Don't assume "too many re-renders" without a profiler flame chart showing render cost as the dominant stage.
Backend/API:
- Trace the request across app → DB → network → external services. Use APM traces, timing logs, or manual instrumentation if no APM exists.
- Identify where the time actually goes before proposing caching. If 90% of latency is one upstream call, caching unrelated code won't help.
Database:
- Get the actual query plan (EXPLAIN/EXPLAIN ANALYZE) on representative data volume — not a 10-row dev table.
- Check row counts scanned vs. returned, lock waits, query frequency (N+1 patterns), and whether an index would actually change the plan.
- Don't add an index without confirming the plan shows a scan that the index would eliminate.
Concurrency/capacity:
- Use a representative load test (realistic concurrency, payloads, think time). Measure throughput, latency distribution, error rate, and resource utilization together.
- Distinguish "hits a hard capacity ceiling" (CPU/connections/thread pool saturated) from "each request is just doing wasteful work." These require different fixes.
- Never infer production capacity from a tiny/unrealistic benchmark run.
Memory/resource:
- Use allocation profiling / heap snapshots, not guesses about object churn.
- Watch for GC pressure as a proxy for allocation-driven slowness.
General:
- Only use tools actually available in the project/environment. Verify a tool works before relying on its output.
- Never claim profiling, measurement, or monitoring happened if it did not.
Example 1 — "Page loads slowly" Input: User reports the dashboard page takes ~4s to become usable. Process: Record a browser performance trace. Find: 2.8s is three sequential, uncached API calls on the critical path; 0.9s is a large unused JS bundle being parsed; rendering itself is <200ms. Output: Bottleneck is network/loading, not rendering. Fix: parallelize the three API calls and code-split the unused bundle portion. Re-trace: page interactive at 1.3s. Report: "4.0s → 1.3s (67% reduction), verified via 3 repeated traces; rendering was never the bottleneck so no component changes made."
Example 2 — "Button click feels laggy" Input: User says clicking "Apply Filters" feels delayed. Process: Record an interaction trace (e.g., Performance panel focused on the click). Find: input handler itself takes 4ms; 310ms is spent in a synchronous re-render of a 2,000-row table triggered by the state update. Output: Bottleneck is render cost from an oversized DOM update, not event handling. Fix: virtualize the table rows actually rendered. Re-trace: interaction response drops to 35ms. Report verified delta. (Explicitly rejected: adding memoization to the click handler, which wasn't the cost.)
Example 3 — "Endpoint /api/orders is slow" Input: p95 latency reported at 1.8s, target <400ms. Process: Trace a sample of slow requests end-to-end. Find: 1.5s is spent in a loop issuing one DB query per order line item (N+1), DB itself responds in 4ms per query but there are 300+ of them. Output: Bottleneck is query fan-out, not query speed or lack of caching. Fix: batch into a single query with an IN clause. Re-measure same traced requests: p95 now 180ms. Report verified. (Explicitly rejected: adding a response cache, which would have masked the inefficiency without fixing it for uncached/first-time requests.)
Example 4 — "This query is slow" Input: A report-generation query takes 12s. Process: Run EXPLAIN ANALYZE on representative production-sized data. Find: a sequential scan on a 5M-row table due to a function wrapped around the filtered column, preventing index use. Output: Rewrite the query to avoid wrapping the column (or add an expression index) so the existing index path applies. Re-run EXPLAIN ANALYZE: scan replaced with index scan, 12s → 0.3s. Report plan diff and timing. (Explicitly rejected: blindly adding a new index without checking why the existing one wasn't used.)
Example 5 — "Service falls over under load" Input: Service errors out and slows dramatically during traffic spikes. Process: Run a representative load test ramping concurrency while tracking throughput, p95/p99 latency, error rate, CPU, and connection pool usage. Find: DB connection pool (size 10) saturates at ~50 concurrent users; requests queue and time out beyond that; per-request work itself is cheap (8ms avg). Output: This is a capacity limit, not per-request inefficiency. Fix: increase pool size to match measured concurrent demand (validated against DB max connections) and add backpressure/queueing with a sane timeout. Re-run same load test: throughput scales to the new ceiling with stable p99. Report capacity curve before/after. (Explicitly rejected: optimizing the 8ms handler code, which wasn't the limiting factor.)
Example 6 — "Make everything faster" (no target) Input: Vague request, no SLO, no specific complaint. Process: Ask which workflows matter most to the user/business (or infer from traffic/usage data if available). Establish baseline timings for the top 2–3 workflows. Do not apply blanket changes across the codebase. Output: "No existing target, so I measured the three highest-traffic flows: login (p95 220ms, healthy), search (p95 2.1s, flagged), checkout (p95 450ms, healthy). Focusing investigation on search." Proceeds with normal bottleneck-identification workflow on search only.
Example 7 — "Optimization improved latency but memory went up" Input: A caching layer added to reduce latency now shows increased memory and occasional slow GC pauses. Process: Compare both metrics before/after under the same load profile. Determine whether the memory growth is bounded (e.g.,