Engineering System Resilience

Given a system description, produce three deliverables:

1. Resilience Map — visualize components, dependencies, single points of failure (SPOFs)
2. Failure Analysis — enumerate failure modes, likelihood, blast radius
3. Recovery Design — concrete mechanisms to detect, contain, and recover from each failure

Example invocation: "Analyze resilience of our payment processing pipeline (API gateway → auth service → payment service → 3rd-party PSP → DB)."

Progress:

• Step 1: Map the system — components, dependencies, data flow, external integrations
• Step 2: Identify SPOFs and trust boundaries
• Step 3: Run failure analysis (FMEA-style) per component
• Step 4: Classify failures by impact (degraded / partial outage / full outage) and likelihood
• Step 5: Design redundancy (active-active, active-passive, N+1, geo-redundancy)
• Step 6: Design detection (health checks, timeouts, circuit breakers, alerting)
• Step 7: Design recovery (failover, retries with backoff, graceful degradation, rollback)
• Step 8: Define recovery targets (RTO/RPO) and validate via chaos testing recommendations
• Step 9: Compile into Resilience Map, Failure Analysis, Recovery Design

Step Details

Resilience Map: Diagram or structured list of nodes (services, DBs, queues, third-party deps) and edges (calls, data flow). Mark each node with: redundancy level (none/N+1/multi-region), criticality (critical/degradable/optional).

Failure Analysis: For each component, list failure modes (crash, latency spike, data corruption, network partition, dependency timeout, resource exhaustion). Score likelihood (low/med/high) × impact (low/med/high/critical).

Recovery Design: For each high-priority failure, specify:

• Detection mechanism (what signals it, how fast)
• Containment (circuit breaker, bulkhead, rate limiter)
• Recovery action (failover target, retry policy, fallback response)
• RTO (time to recover) / RPO (acceptable data loss)

Example 1: Input: "E-commerce checkout: web app → order service → payment service (single instance) → single Postgres DB, no cache."

Output:

RESILIENCE MAP
- web app (stateless, load-balanced) — redundancy: N+1 ✅
- order service (2 instances) — redundancy: N+1 ✅
- payment service (1 instance) — redundancy: NONE ⚠️ SPOF
- Postgres DB (single instance) — redundancy: NONE ⚠️ SPOF
- No caching layer — all reads hit DB directly

FAILURE ANALYSIS
1. Payment service crash — Likelihood: Med, Impact: Critical (checkout fully blocked)
2. DB instance failure — Likelihood: Low, Impact: Critical (total outage, possible data loss)
3. DB connection pool exhaustion under load — Likelihood: High, Impact: High (cascading timeouts)
4. Payment gateway (3rd party) timeout — Likelihood: Med, Impact: Med (checkout stalls, no fallback)

RECOVERY DESIGN
1. Payment service SPOF
   - Detection: health check every 5s, 2 failed checks trigger alert
   - Containment: circuit breaker on order→payment calls (trip after 5 failures/10s)
   - Recovery: deploy 2nd instance behind LB (active-active); RTO < 30s
2. DB SPOF
   - Detection: replication lag / connectivity monitor
   - Recovery: add read replica + automated failover (e.g., Patroni/RDS Multi-AZ); RTO < 60s, RPO near-zero with sync replication
3. Connection pool exhaustion
   - Containment: bulkhead pattern — separate pools per service; set max connections with queuing + backpressure
4. Payment gateway timeout
   - Containment: timeout + retry with exponential backoff (max 3 tries)
   - Recovery: fallback to secondary PSP or queue order for async retry, notify user "processing"