AI Skill Report Card
Architecting Self Healing Systems
Self-Healing Architect
Quick Start14 / 15
Given a system description, produce three artifacts:
- Self-Healing Flow - end-to-end diagram/narrative of detect → diagnose → recover → verify
- Detection Logic - concrete rules/signals that identify failure states
- Recovery Logic - concrete actions taken per failure type, with fallback escalation
Example trigger: "Design self-healing for our API gateway that intermittently times out."
Recommendation▾
Add a third example covering a stateful/distributed consensus scenario (e.g., leader election failure) to show broader applicability
Workflow15 / 15
Progress:
- Step 1: Identify failure domains (what can break: network, process, data, dependency, resource)
- Step 2: Define health signals for each domain (heartbeats, error rates, latency, queue depth, checksum)
- Step 3: Set detection thresholds and confirmation windows (avoid false positives from transient blips)
- Step 4: Map each detected failure to a recovery action (restart, retry, failover, rollback, scale, isolate)
- Step 5: Define escalation path if recovery fails N times (alert human, quarantine component, degrade gracefully)
- Step 6: Define verification step post-recovery (confirm health signal returns to baseline)
- Step 7: Output the three deliverables in structured format
Step Details
Detection Logic must specify:
- Signal source (log, metric, probe, exception)
- Threshold/condition (e.g., 3 consecutive failed health checks in 30s)
- Confidence gate (debounce to avoid flapping)
Recovery Logic must specify:
- Trigger condition (which detection state activates it)
- Action sequence (ordered steps, e.g., retry with backoff → restart process → failover to replica)
- Circuit breaker / max attempts before escalation
- Rollback safety (does the action risk data loss? if so, gate behind confirmation or safe-mode)
Self-Healing Flow ties it together as a loop:
Monitor → Detect Anomaly → Diagnose Root Cause → Select Recovery Strategy
→ Execute Recovery → Verify Health → (loop back to Monitor)
↳ If fails repeatedly → Escalate to Human
Recommendation▾
Include a brief section on testing/validating self-healing logic (chaos engineering, fault injection)
Examples19 / 20
Example 1: Input: "API gateway times out intermittently under load."
Output:
Self-Healing Flow:
Monitor request latency & error rate → Detect P95 latency > 2s for 60s
→ Diagnose (check upstream service health, connection pool saturation)
→ Recovery: scale connection pool, then restart gateway pod if unresolved
→ Verify latency returns to baseline within 90s → Resume monitoring
→ If 3 recovery cycles fail: escalate to on-call + enable degraded mode (serve cached responses)
Detection Logic:
- Signal: P95 latency metric, sampled every 10s
- Condition: P95 > 2000ms sustained for 6 consecutive samples (60s)
- Confidence gate: require 2 independent probes (internal + external) to agree
- Secondary signal: connection pool utilization > 90%
Recovery Logic:
- Trigger: latency breach confirmed
- Action 1: Increase connection pool size by 50% (soft recovery, no downtime)
- Action 2 (if unresolved in 60s): rolling restart of gateway instances, one at a time
- Action 3 (if unresolved after restart): failover traffic to standby region
- Max attempts: 3 cycles, then escalate to human + activate degraded/cached mode
- Rollback safety: pool resize is non-destructive; restart is instance-by-instance to avoid full outage
Example 2: Input: "Batch data pipeline occasionally produces corrupted output files."
Output:
Self-Healing Flow:
Monitor output checksum & schema validation → Detect validation failure
→ Diagnose (identify failing stage via stage-level checksums)
→ Recovery: quarantine bad output, re-run failing stage from last good checkpoint
→ Verify new output passes validation → Resume pipeline
→ If re-run fails twice: halt pipeline, alert data engineering team
Detection Logic:
- Signal: post-stage schema + checksum validation
- Condition: any validation failure (binary, no threshold needed)
- Confidence gate: none required (deterministic check)
Recovery Logic:
- Trigger: validation failure at stage X
- Action 1: move corrupted output to quarantine bucket (never overwrite)
- Action 2: re-run stage X from last checkpoint with same input
- Action 3 (if fails again): re-run from two checkpoints back
- Max attempts: 2, then halt and alert (do not auto-proceed with unverified data)
- Rollback safety: quarantine ensures no data loss; pipeline halts rather than propagating corruption
Recommendation▾
Provide a compact output template/schema so the three deliverables can be programmatically parsed
Best Practices
- Always separate detection confidence from action severity — cheap/reversible actions (retry, scale) can trigger on weak signals; destructive actions (restart, failover, rollback data) need strong confirmation.
- Use exponential backoff for retries to avoid thundering-herd recovery storms.
- Always define a circuit breaker: cap automated retries and escalate to humans rather than looping forever.
- Verification is mandatory — recovery isn't complete until health signal is confirmed back to baseline, not just "action executed."
- Prefer graceful degradation (serve stale cache, reduced feature set) over full outage while recovering.
- Log every detect/recover cycle for postmortem and threshold tuning.
Common Pitfalls
- Don't trigger destructive recovery (restart, rollback) on a single noisy sample — always require sustained/confirmed signal.
- Don't recover silently forever — missing escalation paths turn self-healing into a mask that hides chronic problems.
- Don't couple detection thresholds too tightly to current traffic patterns — build in adaptive/relative thresholds where possible.
- Don't skip the verify step — an executed recovery action is not the same as a resolved issue.
- Don't let repeated failed recoveries retry indefinitely — always cap attempts and define a safe fallback state.