AI Skill Report Card

Architecting Self Healing Systems

A89·Sep 27, 2026·Source: Web

Self-Healing Architect

14 / 15

Given a system description, produce three artifacts:

  1. Self-Healing Flow - end-to-end diagram/narrative of detect → diagnose → recover → verify
  2. Detection Logic - concrete rules/signals that identify failure states
  3. Recovery Logic - concrete actions taken per failure type, with fallback escalation

Example trigger: "Design self-healing for our API gateway that intermittently times out."

Recommendation▾
Add a third example covering a stateful/distributed consensus scenario (e.g., leader election failure) to show broader applicability
15 / 15

Progress:

  • Step 1: Identify failure domains (what can break: network, process, data, dependency, resource)
  • Step 2: Define health signals for each domain (heartbeats, error rates, latency, queue depth, checksum)
  • Step 3: Set detection thresholds and confirmation windows (avoid false positives from transient blips)
  • Step 4: Map each detected failure to a recovery action (restart, retry, failover, rollback, scale, isolate)
  • Step 5: Define escalation path if recovery fails N times (alert human, quarantine component, degrade gracefully)
  • Step 6: Define verification step post-recovery (confirm health signal returns to baseline)
  • Step 7: Output the three deliverables in structured format

Step Details

Detection Logic must specify:

  • Signal source (log, metric, probe, exception)
  • Threshold/condition (e.g., 3 consecutive failed health checks in 30s)
  • Confidence gate (debounce to avoid flapping)

Recovery Logic must specify:

  • Trigger condition (which detection state activates it)
  • Action sequence (ordered steps, e.g., retry with backoff → restart process → failover to replica)
  • Circuit breaker / max attempts before escalation
  • Rollback safety (does the action risk data loss? if so, gate behind confirmation or safe-mode)

Self-Healing Flow ties it together as a loop:

Monitor → Detect Anomaly → Diagnose Root Cause → Select Recovery Strategy
   → Execute Recovery → Verify Health → (loop back to Monitor)
                                      ↳ If fails repeatedly → Escalate to Human
Recommendation▾
Include a brief section on testing/validating self-healing logic (chaos engineering, fault injection)
19 / 20

Example 1: Input: "API gateway times out intermittently under load."

Output:

Self-Healing Flow:
Monitor request latency & error rate → Detect P95 latency > 2s for 60s
→ Diagnose (check upstream service health, connection pool saturation)
→ Recovery: scale connection pool, then restart gateway pod if unresolved
→ Verify latency returns to baseline within 90s → Resume monitoring
→ If 3 recovery cycles fail: escalate to on-call + enable degraded mode (serve cached responses)

Detection Logic:
- Signal: P95 latency metric, sampled every 10s
- Condition: P95 > 2000ms sustained for 6 consecutive samples (60s)
- Confidence gate: require 2 independent probes (internal + external) to agree
- Secondary signal: connection pool utilization > 90%

Recovery Logic:
- Trigger: latency breach confirmed
- Action 1: Increase connection pool size by 50% (soft recovery, no downtime)
- Action 2 (if unresolved in 60s): rolling restart of gateway instances, one at a time
- Action 3 (if unresolved after restart): failover traffic to standby region
- Max attempts: 3 cycles, then escalate to human + activate degraded/cached mode
- Rollback safety: pool resize is non-destructive; restart is instance-by-instance to avoid full outage

Example 2: Input: "Batch data pipeline occasionally produces corrupted output files."

Output:

Self-Healing Flow:
Monitor output checksum & schema validation → Detect validation failure
→ Diagnose (identify failing stage via stage-level checksums)
→ Recovery: quarantine bad output, re-run failing stage from last good checkpoint
→ Verify new output passes validation → Resume pipeline
→ If re-run fails twice: halt pipeline, alert data engineering team

Detection Logic:
- Signal: post-stage schema + checksum validation
- Condition: any validation failure (binary, no threshold needed)
- Confidence gate: none required (deterministic check)

Recovery Logic:
- Trigger: validation failure at stage X
- Action 1: move corrupted output to quarantine bucket (never overwrite)
- Action 2: re-run stage X from last checkpoint with same input
- Action 3 (if fails again): re-run from two checkpoints back
- Max attempts: 2, then halt and alert (do not auto-proceed with unverified data)
- Rollback safety: quarantine ensures no data loss; pipeline halts rather than propagating corruption
Recommendation▾
Provide a compact output template/schema so the three deliverables can be programmatically parsed
  • Always separate detection confidence from action severity — cheap/reversible actions (retry, scale) can trigger on weak signals; destructive actions (restart, failover, rollback data) need strong confirmation.
  • Use exponential backoff for retries to avoid thundering-herd recovery storms.
  • Always define a circuit breaker: cap automated retries and escalate to humans rather than looping forever.
  • Verification is mandatory — recovery isn't complete until health signal is confirmed back to baseline, not just "action executed."
  • Prefer graceful degradation (serve stale cache, reduced feature set) over full outage while recovering.
  • Log every detect/recover cycle for postmortem and threshold tuning.
  • Don't trigger destructive recovery (restart, rollback) on a single noisy sample — always require sustained/confirmed signal.
  • Don't recover silently forever — missing escalation paths turn self-healing into a mask that hides chronic problems.
  • Don't couple detection thresholds too tightly to current traffic patterns — build in adaptive/relative thresholds where possible.
  • Don't skip the verify step — an executed recovery action is not the same as a resolved issue.
  • Don't let repeated failed recoveries retry indefinitely — always cap attempts and define a safe fallback state.
0
Grade AAI Skill Framework
Scorecard
Criteria Breakdown
Quick Start
14/15
Workflow
15/15
Examples
19/20
Completeness
18/20
Format
15/15
Conciseness
14/15