AI Skill Report Card
Designing Observability Stacks
Quick Start14 / 15
Given a system description, produce three deliverables:
- Observability Stack — tools/pipeline for logs, traces, metrics
- Metrics Design — what to measure, how, and cardinality considerations
- Alert Strategy — what pages, what tickets, thresholds, and routing
Example prompt: "Design observability for a Go microservice handling payment processing (500 req/s, calls 2 downstream APIs)."
Immediate output shape:
Recommendation▾
Add a third example showing a smaller-scale or non-HTTP system (e.g., batch/event-driven pipeline) to broaden coverage beyond request-driven services.
Observability Stack
- Logging: structured JSON via [tool], shipped to [backend]
- Tracing: OpenTelemetry SDK, sampled at X%, exported to [backend]
- Metrics: Prometheus client, scraped every 15s
- Correlation: trace_id injected into logs
Metrics Design
- RED metrics (Rate, Errors, Duration) per endpoint
- USE metrics (Utilization, Saturation, Errors) per resource
- Business metrics: e.g. payment_success_total
Alert Strategy
- SLO-based alerting (burn rate)
- Page: error rate > 5% for 5m
- Ticket: latency p99 > 500ms for 15m
Workflow15 / 15
Progress:
- [ ] Understand system topology (services, dependencies, traffic profile)
- [ ] Identify critical user journeys / SLIs
- [ ] Design logging strategy (structure, levels, retention)
- [ ] Design tracing strategy (instrumentation points, sampling)
- [ ] Design metrics (RED/USE + business KPIs)
- [ ] Define SLOs and error budgets
- [ ] Define alert rules (page vs ticket vs dashboard-only)
- [ ] Define dashboards per audience (on-call, eng, business)
- [ ] Document runbook links per alert
Step 1: Understand the system Ask (or infer): language/stack, request volume, latency sensitivity, dependencies, current pain points (blind spots, alert fatigue, MTTR issues).
Step 2: Logging
- Use structured logging (JSON), never free-text in production.
- Standard fields:
timestamp,level,service,trace_id,span_id,message, contextual fields. - Log levels: ERROR (actionable failures), WARN (degraded but recovering), INFO (state changes), DEBUG (dev only, disabled in prod by default).
- Avoid logging PII/secrets; define redaction rules.
- Set retention by tier: hot (7d, fast query) → cold (90d+, cheap storage).
Step 3: Tracing
- Instrument service boundaries: inbound requests, outbound calls (DB, HTTP, queue), critical internal spans.
- Use OpenTelemetry as vendor-neutral standard unless stack already commits elsewhere.
- Sampling: head-based for cost control (e.g., 5-10% baseline) + tail-based/always-sample for errors and high latency.
- Propagate context across async boundaries (queues, background jobs) — this is the most commonly missed part.
Step 4: Metrics
- Apply RED for request-driven services: Rate, Errors, Duration (per route/method).
- Apply USE for resources: Utilization, Saturation, Errors (CPU, memory, connection pools, queues).
- Add business metrics tied to outcomes (e.g.,
orders_completed_total,payment_failed_total{reason}). - Watch cardinality: never use unbounded labels (user_id, request_id) as metric labels — use them in traces/logs instead.
- Use histograms for latency, not averages; report p50/p95/p99.
Step 5: SLOs and Alerting
- Define SLIs first (e.g., availability = successful requests / total requests).
- Set SLOs with error budgets (e.g., 99.9% over 30 days).
- Alert on symptoms, not causes — alert on error rate/latency breach, not "CPU high."
- Use multi-window, multi-burn-rate alerts for SLO-based paging (fast burn = page, slow burn = ticket).
- Every page-level alert must link to a runbook.
- Route: page → on-call (actionable, urgent), ticket → backlog (non-urgent), dashboard-only → informational.
Step 6: Dashboards
- On-call dashboard: current health, RED metrics, active alerts, recent deploys.
- Engineering dashboard: trends, dependency health, resource saturation.
- Business dashboard: KPIs, conversion, volume trends.
Recommendation▾
Include a brief note on cost tradeoffs (e.g., trace storage costs vs sampling rate) to strengthen decision-making guidance.
Examples18 / 20
Example 1: Input: "E-commerce checkout service, Node.js, 200 req/s peak, depends on inventory-service and payment-gateway."
Output:
Recommendation▾
Consider adding a short section on tool selection criteria (when to choose Prometheus vs Datadog vs vendor-managed stacks) since 'tools/pipeline' is promised in Quick Start but mostly assumes one stack throughout.
Observability Stack
- Logging: pino (JSON) → Loki, 14-day hot retention
- Tracing: OpenTelemetry Node SDK, auto-instrument HTTP/DB, export to Tempo
- Sampling: 10% baseline, 100% for 5xx and >2s latency
- Metrics: prom-client → Prometheus, 15s scrape interval, Grafana dashboards
- Correlation: trace_id in every log line via OTel context
Metrics Design
- checkout_requests_total{status}
- checkout_duration_seconds (histogram, buckets tuned to 100ms-5s)
- inventory_check_errors_total{reason}
- payment_gateway_latency_seconds (histogram)
- checkout_completed_total / checkout_abandoned_total (business KPI)
- Avoid: labeling by user_id or cart_id (unbounded cardinality)
Alert Strategy
- Page: checkout success rate < 99% over 5m (fast burn on 99.5% SLO)
- Page: payment_gateway p99 latency > 3s for 5m
- Ticket: inventory_check_errors_total rate > 1% over 1h (slow burn)
- Dashboard-only: checkout_abandoned_total trend (business review)
- Every page alert links to runbook: "Checkout Degradation Response"
**Example 2:**
Input: "We have Prometheus and Grafana but constant alert fatigue — 40+ pages a week, most not actionable."
Output:
Diagnosis
- Alerts likely cause-based (CPU, memory) instead of symptom-based (SLO breach)
- Missing multi-window burn-rate logic → single-window noise on transient spikes
Observability Stack
- Keep Prometheus/Grafana; add Alertmanager routing tiers (page/ticket/silence)
Metrics Design
- Re-derive SLIs from user-facing symptoms (error rate, latency, availability)
- Deprecate resource-based alerts; convert to dashboard panels only
Alert Strategy
- Replace single-threshold alerts with multi-window burn rate (1h + 6h windows)
- Page only on fast-burn (budget exhausted in <2 days at current rate)
- Ticket on slow-burn (budget exhausted in <30 days)
- Target: reduce pages to <5/week, all actionable with runbooks
Best Practices
- Instrument at the edge first (ingress/egress), then fill in internals — highest signal-to-effort ratio.
- Correlate all three pillars via shared
trace_id/request_id. - Treat cardinality budget as a first-class design constraint, not an afterthought.
- Alert on user-facing symptoms; use cause-based metrics for debugging/dashboards, not paging.
- Every alert must be actionable — if there's no runbook or action, it shouldn't page.
- Default to open standards (OpenTelemetry) to avoid vendor lock-in.
- Version and review alert rules like code (PRs, changelogs).
Common Pitfalls
- High-cardinality labels (user IDs, emails, UUIDs) on metrics — causes cardinality explosion.
- Alerting on infrastructure causes (CPU/memory) instead of user impact.
- No sampling strategy for tracing — either drowning in cost or missing error traces.
- Logging without correlation IDs — impossible to connect logs to traces during incidents.
- Using averages instead of percentiles for latency — hides tail latency problems.
- Alert thresholds picked arbitrarily without SLO backing — leads to alert fatigue or missed incidents.
- No runbook linked to alerts — increases MTTR and on-call burnout.
- Treating dashboards as a substitute for alerting, or alerting as a substitute for dashboards — they serve different purposes.