Engineering DevOps Infrastructure

Given a service/application description, produce five deliverables:

1. Infrastructure Diagram (components + connections)
2. Deployment Flow (CI/CD stages)
3. Monitoring Stack (metrics/alerts/dashboards)
4. Disaster Recovery (backup/failover plan)
5. Scaling Strategy (horizontal/vertical rules)

Example trigger: "Design the infrastructure for our new payment API service."

Progress:

• Step 1: Clarify service topology — what components exist (API, DB, cache, queue, workers)?
• Step 2: Draft Infrastructure Diagram (containers, K8s objects, networking, ingress)
• Step 3: Define Deployment Flow (build → test → containerize → push → deploy → verify)
• Step 4: Specify Monitoring Stack (metrics collector, log aggregator, alerting, dashboards)
• Step 5: Write Disaster Recovery plan (backup cadence, RTO/RPO, failover steps)
• Step 6: Write Scaling Strategy (triggers, thresholds, HPA/VPA config, capacity limits)
• Step 7: Review for consistency across all five outputs (same components referenced throughout)

Step details

Infrastructure Diagram — describe in text/ASCII form:

• Client → Ingress/LoadBalancer → Service → Pods → StatefulSet/DB
• Namespaces, network policies, secrets management (Vault/K8s Secrets)
• External dependencies (managed DB, object storage, CDN)

Deployment Flow:

• Source push → CI build → unit/integration tests → Docker image build → push to registry → K8s manifest apply (via ArgoCD/Helm/kubectl) → rolling update → smoke test → rollback trigger if failed

Monitoring Stack (default recommendation):

• Metrics: Prometheus + node-exporter + kube-state-metrics
• Visualization: Grafana dashboards (per-service, per-node)
• Logging: Loki or ELK (Elasticsearch/Fluentd/Kibana)
• Tracing: Jaeger/OpenTelemetry for distributed tracing
• Alerting: Alertmanager → Slack/PagerDuty, define SLO-based alert rules

Disaster Recovery:

• Backup schedule (DB snapshots, etcd backups for K8s)
• RTO (Recovery Time Objective) and RPO (Recovery Point Objective) targets
• Multi-AZ/multi-region failover plan
• Runbook: detection → isolate → restore → validate → postmortem

Scaling Strategy:

• HPA rules based on CPU/memory/custom metrics (requests/sec, queue depth)
• Cluster autoscaler for node pool
• Vertical scaling for stateful components (DB read replicas)
• Load testing thresholds that trigger scale-up review

Example 1: Input: "Design infrastructure for a Node.js REST API with PostgreSQL, expecting 10k daily active users."

Output:

INFRASTRUCTURE DIAGRAM
Client → CloudFlare CDN → NGINX Ingress → K8s Service (ClusterIP)
  → Deployment (3 replicas, Node.js API pods)
  → PostgreSQL (managed RDS, Multi-AZ) + Redis (session cache)
  Namespace: production | Secrets: K8s Secrets + Sealed Secrets

DEPLOYMENT FLOW
GitHub push → GitHub Actions (lint, test, build)
  → Docker build & push to ECR
  → Helm upgrade --install (staging) → smoke tests
  → Manual approval gate → Helm upgrade (production, rolling update, maxSurge=1)
  → Post-deploy health check → auto-rollback on failure

MONITORING STACK
- Prometheus scrapes /metrics from API pods + node-exporter
- Grafana dashboards: request latency (p50/p95/p99), error rate, pod CPU/mem
- Loki + Promtail for log aggregation, searchable via Grafana
- Alertmanager: error rate >5% for 5min → PagerDuty page

DISASTER RECOVERY
- RDS automated daily snapshots, 7-day retention + PITR enabled
- RTO: 30 min, RPO: 5 min
- Failover: RDS Multi-AZ auto-failover; K8s cluster backed by managed control plane (EKS)
- Runbook: alert triggers → on-call verifies DB status → promote standby → update DNS/connection string → validate app health

SCALING STRATEGY
- HPA: scale 3→10 replicas when CPU >70% or req/sec >500 per pod
- Cluster autoscaler adds nodes when pending pods detected
- Redis: vertical scale if memory >80%
- Load test quarterly at 3x current peak to validate thresholds