AI Skill Report Card
Debugging Kubernetes and Terraform
Quick Start14 / 15
Kubernetes debugging triage:
Bashkubectl get pods -A | grep -v Running kubectl describe pod <pod> -n <namespace> kubectl logs <pod> -n <namespace> --previous kubectl get events -n <namespace> --sort-by='.lastTimestamp'
Terraform module skeleton:
modules/<name>/
├── main.tf # resources
├── variables.tf # inputs with types, validation, descriptions
├── outputs.tf # outputs other modules/root need
├── versions.tf # required_providers + version constraints
└── README.md # usage example
Recommendation▾
Split into two focused skills (K8s debugging vs Terraform authoring) since they're distinct domains bundled together, which dilutes focus
Workflow14 / 15
Kubernetes Debugging
Progress:
- Identify symptom scope (one pod, one node, cluster-wide?)
- Check pod status and events
- Check logs (current + previous container if crashed)
- Check resource constraints (requests/limits, node capacity)
- Check networking (service, endpoints, DNS, network policies)
- Check control plane health if cluster-wide
- Form hypothesis, apply minimal fix, verify
- Scope the problem.
kubectl get pods -A -o wideto see if it's isolated or widespread. Isolated → workload issue. Widespread → node/control-plane/network issue. - Pod not starting: check
Eventsindescribe podfirst — scheduling failures (resource, taints, affinity), image pull errors, and volume mount issues show here immediately. - Pod crashing (CrashLoopBackOff):
kubectl logs --previousfor the last crash's output. Check exit code indescribe pod— 137 = OOMKilled, 1 = app error, 143 = SIGTERM. - Pod running but not working: check readiness/liveness probes, then service endpoints (
kubectl get endpoints <svc>) to confirm pod is actually selected, then DNS (kubectl exec -it <pod> -- nslookup <svc>). - Node issues:
kubectl describe node <node>for conditions (MemoryPressure, DiskPressure, PIDPressure), thenkubectl top nodefor actual usage. - Cluster-wide weirdness: check control plane component health, etcd, and recent changes (deployments, upgrades, CNI/CSI plugin versions).
Terraform Module Writing
Progress:
- Define clear single-responsibility scope for the module
- Write variables.tf with types, defaults, and validation blocks
- Write main.tf using data sources over hardcoded values
- Write outputs.tf exposing only what consumers need
- Pin provider versions in versions.tf
- Add README with a working example
- Run
terraform validateandtflint
- Scope the module to one logical unit (e.g., "an EKS cluster with node groups", not "all of AWS networking").
- Every variable gets a
description, atype, andvalidationwhere constraints exist (e.g., CIDR format, allowed enum values). - Avoid hardcoding account/region-specific values — pull via
datasources or pass as variables. - Use
for_eachovercountfor resources keyed by name (avoids reindex-on-removal issues). - Output IDs/ARNs/names needed by consumers; don't leak internal-only resources.
- Pin provider version constraints (
~> major.minor) inversions.tf, not loosely in root.
Recommendation▾
Add a bad-output example (e.g., a common misdiagnosis or anti-pattern Terraform module) to contrast with good examples
Examples15 / 20
Example 1:
Input: Pod stuck in Pending for 10 minutes.
Output:
Bashkubectl describe pod myapp-7f9c -n prod # Events: "0/5 nodes are available: 5 Insufficient cpu"
Diagnosis: resource requests exceed available node capacity. Fix: lower resources.requests.cpu, add nodes, or enable cluster autoscaler.
Example 2: Input: Need a Terraform module for an S3 bucket with versioning and encryption, reused across 3 environments. Output:
HCL# variables.tf variable "bucket_name" { type = string description = "Name of the S3 bucket" validation { condition = can(regex("^[a-z0-9.-]{3,63}$", var.bucket_name)) error_message = "Bucket name must be valid S3 naming convention." } } variable "enable_versioning" { type = bool default = true }
HCL# main.tf resource "aws_s3_bucket" "this" { bucket = var.bucket_name } resource "aws_s3_bucket_versioning" "this" { bucket = aws_s3_bucket.this.id versioning_configuration { status = var.enable_versioning ? "Enabled" : "Suspended" } } resource "aws_s3_bucket_server_side_encryption_configuration" "this" { bucket = aws_s3_bucket.this.id rule { apply_server_side_encryption_by_default { sse_algorithm = "aws:kms" } } }
Called per-environment with bucket_name = "myapp-${var.env}-data".
Recommendation▾
Include an example for a networking/DNS failure scenario since that's called out in the workflow but not demonstrated in Examples
Best Practices
- Always check
kubectl get eventsbefore logs — scheduling and infra issues show there, not in container logs. - Use
kubectl exec -it <pod> -- shto debug from inside, but prefer ephemeral debug containers (kubectl debug) for distroless/minimal images with no shell. - In Terraform, keep state blast radius small: separate state per environment/module, never one giant monolith state.
- Use
terraform plan -out=tfplanand review diffs before every apply, especially for destructive changes (shown as-/+). - Tag/label all cloud resources from Terraform for cost tracking and ownership.
Common Pitfalls
- Don't assume
CrashLoopBackOffis the root cause — it's a symptom; always check the exit code and previous logs. - Don't restart pods to "fix" issues without checking events/logs first — you lose the previous container's state.
- Don't use
countfor Terraform resources that may be reordered/removed from the middle of a list — causes unnecessary destroy/recreate. - Don't hardcode region/account IDs in modules — breaks reusability across accounts.
- Don't skip
terraform planreview on shared infrastructure — unreviewed applies are the top cause of outages in IaC-managed environments.