Debugging Kubernetes and Writing Terraform Modules

Progress:

• Identify blast radius (one pod? one node? cluster-wide?)
• Check pod status and events
• Check logs (current + previous container)
• Check resource pressure (node/pod)
• Check networking (DNS, Service, NetworkPolicy)
• Check control plane health if cluster-wide
• Form hypothesis, verify, fix, confirm resolution

Step detail

1. Scope the problem. kubectl get pods -A -o wide — is it one pod, one node, one namespace, or everywhere? Cluster-wide issues point to control plane, DNS, CNI, or a shared dependency.

2. Pod status diagnosis — match symptom to cause:

   • CrashLoopBackOff → check kubectl logs <pod> --previous; usually app crash, bad config, or failed healthcheck
   • ImagePullBackOff → bad tag, missing registry auth, or network egress blocked
   • Pending → kubectl describe pod Events; usually unschedulable (resource requests, taints, affinity, PVC not bound)
   • OOMKilled (check kubectl describe pod last state reason) → raise memory limits or fix leak
   • Running but not ready → failing readiness probe; check probe config vs actual startup time

3. Logs. Always check both current and --previous logs. Use -f to tail live, --since=10m to scope noise.

4. Resource pressure. kubectl top nodes / kubectl top pods. Compare against requests/limits. Node pressure causes evictions — check kubectl describe node for Conditions (MemoryPressure, DiskPressure).

5. Networking.

   • DNS: kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup <service>
   • Service routing: confirm Endpoints/EndpointSlice has pod IPs (kubectl get endpoints <svc>) — empty means selector mismatch
   • NetworkPolicy: check for policies blocking traffic with kubectl get networkpolicy -A

6. Control plane (cluster-wide symptoms only): kubectl get componentstatuses (if available), check etcd health, API server logs, and node kubelet status via kubectl get nodes conditions.

7. Fix and verify. Apply minimal fix, re-check the original symptom, and check for regressions elsewhere (e.g., HPA reacting to the change).

Progress:

• Define interface (variables/outputs) before resources
• Write resources with sane defaults, no hardcoded values
• Add validation blocks to variables
• Pin provider versions
• Write outputs consumers actually need
• Add README with usage example
• Run terraform validate + tflint + terraform plan against example

Step detail

1. Design the interface first. Decide what the module takes as input and returns as output before writing resources — this is the module's contract.

2. Variables: every variable gets type, description, and sensible default (or no default if required). Use validation blocks for constraints instead of relying on provider-side errors:

   HCL
   variable "environment" {
     type        = string
     description = "Deployment environment"
     validation {
       condition     = contains(["dev", "staging", "prod"], var.environment)
       error_message = "environment must be dev, staging, or prod."
     }
   }

3. No hardcoded values — region, account ID, naming prefixes all come from variables or data sources, never literals.

4. Pin versions in versions.tf:

   HCL
   terraform {
     required_version = ">= 1.5.0"
     required_providers {
       aws = {
         source  = "hashicorp/aws"
         version = "~> 5.0"
       }
     }
   }

5. Outputs expose IDs/ARNs/endpoints consumers need to wire into other modules — not everything the resource has.

6. Tag/label everything created resources for cost tracking and ownership (var.tags merged with module-level defaults).

7. Test with a minimal examples/basic directory that calls the module — this doubles as living documentation and a smoke test via terraform plan.

• Reproduce the minimal failing case before changing cluster-wide config.
• Prefer kubectl describe over get -o yaml for diagnosis — Events are the fastest signal.
• In Terraform, keep modules single-purpose (one module = one logical resource group); compose via root modules, don't build mega-modules.
• Always run terraform plan and read the full diff before apply, especially for destroy/recreate operations.
• Use for_each over count for resource collections to avoid index-shift diffs.
• Store Terraform state remotely with locking (S3+DynamoDB, Terraform Cloud, etc.) — never local state in shared environments.

• Debugging: don't restart pods before capturing logs — you lose the crash evidence (--previous only works if the container already restarted and kept history).
• Debugging: don't assume DNS is fine because one lookup worked — test from the actual failing pod's namespace/context, not a fresh debug pod with different NetworkPolicy exposure.
• Terraform: don't put secrets in variables.tf defaults or commit .tfvars with real credentials — use a secrets manager + data source.
• Terraform: avoid count with conditional logic across resources that depend on each other's indices — it causes brittle, hard-to-read diffs.
• Terraform: don't skip required_version/provider pinning — "works on my machine" state drift is a top cause of broken applies.