Designing Microservices Architecture

When asked to design or evaluate a microservices architecture, work through this stack in order — each layer depends on the one before it:

1. Service boundaries — decompose by business capability (DDD bounded contexts), not by technical layer
2. Communication — pick sync (REST/gRPC) vs async (events/queues) per interaction, not globally
3. Discovery & gateway — how services find each other, how clients find the system
4. Resilience — timeouts, retries, circuit breakers, bulkheads at every network call
5. Observability — distributed tracing, correlation IDs, centralized logs/metrics before you need them

Don't start with technology selection (Kubernetes, Kong, Istio). Start with boundaries and communication patterns — tech choices fall out of those decisions.

Progress:

• Step 1: Identify service boundaries via domain decomposition
• Step 2: Classify each inter-service interaction (sync/async, read/write, internal/external)
• Step 3: Choose discovery + gateway pattern
• Step 4: Design failure handling per dependency
• Step 5: Instrument for observability
• Step 6: Select concrete providers/tools per layer

Step 1 — Service boundaries Draw boundaries around business capabilities that change for the same reason and are owned by one team. Red flags of wrong boundaries: two services always deployed together, a single DB table written by multiple services, chatty synchronous calls for every user action.

Step 2 — Classify interactions For every edge between services, decide:

• Does the caller need an immediate response? → sync (REST/gRPC)
• Can it be eventually consistent? → async (event/queue)
• Is it internal (same trust boundary) or external (needs auth/rate limiting)? → gateway vs. direct mesh call

Step 3 — Discovery + gateway

• North-south traffic (client → system): API gateway (Kong, Amazon API Gateway, Tyk, Apache APISIX)
• East-west traffic (service → service): service mesh (Istio/Ambient Mesh, AWS App Mesh) or lightweight registry (Amazon Cloud Map, ZooKeeper, Consul-equivalent)
• Don't adopt a full service mesh for <10 services — a mesh's operational cost often exceeds its benefit below that scale. Use direct discovery + sidecar-free libraries instead.

Step 4 — Resilience Every synchronous network call needs: timeout, retry budget, circuit breaker. Every async consumer needs: idempotency, dead-letter handling, backpressure. Document these per dependency, not as a vague platform-wide policy.

Step 5 — Observability Non-negotiable from day one: correlation/trace IDs propagated on every call, centralized structured logs, distributed tracing (Zipkin, AWS X-Ray, Apache SkyWalking). Debugging a distributed system without tracing is effectively impossible once you exceed ~5 services.

Step 6 — Provider selection Match concrete tools to layer needs (see table below) rather than picking a single vendor's full stack by default — mixing is normal (e.g., Kong gateway + AWS SQS + Zipkin tracing).

Example 1: Input: "We have a monolithic e-commerce app. Orders, inventory, and payments are tangled in one codebase and deploys are risky." Output:

• Boundaries: split into Orders, Inventory, Payments services — each owns its own data store
• Orders→Inventory: synchronous gRPC call to check stock at checkout (needs immediate consistency)
• Orders→Payments: synchronous call to charge, but publish an OrderPlaced event asynchronously for downstream (shipping, notifications)
• Gateway: Kong in front of all three for client-facing REST, with rate limiting and auth
• Resilience: circuit breaker on Inventory calls (fail checkout gracefully if Inventory is down), retry with backoff on Payments (idempotency key required)
• Observability: trace ID propagated from gateway through all three services; Zipkin for trace visualization
• Migration path: strangler-fig — route new Inventory reads through the new service while the monolith still owns writes, cut over incrementally

Example 2: Input: "Which is better for inter-service communication, REST or events?" Output: Neither universally — classify per interaction. Use REST/gRPC when the caller needs a synchronous answer to proceed (e.g., "is this item in stock"). Use events when the action can be eventually consistent and multiple consumers may care (e.g., "order was placed" triggering shipping, email, analytics independently). A system typically needs both; forcing everything into one model causes either brittle sync chains or overcomplicated event choreography for simple request/response needs.