Designing Agentic LLM Systems
Before writing any code, answer this question: can a single, well-crafted LLM call with good context/retrieval solve this?
- Yes → Just do that. Don't build a system.
- No, but the steps are fixed and predictable → Use a workflow pattern (see below).
- No, and the steps/number of iterations can't be predicted in advance → Use an agent.
Start with direct LLM API calls, not a framework. Most patterns below are ~50-150 lines of plain code. Add a framework only after you understand what it would abstract away.
Progress:
- [ ] Step 1: Determine if a single augmented LLM call suffices
- [ ] Step 2: If not, identify whether the task decomposes into fixed steps (workflow) or open-ended steps (agent)
- [ ] Step 3: Pick the specific workflow pattern that matches the task shape
- [ ] Step 4: Implement with direct API calls; add tools/retrieval/memory as needed
- [ ] Step 5: Add programmatic gates/checks between steps for reliability
- [ ] Step 6: Evaluate against real examples; only add complexity that measurably improves results
- [ ] Step 7: If building an agent, invest heavily in tool design (the "agent-computer interface")
Step 1: The augmented LLM
The base unit is an LLM plus retrieval, tools, and memory. Design these augmentations for the specific use case, with a clean, well-documented interface the model can reliably use (e.g., via MCP). Every pattern below assumes calls have access to these augmentations.
Step 2-3: Choose a pattern by task shape
| Task shape | Pattern |
|---|---|
| Fixed sequence of subtasks, each simpler than the whole | Prompt chaining |
| Distinct input categories needing different handling | Routing |
| Independent subtasks that can run simultaneously, or need multiple independent attempts | Parallelization |
| Subtasks unknown ahead of time, depend on the input | Orchestrator-workers |
| Clear evaluation criteria, benefits from iterative critique | Evaluator-optimizer |
| Open-ended, unpredictable number of steps, needs autonomous tool use over time | Agent |
Prompt chaining — Decompose into sequential LLM calls, output of one feeds the next. Add a "gate" (programmatic check) between steps to catch drift early. Trades latency for accuracy by making each call an easier task.
- Examples: draft copy → translate; write outline → validate outline against criteria → write full doc from outline.
Routing — Classify the input first, then dispatch to a specialized downstream prompt/model. Keeps prompts specialized instead of one prompt trying to handle everything.
- Examples: route support queries (general/refund/technical) to different flows; route easy queries to a cheap/fast model (e.g. Haiku) and hard ones to a stronger model (e.g. Sonnet).
Parallelization — Two variants:
- Sectioning: split task into independent parts run concurrently, then aggregate (e.g., one call generates a response, a separate call screens for policy violations — better than one call doing both).
- Voting: run the same task multiple times for diverse outputs and aggregate/vote (e.g., multiple independent code-vulnerability reviews; multiple content-moderation judgments with different thresholds).
Orchestrator-workers — A central LLM dynamically decomposes the task and spins up worker LLM calls per subtask, then synthesizes results. Unlike parallelization, subtasks aren't predefined — the orchestrator decides them based on the specific input.
- Examples: coding agent that determines which files need changes and how; multi-source research/search synthesis.
Evaluator-optimizer — One LLM generates, another evaluates and gives feedback, loop until the evaluator is satisfied. Works when: (1) feedback demonstrably improves output, and (2) an LLM can plausibly generate that feedback.
- Examples: literary translation with an evaluator critiquing nuance; iterative search-and-analysis where an evaluator decides if more searching is needed.
Agent — LLM operates in a loop: takes action (tool call), observes environment feedback (ground truth — tool results, execution output), decides next step, repeats until done or a stopping condition (max iterations, human checkpoint) is hit. Structurally simple; success hinges on tool/environment design, not clever orchestration logic.
- Use when: task is open-ended, step count is unpredictable, and you can trust the model's judgment in a semi-sandboxed environment.
- Examples: SWE-bench-style coding agents editing multiple files; computer-use agents.
Example 1: Input: "Build a system that answers customer support tickets, pulls order history, and can issue refunds." Output: An agent, not a workflow — conversation flow is open-ended, but success (resolution) is clearly measurable and tools (lookup order, issue refund, search KB) provide ground truth at each step. Add human-in-the-loop for refund approval above a threshold.
Example 2: Input: "Take a rough blog draft, ensure it hits SEO keywords, then translate it into 3 languages." Output: Prompt chaining — draft check (gate: does it hit keyword targets?) → revise → translate ×3 as parallel sectioned calls.
Example 3: Input: "Given a GitHub issue, make the necessary code changes across the repo." Output: Orchestrator-workers agent — orchestrator reads the issue, determines which files are relevant (unpredictable ahead of time), delegates edits to worker calls, synthesizes into a diff, runs tests as ground truth feedback.
- Default to simplicity. Add a workflow only when a single call underperforms; add an agent only when a workflow can't handle the unpredictability.
- Measure before adding complexity. Every additional LLM call, loop, or agent step should be justified by evaluated improvement, not intuition.
- Use direct API calls first. Frameworks (LangGraph-style tools, Rivet, Vellum, etc.) add abstraction that can hide prompts/responses and make debugging harder. If used, fully understand what's happening underneath.
- Give agents ground truth at every step. Tool call results, test outputs, execution logs — these let the agent (and you) verify progress instead of drifting on assumptions.
- Invest in the agent-computer interface (ACI). Tool definitions and docs deserve the same rigor as a human-facing API: clear names, explicit parameter docs, examples of correct usage, and testing against real agent behavior — not just human intuition about clarity.
- Add stopping conditions to agents. Max iterations, cost budgets, or explicit human checkpoints prevent runaway loops and compounding errors.
- Use cheaper/faster models via routing for the easy majority of cases; reserve expensive models for the hard tail.
- Sandbox agents during testing. Autonomy means higher cost and compounding-error risk; validate extensively before granting real-world side effects (refunds, deployments, etc.).
- Reaching for an "agent" when a simple prompt chain or even a single augmented call would do — this needlessly increases latency, cost, and failure surface.
- Adopting a heavyweight framework before understanding the 3-line version of the pattern it wraps.
- Letting one LLM call handle both the core task and guardrails/safety checks — separate these into parallel sectioned calls instead.
- Predefining subtasks for problems whose subtasks are genuinely input-dependent (use orchestrator-workers, not static parallelization).
- Skipping intermediate gates/checks in prompt chains, allowing errors to silently compound down the chain.
- Under-documenting tools for agents — vague tool descriptions are a leading cause of unreliable agent behavior; treat tool docs like a public API contract.
- Running agents indefinitely without max-iteration or cost limits.