Designing System One AI Workflows

Don't ask an AI to "handle" a task end-to-end. Instead:

1. Write the workflow as normal code (control flow, branching, side effects).
2. Wherever a decision requires interpreting unstructured input, replace that step with one or more atomic, typed questions.
3. Send only the state each question needs.
4. Run all independent questions in parallel in a single request.
5. Combine answers with deterministic code (weighted sums, rules, thresholds) and route on confidence.

Python
# Bad: one broad AI judgment hides many sub-decisions
{"is_spam": {"type": "noul", "instructions": "Is `message` spam?"}}

# Good: decomposed, atomic, inspectable judgments
{
  "requests_credentials": {"type": "noul", "instructions": "Does `message.body` ask for a password or login credential?"},
  "creates_time_pressure": {"type": "noul", "instructions": "Does `message.body` pressure the recipient to act quickly?"},
  "sender_identity_mismatch": {"type": "noul", "instructions": "Does the org in `message.sender.display_name` conflict with `message.sender.email`?"}
}

Combine in code:

Python
answers = response.answers
spam_score = (
    0.4 * answers["requests_credentials"].noul +
    0.3 * answers["creates_time_pressure"].noul +
    0.3 * answers["sender_identity_mismatch"].noul
)

Progress:

• Step 1: Identify deterministic parts of the workflow — keep them in code
• Step 2: Identify the minimum state each remaining decision needs
• Step 3: Decompose each broad judgment into narrow, atomic, typed questions
• Step 4: Add structure (nested paths, criteria objects) where ambiguity exists
• Step 5: Batch all independent questions into one parallel request
• Step 6: Compose answers in code (rules, weighted sums, or as ML features)
• Step 7: Route on confidence — auto-act, review, or escalate

Step 1: Use code when you can

Deterministic rules (dates, thresholds, math, lookups) belong in code, not in a model call.

Python
days_overdue = (today - invoice.due_date).days
if days_overdue > 30:
    route_to_collections(invoice)

Never use an agentic while-loop where a plain software workflow suffices — every loop iteration is a chance to go off the rails.

Step 2: Decompose the input state

Send only the fields relevant to the current questions. Don't rely on model world-knowledge when your own data source has the authoritative answer (e.g., pass in refund_policy text rather than expecting the model to know it).

Step 3: Use structure in the input state

Use nested JSON for state. Point questions at exact paths with backticks (e.g., `support.tickets[0].message`) to remove ambiguity about what's being evaluated.

Step 4: Decompose the questions (most important step)

Never ask one broad question that hides multiple judgments (e.g., "is this spam?", "is this tool trace correct?"). Instead, ask one atomic question per property:

• Each question should test exactly one condition.
• Each question should be independently answerable without depending on another question's answer.
• Prefer noul (boolean-like probability), choice, or score types depending on the judgment shape.

Step 5: Use structure in the questions

When instructions or criteria need multiple kinds of guidance (what it is, what it's not, examples), use objects/arrays with named fields instead of one dense prose string:

JSON
{
  "card_help_topic": {
    "type": "choice",
    "instructions": {"question": "...", "focus": "..."},
    "criteria": {
      "option_a": {"what": "...", "not_for": "...", "examples": ["..."]},
      "option_b": {"what": "...", "not_for": "...", "examples": ["..."]}
    }
  }
}

Use identical field names across options/criteria so the model can compare them directly. Keep short, unambiguous questions as plain strings.

Step 6: Ask a lot of questions per request

Batch many narrow questions about the same state into a single request. They evaluate in parallel — decomposition does not add latency or round trips.

Step 7: Combine question outputs in code

Python
quality = (
    0.4 * answers["answers_request"].noul +
    0.4 * answers["citations_are_supported"].noul +
    0.2 * (1 - answers["contradicts_context"].noul)
)

If training a downstream classical ML model, use these probabilities as input features.

Step 8: Route on uncertainty

Python
answer = response.answers["card_help_topic"]
if answer.confidence < 0.8:
    route_to_human_review(ticket)
else:
    route_to_handler(answer.choice, ticket)

Calibrate thresholds by plotting confidence against accuracy on real data.

Example 1: Spam detection Input: One broad question — "is_spam": "Is message spam?" Output: Decomposed into requests_credentials, offers_unexpected_reward, creates_time_pressure, sender_identity_mismatch, link_domain_mismatch, disguises_link_destination — each independently scored, then combined in code into a single spam probability.

Example 2: Tool-call trace verification Input: One broad question — "tool_calls_are_correct": "Is trace.tool_calls correct?" Output: Decomposed per tool call into relevance, argument-schema conformance, argument-value correctness, and result-linkage checks (e.g., geocode_tool_is_relevant, geocode_arguments_match_schema, weather_uses_geocoded_coordinates), each independently verifiable.

Example 3: Support ticket triage Input: Incoming ticket + customer record. Output: Code short-circuits closed tickets deterministically; sends only relevant state (message, sender, open orders, policy) into parallel questions (topic choice, requests_credentials noul, sender_identity_mismatch noul, unexpected_reward noul, refund_requested noul); code composes results and routes by confidence.