AI Skill Report Card
Architecting AI Orchestration Systems
Quick Start13 / 15
Given a target system (e.g., "customer support AI agent" or "content generation pipeline"), produce a single skill.md (or ARCHITECTURE.md) that defines every operational layer of that system using the category template below. Don't theorize — instantiate each section with concrete, working definitions for the actual system being described.
Minimal example instantiation for one category:
Markdown### Hooks **Definition:** Named interception points where custom logic runs before/after a core action. - `pre_generate` — validates prompt token budget before LLM call - `post_generate` — strips PII, logs payload, triggers `on_deliverable_ready` - `on_error` — routes to fallback model, increments retry counter
Recommendation▾
Add a fully worked end-to-end example (all 16 categories filled for one concrete system) rather than partial snippets — current examples only show 3-4 categories per scenario.
Workflow14 / 15
Progress:
- Step 1: Identify the system's boundary (what it does, doesn't do, inputs/outputs)
- Step 2: Walk through all 16 categories below; skip only categories genuinely inapplicable (state why)
- Step 3: For each category, write Definition → Structure/Schema → Concrete Example
- Step 4: Cross-link related categories (e.g., Hooks → Logic Gates → Pipelines)
- Step 5: Add a "System Map" diagram-in-prose showing how categories connect end-to-end
- Step 6: Validate — trace one real request through the whole doc, category by category, and confirm nothing is missing
Recommendation▾
The 16-category list is dense and borders on over-explaining theory; consider trimming definitions to one line each and moving elaboration into the example section.
Category Template (apply to every section)
Each category in the final document must follow this three-part structure:
- Definition — one or two sentences, precise, no fluff
- Structure — schema, pseudocode, table, or list format it takes in this system
- Example — a real, filled-in instance (not a placeholder like
<value>)
The 16 Core Categories
- Actions — atomic, callable units of AI behavior (e.g.,
summarize,classify,generate_image). Define name, inputs, outputs, side effects. - Tasks — a goal-oriented grouping of one or more Actions with success criteria. Define acceptance criteria explicitly.
- Mechanics — the underlying rules governing how Actions execute (retry limits, timeouts, concurrency, state transitions).
- Logic — decision rules and conditionals that route execution (if/else, scoring thresholds, priority rules).
- Hooks — named interception points (
pre_,post_,on_error,on_success) for injecting custom behavior without modifying core flow. - Scaffolding — the reusable skeletal structure (templates, boilerplate prompts, folder/data structures) that tasks are built on top of.
- Systems — the bounded, self-contained unit combining multiple tasks/pipelines toward one purpose (e.g., "Support Ticket Triage System").
- Cognition — how the AI reasons: chain-of-thought depth, memory/context strategy, reasoning mode (fast/heuristic vs. deliberate/multi-step).
- Architecture — the technical layout: model(s) used, data flow, storage, APIs, orchestration layer (e.g., agent framework, queue, event bus).
- Workflows — human-readable, ordered sequences of Tasks with decision points and handoffs (can include human-in-the-loop steps).
- Pipelines — the automated, machine-executed version of a Workflow: stages, data transforms, checkpoints, rollback behavior.
- Schedules — timing and trigger definitions (cron, event-driven, on-demand) governing when Pipelines/Tasks run.
- Logic Gates — explicit boolean/conditional checkpoints that permit or block progression (e.g.,
AND(confidence>0.8, no_pii_detected)). - Methods & Benchmarking — the evaluation approach: metrics, test sets, scoring rubrics, and how success/regression is measured.
- Prompts & Personalization — prompt templates, variable slots, and user-preference injection strategy (tone, format, history-based adaptation).
- Deliverables, Payloads & Assets — the final generative outputs: schema of the payload, asset types (text/image/code/file), versioning and delivery format.
Examples14 / 20
Example 1: Input: "Document the architecture for an AI blog-writing pipeline with user tone personalization." Output: A skill.md with all 16 sections filled in, e.g.:
- Actions:
outline_generate,draft_write,tone_adjust,seo_check - Logic Gates:
IF seo_score < 70 THEN route_to(seo_check_retry) - Prompts & Personalization: template
Write in {{user.tone_profile}} for audience {{user.icp}}with fallback todefault_tone: "neutral-professional" - Deliverables: payload schema
{title, body_md, meta_description, tone_score, version}
Example 2: Input: "System is a real-time fraud-scoring agent." Output:
- Mechanics: max latency 200ms, no retries on timeout (fail-open with flag), stateless per request
- Logic Gates:
AND(risk_score > 0.9, transaction_amount > threshold) → block - Schedules: event-driven only (no cron), triggered by
transaction.created - Methods & Benchmarking: precision/recall on labeled fraud dataset, weekly drift check against last 30 days
Recommendation▾
Include a bad-output example (e.g., a vague/incomplete category like 'if it seems risky') alongside a corrected version to reinforce the pitfalls section concretely.
Best Practices
- Always fill Definition/Structure/Example — a category with only a definition is incomplete.
- Make Logic Gates boolean-explicit; avoid vague conditions like "if it seems risky."
- Keep Hooks named consistently (
pre_*,post_*,on_*) across the whole document for grep-ability. - Treat Workflows (human-oriented) and Pipelines (machine-oriented) as distinct — don't merge them.
- Version Deliverables/Payloads schemas explicitly (
v1,v2) so downstream consumers can detect breaking changes. - End with a System Map section tracing one request through Actions → Tasks → Pipeline → Logic Gates → Deliverable.
Common Pitfalls
- Don't write categories as abstract theory — every section needs a real, working example from the target system.
- Don't conflate Mechanics (execution rules) with Logic (decision rules) — keep them separate sections.
- Don't skip Methods & Benchmarking — an orchestration doc without success metrics is unverifiable.
- Don't leave Personalization as "TBD" — define at least a default fallback preference profile.
- Don't produce a document over ~500 lines for one system; split into
architecture.md,pipelines.md,prompts.mdif it grows larger, and link them from a top-levelskill.mdindex.