AI Skill Report Card
Designing AI Architecture
Quick Start14 / 15
Given a request to design an AI system, produce these six artifacts in order:
- AI Architecture — high-level system diagram (components, data flow)
- Agent Topology — agent roles, hierarchy, communication pattern
- Memory Layer — short-term/long-term/episodic memory design
- Reasoning Flow — step-by-step decision/inference pipeline
- Execution Flow — how requests move from input to action to output
- Scaling Plan — how the system grows (more agents, more load, more data)
Every design must state how it satisfies: secure-by-default, auditability, observability, AI governance.
Recommendation▾
Add a second example covering a different topology (e.g., single-agent or batch pipeline) to show range beyond customer support
Workflow14 / 15
Progress:
- Clarify the use case (single-agent vs multi-agent, sync vs async, real-time vs batch)
- Define AI Architecture (models, orchestration layer, data stores, APIs)
- Define Agent Topology (roles, supervisor/worker pattern, handoff rules)
- Define Memory Layer (what's stored, where, retention, retrieval method)
- Define Reasoning Flow (prompt chain, tool-use decisions, fallback logic)
- Define Execution Flow (request → routing → agent(s) → tools → response)
- Define Scaling Plan (horizontal scaling, cost controls, model swapping)
- Verify all four priorities are addressed explicitly
- Present as structured document with clear section headers
Recommendation▾
Include a template/skeleton for the six artifacts that Claude can fill in directly rather than only a worked example
Examples16 / 20
Example 1: Input: "Design a customer support AI that can answer questions, escalate to humans, and remember past conversations."
Output:
Recommendation▾
Add explicit guidance on when to scale down complexity (e.g., single-agent sufficiency) to avoid over-engineering simple use cases
AI Architecture
- Orchestrator (LLM router) + 2 specialized agents (FAQ Agent, Escalation Agent)
- Vector DB (conversation embeddings) + Relational DB (ticket state)
- Model: primary LLM for reasoning, smaller model for classification/routing
Agent Topology
- Supervisor agent routes intent → FAQ Agent or Escalation Agent
- FAQ Agent: read-only tool access (knowledge base search)
- Escalation Agent: write access (create ticket, notify human)
- Handoff protocol: confidence score < 0.7 triggers escalation
Memory Layer
- Short-term: last 10 turns in session cache (Redis, TTL 1hr)
- Long-term: customer history in vector DB, retrieved via semantic search
- Audit log: every memory read/write timestamped and user-attributed
Reasoning Flow
- Classify intent (FAQ vs complaint vs unknown)
- Retrieve relevant memory + KB context
- Generate response with citation to source
- Self-check: confidence score, policy compliance check
- If low confidence → escalate
Execution Flow
Input → Orchestrator → Intent Classification → Agent Selection → Tool Calls (KB search / ticket API) → Response Generation → Safety Filter → Output + Log Entry
Scaling Plan
- Stateless agent workers behind load balancer, autoscale on queue depth
- Cache frequent FAQ answers to reduce LLM calls
- Model tiering: cheap model for classification, strong model only when needed
Governance & Security
- Secure-by-default: all tool calls require scoped API keys, no raw DB access from LLM
- Auditability: every decision logged with input, output, model version, confidence
- Observability: tracing per request (latency, token usage, tool calls) via OpenTelemetry
- AI Governance: human-in-loop for escalations, periodic bias/quality audits on responses
Best Practices
- Always separate orchestration logic from agent logic — makes topology changes cheap.
- Design memory with explicit retention/expiry policy; never store indefinitely by default.
- Every agent-to-agent handoff must be logged with reason and confidence score.
- Prefer smaller/cheaper models for routing/classification; reserve large models for final reasoning.
- Build in a "kill switch" and rate limits at the orchestrator level before scaling agents.
- Make reasoning steps traceable — log intermediate thoughts/tool calls, not just final output.
Common Pitfalls
- Don't let agents share unrestricted memory access — scope memory per agent role.
- Don't skip the observability layer "for now" — retrofitting tracing is expensive.
- Don't design unbounded agent-to-agent loops without a max iteration/timeout guard.
- Don't conflate short-term session memory with long-term user profile memory.
- Don't scale agent count before scaling the orchestrator's routing and monitoring capacity.