Creating and Iterating Skills

If the user says "turn this into a skill" after a workflow just happened in conversation, extract the steps, tools, and corrections from history, draft a SKILL.md, confirm with the user, then move straight to testing. If they're starting from scratch, run the intent interview first. If they already have a draft skill, skip straight to running test cases.

Core loop (always keep this in view):

1. Capture intent → 2. Draft SKILL.md → 3. Write test prompts → 4. Run with-skill + baseline in parallel → 5. Review qualitatively (viewer) + quantitatively (benchmarks) → 6. Improve based on feedback → 7. Repeat → 8. Optimize description for triggering → 9. Package

Progress:

• Capture intent (what/when/output format/need test cases?)
• Draft SKILL.md (name, description, body)
• Write 2-3 realistic test prompts, confirm with user
• Spawn with-skill AND baseline runs together, same turn
• Draft assertions while runs are in flight
• Capture timing data as runs complete
• Grade runs, aggregate benchmark, launch eval viewer
• Read user feedback, revise skill
• Repeat iteration loop until satisfied
• Optimize description for triggering accuracy
• Package final skill

1. Capture Intent

Ask (or extract from conversation): what should the skill enable, when should it trigger, what's the expected output, does it need objective test cases (file transforms, data extraction, fixed workflows) or is it subjective (style, taste — often skip formal evals). Confirm understanding before drafting.

2. Draft the SKILL.md

Write the YAML frontmatter first, then the body. The description is the only triggering signal — be deliberately "pushy" about when to use it, since models tend to undertrigger skills. Name specific contexts, even ones the user might not phrase explicitly.

Keep the body under ~500 lines. If a skill spans multiple domains/frameworks, split into references/<variant>.md and point to them from the main file rather than inlining everything.

Explain the why behind instructions rather than stacking ALL-CAPS MUSTs — a model with good theory of mind generalizes better from reasoning than from rigid rules.

3. Test Cases

Write 2-3 realistic prompts (the kind a real user would type, not abstract placeholders). Confirm with the user before running. Save to evals/evals.json:

JSON
{
  "skill_name": "example-skill",
  "evals": [
    {"id": 1, "prompt": "User's task prompt", "expected_output": "Description of expected result", "files": []}
  ]
}

Don't write assertions yet — just prompts.

4. Run With-Skill and Baseline in Parallel

For each test case, spawn two subagents in the same turn: one given the skill, one without (or given the old version, if this is an iteration on an existing skill). Never stagger these — launch all at once so timing comparisons are fair.

Results go in <skill-name>-workspace/iteration-N/eval-<descriptive-name>/{with_skill,without_skill}/outputs/, created incrementally, not pre-scaffolded.

Write eval_metadata.json per test case:

JSON
{"eval_id": 0, "eval_name": "descriptive-name-here", "prompt": "...", "assertions": []}

5. Draft Assertions (while runs are in flight)

Use this idle time productively. Write objectively verifiable, descriptively-named assertions for anything checkable by script. Leave subjective quality judgments (style, taste) to human review instead of forcing brittle assertions onto them.

6. Capture Timing

As each subagent completes, immediately save its total_tokens / duration_ms to timing.json in that run's directory — this data only arrives once, via the completion notification.

7. Grade, Aggregate, Launch Viewer

• Grade each run against assertions → grading.json with fields text, passed, evidence (exact names required by the viewer). Prefer scripts over eyeballing for anything checkable programmatically.
• Aggregate: python -m scripts.aggregate_benchmark <workspace>/iteration-N --skill-name <name>
• Do an analyst pass over the aggregate numbers: flag non-discriminating assertions (pass regardless of skill), high-variance evals (possibly flaky), time/token tradeoffs.
• Launch the browser reviewer (always use generate_review.py, never hand-rolled HTML):

Bash
nohup python eval-viewer/generate_review.py <workspace>/iteration-N \
  --skill-name "my-skill" --benchmark <workspace>/iteration-N/benchmark.json \
  > /dev/null 2>&1 &

Add --previous-workspace <workspace>/iteration-<N-1> from iteration 2 onward. No display available (headless/Cowork)? Use --static <output_path> instead — the user downloads feedback.json manually; copy it into the workspace afterward.

8. Read Feedback and Iterate

Read feedback.json. Empty feedback = fine as-is; focus revisions on entries with specific complaints. Generalize fixes rather than overfitting to the exact failing example — the goal is a skill that works across many future invocations, not just these 2-3 test cases. Watch transcripts (not just outputs) for repeated ad-hoc scripts the subagents wrote independently; bundle those into scripts/ so future runs don't reinvent them.

Rerun the full loop into iteration-N+1/ until the user is satisfied, feedback comes back empty, or progress stalls.

9. Optimize Triggering Description

Once the skill itself is solid, generate 20 realistic trigger-eval queries (8-10 should-trigger with varied phrasing/formality, 8-10 should-not-trigger that are genuinely tricky near-misses — not obviously irrelevant). Let the user review/edit via assets/eval_review.html, exporting eval_set.json. Then run:

Bash
python -m scripts.run_loop --eval-set <path> --skill-path <path> \
  --model <current-session-model-id> --max-iterations 5 --verbose

This splits train/test, iterates on the description up to 5 rounds, and reports best_description chosen by held-out test score. Apply it to the frontmatter and show before/after.

10. Package

If present_files is available: python -m scripts.package_skill <path/to/skill-folder>, then hand the .skill file to the user.

Example 1: Input: "I keep manually reformatting CSVs for our weekly report — can you make this a skill?" Output: Interview reveals input columns, desired output template, and edge cases (missing values, varying delimiters) → draft SKILL.md with a scripts/reformat.py helper → 3 test CSVs as eval prompts → parallel with-skill/baseline runs → viewer shows with-skill output matches template, baseline improvises inconsistently → user approves → package.

Example 2: Input: "My 'pdf-filler' skill isn't triggering when people ask about filling out forms." Output: Skip straight to description optimization — generate 20 trigger queries (including near-misses like "extract data from this form" which should not trigger a filling skill), run run_loop.py, apply the improved description.