Creating and Improving Skills
Creates new Claude Code skills from scratch, iteratively improves existing skills, and measures skill performance through testing and benchmarking. Use when users want to create a skill from a description of expertise, audit or optimize an existing skill, test a skill against prompts, benchmark skill performance with variance analysis, or optimize a skill's trigger description for better activation accuracy.
Skill Creator
Creates, tests, and improves Claude Code skill files through an iterative, evidence-driven workflow. Treats skill creation as an engineering loop: draft → test → measure → refine, not a one-shot writing exercise.
To create a new skill:
- Clarify the skill's purpose, trigger conditions, and ideal workflow (ask only if genuinely ambiguous; otherwise infer from context).
- Write a draft skill file (YAML frontmatter + structured body, see Skill File Format below).
- Write 3-5 test prompts that should trigger the skill, plus 1-2 "near-miss" prompts that shouldn't.
- Run the test prompts against
claudewith the skill installed. - Evaluate outputs qualitatively (does it follow the methodology?) and quantitatively (does it hit measurable criteria?).
- Revise based on gaps found. Repeat until stable.
To improve an existing skill, skip to Workflow step 2.
YAML--- name: gerund-form-name description: Third-person description of what it does. Use when [specific triggers]. ---
Body sections, in order: Quick Start, Workflow (with checklist for multi-step processes), Examples (concrete input/output pairs), Best Practices, Common Pitfalls. Target ~500 lines; split into reference files if longer.
Progress:
- Step 1: Define scope — what the skill does, when it triggers, what "good output" looks like
- Step 2: Draft or load the skill file
- Step 3: Build a test set (positive + negative trigger prompts, task-execution prompts)
- Step 4: Run tests, collect outputs
- Step 5: Evaluate qualitatively and quantitatively
- Step 6: Diagnose failure mode (triggering vs. execution vs. output quality)
- Step 7: Revise targeted section of skill file
- Step 8: Re-run affected tests, confirm improvement, check for regressions
Step 1: Define Scope
Ask (internally or of the user): What task does this skill perform? What phrases should trigger it? What does a correct/excellent output look like vs. a mediocre one? If the user hasn't specified, infer reasonable defaults and state them.
Step 2: Draft/Load
For new skills, write a full draft following the Skill File Format. For existing skills, read the current file in full before editing.
Step 3: Build a Test Set
Create prompts across three categories:
- Trigger-positive: prompts that should activate the skill (vary phrasing, directness, context).
- Trigger-negative: adjacent-but-different prompts that should NOT activate it (tests description specificity).
- Execution: prompts that activate the skill and exercise its actual workflow, used to judge output quality.
Aim for 5-10 prompts total for a quick pass; 15-20 for a formal benchmark.
Step 4: Run Tests
Run each prompt against Claude Code with the skill installed (and, for comparison, without it). Capture full transcripts, not just final answers — note whether the skill file was actually read/invoked.
Step 5: Evaluate
Qualitative: Does the output follow the skill's workflow steps? Does it match the tone/format specified? Would a domain expert approve?
Quantitative: Define metrics up front, e.g.:
- Trigger accuracy = (correct activations) / (total trigger-positive + trigger-negative prompts)
- Task success rate = (outputs meeting defined success criteria) / (execution prompts)
- Variance = run each execution prompt 3+ times; measure consistency of structure/quality (not just wording). High variance signals an underspecified workflow or missing defaults.
If no metrics exist yet, draft them now based on Step 1's definition of "good output" — don't wait for the user to supply them.
Step 6: Diagnose
Classify each failure:
- Triggering failure: skill didn't activate when it should have, or activated when it shouldn't have → fix
description. - Execution failure: skill activated but Claude didn't follow the workflow → fix structure/specificity of Workflow section, add explicit steps or checklist.
- Output quality failure: workflow followed but result is weak → fix Examples/Best Practices, add concrete defaults or constraints.
Step 7: Revise
Edit only the diagnosed section first. Avoid rewriting the whole file on one failure — isolate changes so you can attribute improvement correctly.
Step 8: Re-test
Re-run the specific failing prompts plus a sample of previously-passing prompts (regression check). Confirm the metric moved in the right direction before declaring done.
Example 1 — New skill creation
Input: "Create a skill for someone who does financial due diligence on acquisition targets."
Output: A skill file conducting-financial-due-diligence with Quick Start showing a sample checklist (revenue quality, liabilities, working capital), Workflow with document-request → analysis → red-flag-report steps, Examples showing a red-flag writeup, Best Practices (e.g. "always reconcile EBITDA adjustments to source ledger"), Common Pitfalls (e.g. "don't accept management adjustments without support").
Example 2 — Triggering optimization
Input: "My 'reviewing-prs' skill isn't firing when people say 'can you look over this diff'."
Output: Diagnosis: trigger-negative test showed description only mentioned "pull request" and "code review," missing "diff" and "look over." Fix: broaden description to "...Use when users ask to review a pull request, check a diff, look over code changes, or give feedback before merging." Re-test confirms activation.
Example 3 — Variance benchmarking Input: "Benchmark my 'writing-sql-queries' skill for consistency." Output: Run the same 5 execution prompts 3x each (15 runs). Report: 13/15 used CTEs consistently, 2/15 used nested subqueries instead (inconsistent style) → flagged as variance issue → add explicit "Best Practices: prefer CTEs over nested subqueries" line → re-run shows 15/15 consistent.
- Always test negative triggers, not just positive ones — overly broad descriptions cause unwanted activation on unrelated tasks.
- Change one thing at a time between test runs so you can attribute cause and effect.
- Prefer concrete defaults ("use CTEs," "output JSON with these keys") over open-ended guidance — reduces variance.
- Keep a running log of test prompts and results so improvements are reproducible, not anecdotal.
- When variance is high across repeated runs, the fix is almost always more specificity in Workflow/Examples, not a longer description.
- Validate the YAML frontmatter (name format, description length) before testing content.
- Don't declare a skill "done" after one successful test run — single runs hide variance.
- Don't fix a triggering problem by editing the Workflow section, or an execution problem by editing the description — match the fix to the diagnosed failure type.
- Don't let the description balloon with every edge case; keep it tight and rely on Examples/Workflow for nuance.
- Don't skip the trigger-negative tests — a skill that fires on everything is as broken as one that never fires.
- Don't rewrite the entire skill file in response to one bad test prompt — isolate and target the fix.