Evolving Agent Skills with Persistent Wiki
When asked to improve an agent's skills based on execution history, structure the workspace into three layers rather than editing skills directly:
workspace/
raw/ # immutable execution traces (never edit)
wiki/
patterns/ # one markdown file per recurring failure/success pattern
logs.md # evolution log: what was proposed, accepted, rejected, when
skill-impact.md # programmatic record of validation score deltas per skill change
skills/
<skill-name>/
SKILL.md # frontmatter (name, description) + procedural instructions
PURPOSE.md # links back to the wiki patterns that motivated this skill
Process one iteration as: rollout → consolidate to wiki → propose skill diff → validate → gate/rollback (wiki persists regardless).
Progress:
- Step 1: Run rollouts on training tasks using current skill set (or none, on iteration 0)
- Step 2: Store raw traces immutably; never let downstream steps mutate them
- Step 3: Consolidate traces into wiki patterns (failure modes + successful strategies + workarounds)
- Step 4: Update logs.md with what was learned this iteration
- Step 5: Propose skill updates using wiki patterns + raw traces as evidence
- Step 6: Validate proposed skills against held-out validation tasks
- Step 7: Gate: accept if validation score improves, else rollback the skill change
- Step 8: Record the outcome in skill-impact.md regardless of accept/reject
- Step 9: Wiki always persists — never reset it on rollback
- Step 10: Repeat until convergence or iteration budget exhausted
Step details
Step 1-2 (Inference Agent): Execute the agent on training tasks with the current active skills. Capture full step-by-step traces: reasoning, tool calls, tool outputs, final answers. Treat this as write-once storage.
Step 3 (Wiki Maintainer): Read raw traces and existing wiki content. For each recurring issue or successful strategy, write or update a pattern file describing: the trigger condition, what went wrong/right, and an actionable workaround. Check skill-impact.md and logs.md first — do not re-propose interventions that were already rejected.
Step 5 (Skill Proposer): Given the wiki (patterns + logs + impact history) and raw traces, propose concrete SKILL.md edits. Every new or modified skill should have a corresponding PURPOSE.md entry citing which wiki pattern(s) motivated it.
Step 6-7 (Gating): Run the proposed skill set on validation tasks. Compare aggregate score to the current best. Accept only if it improves (or ties with simplification); otherwise roll back the skill files to the prior version. The wiki update from Step 3 is kept either way.
Example 1:
Input: Traces show an agent repeatedly failing spreadsheet tasks by using wrong cell-range syntax (e.g., A1-B5 instead of A1:B5) in 6/20 training tasks.
Output:
- New wiki pattern file
wiki/patterns/cell-range-syntax.md: "Failure: agent uses-instead of:for ranges. Workaround: always validate range syntax matches[A-Z]+[0-9]+:[A-Z]+[0-9]+before executing." - Proposed skill edit to
skills/spreadsheet-editing/SKILL.mdadding an explicit range-syntax rule and example. PURPOSE.mdupdated: "Motivated by wiki/patterns/cell-range-syntax.md, iteration 4."- Validation: score improves from 61% to 68% → accepted;
skill-impact.mdlogs +7pp for this change.
Example 2:
Input: A proposed skill change to add verbose step-by-step self-verification for math tasks decreases validation accuracy from 74% to 70% (likely due to added context length diluting attention).
Output: Roll back the skill to the prior version. Keep the wiki pattern documenting "self-verification verbosity trade-off: excessive verification steps can hurt accuracy on short-context math tasks" so future proposals don't repeat this exact intervention. Log the rejection with score delta (-4pp) in skill-impact.md and logs.md.
- Never mutate raw traces. They are the ground truth for all downstream reasoning; treat
raw/as append-only. - Always cross-reference the wiki before proposing. Check
logs.mdandskill-impact.mdto avoid re-proposing previously rejected changes. - Keep skills lean; keep wiki rich. Skills should contain concise, actionable procedural instructions; detailed reasoning, edge cases, and history belong in the wiki, loaded only when needed (progressive disclosure).
- Link every skill back to its motivating evidence. Use PURPOSE.md to maintain auditability between wiki patterns and skill content.
- Gate skill changes on a held-out validation split, distinct from training tasks used to generate traces, to avoid overfitting skills to training quirks.
- Persist the wiki unconditionally. Even when a skill change is rolled back, the knowledge that motivated it (and the fact that it failed) should remain recorded.
- Prefer targeted, evidence-backed skill edits over wholesale rewrites — smaller diffs are easier to validate and roll back.
- Consider cross-model skill transfer. Skills evolved by a stronger model can sometimes outperform a weaker model's self-evolved skills; don't assume skills must be evolved by the same model that will use them.
- Don't skip the wiki layer and edit skills directly from raw traces. This scatters insights across iterations and causes repeated mistakes (re-proposing rejected ideas, forgetting past failure modes).
- Don't reset or prune the wiki between iterations. Doing so discards accumulated context and re-introduces previously solved problems.
- Don't conflate skill rollback with wiki rollback. A rejected skill change should still leave a trace in the wiki explaining why it failed.
- Don't let skills grow unboundedly verbose. Bloated SKILL.md files consume context budget; push detailed rationale into the wiki/PURPOSE.md instead.
- Don't validate on the same tasks used to generate training traces. This inflates apparent gains and produces skills that don't generalize.
- Don't assume larger models need less skill support. Skill evolution often benefits larger models more, and evolved skills can let smaller models match or beat much larger unskilled models — don't dismiss skill evolution as unnecessary for capable models.