topic

Topic: Skills

emerging (7 sources, but only one of them is about skills - S5 "Don't Ship Skills

What this covers#

Agent skills: what a skill is and how it loads, how it is triggered, how to write one that fires when it should and not when it should not, how to evaluate one, and when to delete it. Also where skills sit against tools, scripts and MCP.

Synthesis#

A skill is a cost ladder, not a document#

A skill is a folder with a SKILL.md plus assets, loaded by progressive disclosure in three layers with three different prices [S5 &t=159s, &t=176s, slide frame_500]:

That ladder is the whole design constraint. It is also a measured instance of context-engineering.md's claim 22 (limiting context beats filling it) at a much finer granularity than any prior source in this brain [S5 &t=471s, &t=489s].

The reliability bar is set by your user's distance from the mechanism#

The framing that makes the rest follow [S5 &t=126s, slide frame_110]:

• Agents we use (Claude Code, Cursor, Codex) keep an engineer in the loop. A skill that fails to trigger is noticed in seconds and repaired by reprompting or a slash command. The human is the eval.
• Agents we build ship to users who do not know skills exist and will never type "use the refund skill". No fallback; the user leaves on first failure.

The further the user is from the skill system, the higher the reliability bar - and the more the checking has to be automated [S5 &t=126s].

This generalises past skills, and is the reason "it works for me in my editor" is not evidence about a product: the author is the most forgiving possible user, silently repairing failures without counting them.

Two kinds of skill, opposite lifespans#

[S5 &t=194s, &t=213s, slide frame_200]

The clean case for a capability skill is a knowledge gap the training cut-off created: the Gemini Interactions API shipped after training, and a skill with 117 test cases took valid-code generation from 39.2% to 91.6% on Gemini 3.1 Pro [S5 &t=767s, &t=805s, slide frame_800]. Vendor measuring its own product - treat the shape as instructive and the magnitude as unreplicated.

S26: a skill that maintains a corpus, and the first outside instance of the three-layer shape#

Every skill this note has seen until now helps a model do a task. S26 shows a different job: enrich-note is a maintenance operation that runs over a knowledge base, adding tags, source attribution and backlinks to notes, invoked by a human or by a nightly scheduler [S26 n4, visuals/frame_404.jpg]. It is worth recording as a shape rather than an anecdote, because two of its properties are things this note has only ever seen asserted by the source that invented the format.

The three-layer cost ladder appears in the wild, built by someone with no stake in it. The skill is .agents/skills/enrich-note/SKILL.md with frontmatter carrying a name and a trigger description, a body of instructions, and references/tags.md as a separately-loaded reference file the body tells the agent to read first [n6]. That is claim 6's three layers exactly - frontmatter, body, references - instantiated by a practitioner at a different company for a non-coding task. It does not corroborate the prices (nothing here measures tokens), and it is a genuine independent instance of the structure, which is more than this note previously had.

And the reference file is doing something the ladder's framing does not anticipate. In S5 the third layer is described as reference material that is free until read - documentation, examples, schemas. Here it is a mutable registry the skill writes back to: the agent must read tags.md before tagging, must prefer existing entries, and must append any new tag with a one-line definition so the next invocation can reuse it [n6]. The third layer is not a static appendix, it is the skill's memory between invocations, and it exists because each call sees one note and a taxonomy is a corpus-wide object. Any per-item skill that must stay globally consistent needs something in that slot.

The other property worth naming is an idempotence contract in the skill body itself - "if the frontmatter already has enrichedAt, the note is done, skip it", with the stamp written on completion [n5]. That is what makes the skill safe to run repeatedly over a whole corpus and therefore safe to put on a timer. Unmeasured, like everything in S26 (n16), and structurally clear.

They measurably work, and badly-made ones measurably hurt#

From SkillsBench 1.1, across open and closed models and multiple harnesses [S5 &t=266s, slides frame_265, frame_310]:

• Curated skills: 33.9% -> 50.5% task resolution (+16.6 points).
• Self-generated (AI-written) skills: -8.1 to -11.5 points. Human-written skills perform best [S5 &t=299s, &t=315s].
• Length is an inverted-U, not a slope: <200 lines +19.0%; 200-500 lines +21.5% (peak); 500-1000 +14.5%; >1000 lines +0.7%, statistically a no-op.

The length curve appears only on the slide; the speaker says only "keep it below 500 lines". "As short as possible" is the wrong reading - the peak is 200-500, and shorter than 200 is slightly worse.

Writing one: the description is the whole ball game#

The description is the trigger mechanism, and the trigger causes 50%+ of all skill failures [S5 &t=1036s, slide frame_425]. Rewriting the description alone fixed 5 of 7 failures in their suite [S5 slide frame_425, visual-only].

• Directives, not passive information. "Use the Interactions API if you are working on a chat application", not "the Interactions API is recommended for multi-chat because it handles session state" [S5 &t=437s, &t=454s].
• Include the what (capability) and the when (trigger context) [S5 &t=420s].
• Declare negative cases. A broad description ("any web development task") hijacks the trigger across unrelated work. Specify what must not fire it [S5 &t=594s, &t=611s].
• Kill no-ops - instructions that do not alter behaviour ("write clear, high-quality code") are common in AI-authored skills and burn reasoning tokens for nothing [S5 &t=680s, &t=697s].
• Set the right level of freedom: dictating every step strips the agent's ability to adapt or recover. Give goals and constraints [S5 &t=558s, &t=577s, slide frame_560].

And the boundary case: if the workflow is fully determined, write a script, not a skill [S5 &t=558s]. This converges with agents.md's claim 17 from a different discipline - determinism is cheaper than inference, so spend inference only where the path is genuinely unknown.

Evaluating one#

• Start with 10-20 real prompts: 5 happy-path, 5 negative or near-miss, 5 drawn from production traces. Real traces beat synthetic guesses [S5 &t=628s, &t=645s].
• Grade outcomes, not paths. Do not assert the skill loaded on turn one; assert the task succeeded. Loading on turn five is a pass [S5 &t=1091s, &t=1109s].
• Isolate every run. Coding agents cheat: they read prior chats and executions to obtain the skill's content without invoking the skill [S5 &t=1109s, &t=1129s, slide frame_950].
• Multiple trials per case (they use up to six) - the system is non-deterministic [S5 &t=1146s].
• Test across harnesses. A skill good on Gemini CLI can be bad on Codex [S5 &t=1163s].
• The harness is small: a JSON/YAML file of cases (prompt, language, should_trigger, expected checks) plus a script that runs the agent and asserts. Most asserts can be regex; LLM-as-judge is for trace-level checks [S5 &t=825s, &t=878s, &t=914s].
• Gate diffs in CI: at Google DeepMind evals sit alongside every skill, run on every change, and a skill change cannot merge unless it improves the test cases [S5 &t=1002s, &t=1019s, slide frame_950].

Retirement: the idea worth keeping#

Ablation is the retirement test - run the eval with and without the skill loaded [S5 &t=713s, &t=1268s, slide frame_720]:

Keep the eval after you retire the skill. It becomes a regression detector on the bare model, and tells you when to reintroduce the skill [S5 &t=1181s, &t=1199s].

This instruments agents.md's claim 31 (every harness component encodes an expiring assumption about what the model cannot do). A skill is a harness component. S4 named the expiry and offered only "remove one component at a time"; S5 supplies the measurement - and adds what S4 does not have: keep the meter after you remove the part.

The topic's first security material: a skill is a write channel#

This note has never held anything on security, and S19 supplies it by treating skill synthesis as a path into persistent memory (S19 n2, claim 158).

S7 gave this topic its category name - a skill is procedural memory. S19 works out what follows when that store is adversarial. Its channel C4, experience-to-procedure write, fires when an agent decides a completed interaction constitutes a reusable skill and synthesises it into procedural memory. The trigger is the shape of the execution trace - a novel workflow, an error recovery, a successful completion - and the write authority is the agent's own judgement that this was worth keeping. No human wrote the skill and no instruction requested it.

Two vulnerabilities attach to that channel and neither has an equivalent for factual memory. V-S4, no validation for skill creation: the content is committed to procedural memory with no inspection before the skill file is written. And V-S5, self-improvement as amplification, which is the one worth carrying (claim 162):

A poisoned skill is not static. Each execution produces an observation, the loop treats every step that ran without error as validated, and later revisions are built around the existing procedure including any adversarially introduced step. Over time "the skill evolves into a well optimized adversarial procedure."

Read that against claim 31 and the two point in opposite directions. Claim 31 frames scaffolding as an expiring bet, with ablation as the test for whether it has expired - a skill earns its place by measurably helping, and you delete it when it stops. Ablation tests whether a skill still helps. It does not test what the skill does when it is wrong on purpose, and V-S5 describes a skill that gets better at helping while carrying a hostile step, which is precisely the case an evals-based retirement policy would keep.

The uncomfortable pairing is with claim 26. This note holds that AI-written skills are a negative intervention (-8 to -11 points, SkillsBench), measured on quality. S19 supplies a second, unrelated reason to be wary of a skill nobody wrote: there is no inspection step on the path, and the self-improvement loop that refines it treats absence of error as evidence of correctness. Neither source knows about the other, and they converge on "be suspicious of skills the system wrote itself" from quality and from security respectively.

Full synthesis in agent-security.md. Gated needs-check: V-S5 is a mechanism argument with no measurement - S19's benchmark measures skill-procedure insertion (58.33% attack success on HERMES), not amplification across successive refinements.

Key claims#

Key visuals#

The reliability gap: an engineer repairs a mis-trigger in seconds; a customer just leaves. The further the user is from the mechanism, the more the checking must be automated. S5 &t=110s.

The inverted-U the narration never states: peak at 200-500 lines, and a >1000-line skill is statistically a no-op. Also: self-generated skills cost 8-11 points. S5 &t=310s.

Ablation as the retirement test: 94% vs 32% means keep; 96% vs 95% means the base model absorbed the knowledge. S5 &t=720s.

Open questions / conflicts#

• Is SkillsBench methodologically sound? Every strong number in this note rests on it and none has been checked against the benchmark's own documentation. The highest-value deep-research target on this topic.
• Does the 200-500 line sweet spot generalise, or is it an artifact of the models and harnesses SkillsBench tested? A context-window-dependent optimum would be expected to move.
• Why do AI-generated skills hurt? No-ops are offered as the explanation [S5 &t=680s] but nothing measures whether removing them recovers the lost 8-11 points.
• Is "50%+ of failures are trigger failures" a property of skills or of these skills? A team with already-disciplined descriptions would presumably see a different split.
• Everything here is coding agents. S5 says harnesses differ [S5 &t=1163s]; nothing says whether the findings survive outside code.
• ~~Nothing yet on skills as an attack surface.~~ Closed 2026-08-05 by S19, and the answer is worse than the question anticipated. The worry was that a skill is instructions injected into context on a trigger the model chooses. S19 adds that skill synthesis is itself a memory write channel (C4) with no content inspection before the file is written (V-S4), and that a self-improving loop optimises a poisoned skill rather than merely carrying it (V-S5, claim 162). See the security section above and agent-security.md. What remains open is narrower: nobody has measured V-S5 - S19's benchmark measures skill insertion, never amplification across refinements - and no source relates skills to prompt injection through the trigger itself, which was the original worry and is still unaddressed.
• Unconnected to MCP. No source in this brain yet relates skills to MCP servers or tools. See mcp.md - emerging since S10, but scoped entirely to tool cost and retrieval, so it says nothing about skills either.

Sources feeding this topic#

• S26 - LLM Knowledge Bases: a practical guide (Ben Holmes, Warp, 2026-08-15). The first skill in this note that maintains a corpus rather than performing a task, and the first instance of the three-layer shape observed outside the vendor that documented it. Contributes the idempotence contract (n5) and the mutable reference registry (n6), both of which exist because the skill runs repeatedly and unattended over many items. Not about skills either - it is a knowledge-base talk, and skills are how its maintenance happens to be packaged - so it corroborates none of S5's claims and nothing in it is measured (n16). Its main event is in rag.md.
• S5 - Don't Ship Skills Without Evals (Philipp Schmid, Google DeepMind, AI Engineer World's Fair 2026). T4 conference talk by a T2 vendor employee. The SkillsBench figures are third-party and are the strongest evidence; the DeepMind-internal figures are self-reported and unreplicated.
• S7 - Memory and dreaming for self learning agents (Anthropic, 2026-05-21). Contributes one thing, and it is a naming: S7's memory-evolution ladder places skills as the procedural memory rung - CLAUDE.md -> memory tool -> skills (procedural memory) -> memory/ (claim 64). A skill is procedural memory: memory of how to do things rather than of facts. That puts skills and memory in one family rather than two adjacent topics, and supplies the category name this note had been describing without. See memory.md.
• S9 - Inside the Microsoft Agent Framework (2026-05-28). Contributes one box in one diagram, and it is worth recording only because of whose diagram it is. S9's harness inventory files Skills under Context, as a peer of Prompts and Memory [S9 fig_AgentHarness] - a different vendor, with no stake in S7's argument and no reference to it, independently placing skills in the same drawer as memory rather than treating them as documentation or as a tool. Weak evidence of the scarce kind: independent. It supports claim 64's family assignment and says nothing about anything else in this note - S9 never defines a skill, never discusses writing or evaluating one, and does not mention skills in its prose at all. This does not move the topic off emerging; a box position is not a second source on the claims that matter here (ADR-0012).
• S11 - How we built LangChain's agent-first data stack (Emily Hawkins, LangChain, 2026-07-27). Contributes one sentence and one instance, and both are the same kind of weak-but-independent evidence as S9. Describing the prose documents that carry business rules with no schema slot - company processes, reporting conventions, which filters apply to customer health - the author writes: "These would be like skills for the data agent" [S11 §Capturing business context, n5]. She is not citing S5, S7 or anyone else; she reaches for the word because it fits. The artifact matches this note's description exactly: markdown, versioned in a GitHub repo, reviewed on change, synced into the agent's context, steering how to do something rather than stating a fact. This is the first instance in this brain of the pattern outside a coding or agent-tooling context - the domain is go-to-market reporting - which is mild evidence that claim 64's family is real rather than an artifact of who writes about agents. It moves nothing. S11 does not define, size, trigger, evaluate or retire a skill, and it ships its guides with no evals at all - which makes it a live counter-example to S5's central claim rather than a corroboration of it (see evals.md, claim 100). Recorded under ADR-0012: a mention is not a source.
• S13 - karpathy/autoresearch (Andrej Karpathy, code, snapshot 228791f, 2026-03-26). Contributes one sentence and one artifact, and the artifact is the more interesting half. The sentence: "The program.md file is essentially a super lightweight 'skill'" [S13 README.md:50, n16]. The artifact: 115 lines of markdown holding an entire research organisation - a setup ritual, the rules of what may and may not be changed, an output contract, a five-column ledger schema, a nine-step LOOP FOREVER, and an autonomy policy. It matches this note's description of a skill exactly (markdown, versioned beside the code, steering how to do something rather than stating a fact) and stretches it in one direction nothing else here does: it is not a capability the agent invokes when relevant, it is the agent's entire operating procedure for a session. S5's cost-ladder and description-as-trigger analysis simply does not apply to an artifact that is loaded once, deliberately, by a human saying "have a look at program.md". The second instance in this brain of the pattern outside a coding-assistant context (after S11's business-context documents), and the first where the author reaches for the word while building something that is not a skill product. It moves nothing. S13 does not define, size, trigger, evaluate or retire a skill, and it ships program.md with no evals - a second live counter-example beside S11. Recorded under ADR-0012: a mention is not a source. The trap named in the status line now has a fifth data point - the count rose 1 -> 5 while the evidence did not move at all. Full synthesis in autonomous-research-loops.md.