Brain

Dex Horthy reports interviewing 100+ people building agents and found that the ones that work in production are barely agentic - ordinary deterministic software with small, tightly-scoped LLM steps inside (n1). From that he distilled 12 factors, named after Heroku's 12-factor app. The through-line is that an agent is a prompt, a switch statement, a context-window builder, and a loop, and reliability problems trace back to letting a framework own one of those four instead of you (n5). https://www.youtube.com/watch?v=8kMaTybvDUw

What that first sentence is worth. The 100+ interviews are the entire empirical basis of the talk and you cannot check them - no names, no method, no counts (n1). Take the factors as a well-argued pattern language from someone who has clearly built this, not as a survey result. Full accounting in "The evidence, weighed".

flowchart TB
    A["an agent is four things"]
    P["a <b>prompt</b>"]
    S["a <b>switch statement</b>"]
    C["a <b>context-window builder</b>"]
    L["a <b>loop</b>"]
    F["hand any one of the four<br/>to a framework..."]
    R["...and your reliability problems<br/>trace back to exactly that one - n5"]
    W["which is why the agents that work in<br/>production are <b>barely agentic</b>:<br/>ordinary deterministic software with<br/>small, tightly-scoped LLM steps inside - n1"]

    A --> P --> F
    A --> S --> F
    A --> C --> F
    A --> L --> F
    F --> R --> W

    style F fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    style W fill:#dcfce7,stroke:#15803d,color:#14532d

This is an ownership diagram, not an architecture diagram, and the question it answers is which parts you must not delegate. The crux is that the four components are the complete list of things an agent is, so every framework convenience is a decision to stop owning one of them. It is drawn with all four converging on a single hand-off node because the failure mode does not depend on which one you give away; it depends only on having given one away. The green terminal is the talk's headline finding and reads as a paradox until the diagram above it is accepted: barely agentic is what you get when you keep all four.

Synthesized from n1 and n5.

Uber's Computer Vision team runs a multimodal agent that enhances low-quality food photos for Uber Eats. The transferable lesson is not about food - it is a blueprint for evaluating an agent pipeline in production. Log every trace first, because nothing else is possible without it. Then stop asking "is the agent good?" and instead evaluate each stage with the metric that fits its job: a router is a classifier judged on recall, a generator is judged on pass@k, an editor is judged by comparison against its own input. Stack the gates so their holes do not line up, then close the loop - sample production traffic, re-label it, and let the system rewrite its own configs as the world drifts. https://www.youtube.com/watch?v=31GUkCBD-Uc

flowchart TB
    Q["<b>'is the agent good?'</b><br/><i>unanswerable, and it hides<br/>where the failure was</i>"]
    D["decompose the pipeline,<br/>then judge each stage by<br/>the metric that fits its job"]
    R["a <b>router</b> is a classifier<br/>-> recall"]
    G["a <b>generator</b> has no single<br/>right answer -> pass@k"]
    E["an <b>editor</b> has a free reference,<br/>its own input -> pairwise"]
    L["and then close the loop: sample live<br/>traffic, re-label it, let the system<br/>rewrite its own configs as the world drifts"]

    Q -.->|"the question to stop asking"| D
    D --> R --> L
    D --> G --> L
    D --> E --> L

    style Q fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    style L fill:#dcfce7,stroke:#15803d,color:#14532d

This is a measurement diagram, not an architecture diagram, and the dashed edge is the move the whole talk turns on. The crux is that each stage fails in a different way, so a single quality score for the pipeline is not a coarse measurement but a meaningless one. It is drawn fanning out and reconverging because the three metrics are not alternatives to choose between: a production pipeline runs all three simultaneously, and the loop at the bottom only works once every stage emits something comparable over time. Notice the food is incidental. What transfers is the mapping from stage type to metric type, which holds for any routed pipeline of small agents.

Synthesized from n3, n5, n7 and n9.

A harness is the scaffolding you put around a model to get work out of it that the model cannot sustain alone. This article builds one out of a planner, a generator and an evaluator, and then reports the honest numbers. The harness cost 22x more than a solo agent and took 18x longer, and it produced a working app where the solo agent produced a broken one (n15, n16). Then it does what almost no vendor write-up does. On a newer model the author deletes half his own scaffolding and reports that result too (n18). The durable idea is the reason for that deletion, which is that every harness component encodes an assumption about what the model cannot do, and those assumptions expire (n17).

flowchart TB
    G["the gap between the task and what<br/>the model can sustain on its own"]
    C["each harness component exists to close<br/>one specific part of that gap"]
    W["it works, and it costs 18x the wall clock<br/>and 22x the money - n15"]
    S["the model gets stronger<br/>and the gap narrows"]
    D["the component becomes pure overhead<br/><b>without anything about it getting worse</b>"]
    R["so on a better model the author<br/>deletes half his own scaffolding - n18"]

    G --> C --> W
    S --> D --> R
    C -.->|"the assumption it encodes"| D

    style W fill:#fff4e5,stroke:#b45309,color:#78350f
    style D fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    style R fill:#dcfce7,stroke:#15803d,color:#14532d

This is an obsolescence diagram, not an architecture diagram, and the dashed edge is the whole argument. The crux is that a harness component's worth is a property of the gap it was built to close rather than of the component itself, so good scaffolding expires without ever becoming bad scaffolding. It is drawn as two chains meeting rather than as a build-then-teardown sequence because the two things are not stages in a project: the assumption is encoded on the day the component is written, and the model release that voids it arrives independently and on somebody else's schedule. The green terminal box is what makes this source unusual, since almost no vendor write-up publishes the deletion of its own machinery. Synthesized from n15, n17, n18 and n19.

OAuth 2.0 solves exactly one problem, which is letting an app act on your behalf without giving it your password. Everything else in it is machinery in service of that, including the jargon, the four flows and the two-step token dance. The shape of that machinery is dictated by a single security fact, namely that the browser can be trusted to talk to a human, but not to hold a secret. OpenID Connect is a thin layer bolted on top, because the industry started using OAuth for login, which it was never built for, and OAuth has no way to say who you are. Learn the authorization code flow and you have learned the protocol &t=1323s.

flowchart TB
    F["<b>the security fact</b><br/>the browser can be trusted to talk to<br/>a human, but not to hold a secret"]
    C["so the protocol splits into two channels:<br/>a <b>front</b> channel through the browser<br/>and a <b>back</b> channel between servers"]
    O["one flow - authorization code -<br/>uses both"]
    V["and the other three flows are that flow<br/>with a channel removed"]
    I["OIDC is bolted on because OAuth has<br/>no way to say <i>who you are</i>,<br/>and the industry used it for login anyway"]

    F --> C --> O --> V
    C --> I

    style F fill:#e8f0fc,stroke:#4338ca,color:#312e81
    style V fill:#dcfce7,stroke:#15803d,color:#14532d

This is a derivation diagram, not a protocol diagram, and it draws why the specification has the shape it does rather than what the shape is. The crux is that a single fact about browsers generates the entire structure, including the jargon and the flow count, so there is really only one flow and three degradations of it. It is drawn descending from the security fact rather than starting at the flows because the usual failure with OAuth is meeting four flows as four options and trying to memorise which to pick. Read this way the choice stops being a lookup and becomes a question about which channels you actually have. OIDC hangs off the side deliberately: it is not part of the derivation, it is a patch for a use the protocol was never built for.

Synthesized from n1, n6 and n8.

Skills are everywhere and almost never tested. SkillsBench indexed 47,000+ across 6,300 repos, and almost none of them carry evals (n1). The talk's argument is that this is a measurement problem rather than laziness, because non-determinism makes a skill's contribution unattributable without a control, so "it worked for me" is not evidence. From there the talk is unusually concrete. A skill is a three-layer cost ladder rather than a document, and its description is the trigger, which causes 50%+ of all failures. Length follows an inverted U peaking at 200-500 lines, and AI-written skills are a negative intervention. You retire a skill by ablation, and then you keep the eval afterwards as a regression detector on the bare model. https://www.youtube.com/watch?v=0vphxNt4wyk

flowchart TB
    R["you ran the task with the skill<br/>loaded, and it failed"]
    Q{"was the skill bad,<br/>or the task too hard?"}
    N["non-determinism means a second run<br/>tells you little more than the first - n2"]
    T["and no amount of looking harder at one<br/>trace separates them, because the<br/>information is not in the trace"]
    C["so what is missing is a <b>control</b>,<br/>not a closer look"]
    A["run the same task <b>without</b> the skill.<br/>The difference is the skill."]
    B["and the same procedure answers<br/>'is this any good?' and<br/>'should I delete it?' - n20"]

    R --> Q --> N --> T --> C --> A --> B

    style C fill:#e8f0fc,stroke:#4285f4,color:#1a3a6b
    style B fill:#dcfce7,stroke:#15803d,color:#14532d

This is an attribution diagram, not a workflow, and the question in the second box is the one the whole talk exists to answer. The crux is that a skill's value is a difference rather than a level, so a single run cannot measure it no matter how carefully you read the trace. It is drawn as an unbranching descent because each step closes off the response an engineer would naturally reach for next, ending at the only move that works. The final box is why this reframing earns its keep: treating a skill as an intervention rather than a document collapses two questions people usually answer with different methods into one procedure. Synthesized from n2, n9 and n20.

Memory written during a conversation is written in that conversation's tense. Nothing revisits it, so it decays into confident wrongness rather than into uselessness (n1). The fix OpenAI ships is architectural rather than a better extractor. It moves the write off the turn into a background synthesis pass, it stores a maintained narrative in place of an append-only fact list, and it revises stale entries instead of expiring them (n3, n4, n5). Correction is then offered on the synthesized summary rather than on the raw records (n6). Its own evals put staleness at a 9.4% baseline, which is a system wrong nine times in ten, and that number is what makes the diagnosis more than rhetoric (n13).

flowchart TB
    W["memory is written <b>during</b> a conversation,<br/>in that conversation's tense"]
    N["and nothing ever revisits it"]
    D["so it decays into <b>confident wrongness</b><br/>rather than into uselessness - n1"]
    F["which is a property of <b>when you write</b>,<br/>not of <b>what you store</b>"]
    A["so the fix is a second clock, not a<br/>better extractor - n3, n4, n5"]

    W --> N --> D --> F --> A

    style D fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    style A fill:#e8f0fc,stroke:#4285f4,color:#1a3a6b

This is a diagnosis diagram, not an architecture diagram, and the fourth box is the note's whole contribution. The crux is that staleness is a timing failure rather than a storage failure, which is why every response aimed at capturing better information leaves it exactly where it was. It is drawn as one unbranching descent because the diagnosis admits no alternative reading once stated, and the design in Movement B is downstream of it rather than beside it. Notice that the failure is confident wrongness rather than irrelevance: a stale memory does not stop being used, it keeps being used and is now wrong, which is the property that makes this worth engineering against.

Synthesized from n1, n3, n4 and n5.

This is the agent-platform half of the memory pair, and the independent counterpart to S6. Same architecture, same name, different vendor. Agents write memory during work, and a decoupled batch pass rewrites it between sessions (n11, n14). The reason to split the loops is objective conflict, not throughput, because one loop asked to both finish the task and curate memory trades them off untunably (n12). Memory is deliberately a file system the model drives with bash and grep, on the same bet that produced skills (n2, n3). To that it adds everything a consumer assistant never needs, namely scoped stores, optimistic concurrency via a content_sha256 precondition, and per-session attribution (n5-n7). A live demo shows agents leaving instructions for their successors, not just facts (n20).

flowchart TB
    O["one loop asked to finish the task<br/><b>and</b> curate memory"]
    C["trades them off <b>untunably</b> - n12"]
    S["so split the loops:<br/>agents write <b>during</b> work,<br/>a batch pass rewrites <b>between</b> sessions - n11, n14"]
    R["the reason is <b>objective conflict</b>,<br/>not throughput"]
    G["and that diagnosis generalises<br/>far past memory"]

    O --> C --> S
    C --> R --> G

    style C fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    style R fill:#e8f0fc,stroke:#4285f4,color:#1a3a6b
    style G fill:#dcfce7,stroke:#15803d,color:#14532d

This is a diagnosis diagram, not an architecture diagram, and the blue node is the reason to read the note rather than the mechanism it describes. The crux is that the second loop exists to separate two objectives that cannot be weighted against each other, which is a different argument from the usual one about batch work being cheaper. It is drawn with the conflict feeding both the fix and the generalisation because the fix alone would be an implementation detail, and the generalisation is what travels: any single loop carrying two objectives will trade one against the other silently, and no amount of prompt tuning surfaces the exchange rate.

Synthesized from n11, n12 and n14.

A knowledge base built on retrieval re-derives its synthesis on every question and keeps none of it; the alternative is to compile the knowledge once into a maintained markdown wiki that an LLM owns, and pay the synthesis cost at ingest time instead of at query time (n1, n2). Three layers - immutable raw sources, an LLM-written wiki, and a schema document both parties co-evolve - plus three operations: ingest, query, lint (n4, n6). The pattern is Vannevar Bush's Memex (1945), which was blocked for eighty years on one thing: who does the maintenance (n15). https://gist.github.com/karpathy (revision ac46de1)

flowchart TB
    Q{"which cost are you paying,<br/>and how often?"}
    L["<b>lookup</b> - finding the documents<br/><i>better chunking, better embeddings,<br/>a reranker on top</i>"]
    S["<b>synthesis</b> - relating them<br/>to each other<br/><i>happens after the right documents<br/>are already in hand</i>"]
    R["retrieval re-pays it on every<br/>single question, then discards it - n1"]
    C["compile it once at ingest,<br/>and keep it current - n2"]
    W["and the whole rest of the design<br/>falls out of that one move"]

    Q --> L
    Q --> S --> R
    S --> C --> W

    classDef aimed fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    classDef right fill:#dcfce7,stroke:#15803d,color:#14532d
    class L,R aimed
    class C,W right

This is a diagnosis diagram, not an architecture diagram, and the left branch is what the industry spends its money on. The crux is that this is not the usual complaint about retrieval, because it holds even when retrieval is perfect: what is being re-paid is the relating of documents rather than the finding of them. It is drawn as a fork on a cost question rather than as a pipeline because the two branches are not stages you pass through, and the failure of the naive approach is not that it works badly but that it is aimed at the cheaper of the two costs. Everything the note describes after this, the three layers and the three operations, is a consequence of moving the right-hand cost from query time to ingest time. Synthesized from n1 and n2.

A vendor design post worth reading for one factoring, not for its product: the agent loop, workflows, and the harness are three separable concerns, and the summary figure shows they are not a stack - the loop is the only mandatory layer (n1, n9). It supplies the brain's only enumerated harness inventory (n7), five named orchestration patterns including Author/Critic (n6), and the sentence that justifies the whole layering: "a strong model with poor tools, weak context and no controls will still produce a poor result" (n8). It also contradicts claim 12 head on - S2 says own your loop, this says let the SDK own it - and never acknowledges the position exists (d1).

flowchart TB
    L["<b>agent loop</b><br/>the only mandatory layer - n1"]
    W["<b>workflows</b><br/>for when you do not want autonomy"]
    H["<b>harness</b><br/>tools, context, planning, middleware"]
    N["<b>not a stack</b> - you may take the loop<br/>without either of the others - n9"]
    C["and it contradicts claim 12 head on,<br/>without ever acknowledging<br/>the position exists - d1"]

    L --> N
    W --> N
    H --> N
    N --> C

    style N fill:#dcfce7,stroke:#15803d,color:#14532d
    style C fill:#fbf1dc,stroke:#b45309,color:#78350f

This is a factoring diagram, not a product diagram, and the negative claim in the green box is the whole reason to read the post. The crux is that three things routinely sold as one are separable, and the summary figure says so by not stacking them - the loop is mandatory and the other two are optional in either order. It is drawn with all three feeding the separability claim rather than layered, because drawing them as a stack would reproduce exactly the error the note exists to correct. The amber box is the note's own finding rather than the source's: S2 says own your loop and this says let the SDK own it, and the post never acknowledges that anybody disagrees.

Synthesized from n1, n9 and divergence d1.

A tool catalog stops being a schema-management problem and becomes a search problem, and the crossover is around ten to fifteen tools. Microsoft Foundry's Toolbox replaces the full tools/list manifest with exactly two meta-tools - tool_search(query, limit) and call_tool(name, arguments) - and keeps the rest of the catalog indexed but never listed (n3, n6). On ToolRet (44,000+ tools) that cuts context from 541k tokens to 15k at 1,180 tools, a 36x reduction, and the figure shows something the prose undersells: the tool-search curve is roughly flat as the catalog grows 24x (n9, n10). The reframing is the real payload - tool names and descriptions become ranking features, so the first tuning pass is editorial, not algorithmic (n19, n13).

flowchart TB
    C["a tool catalog as a<br/><b>schema-management</b> problem"]
    X["crossover at roughly<br/>ten to fifteen tools"]
    S["a tool catalog as a<br/><b>search</b> problem"]
    T["two meta-tools replace the manifest:<br/>tool_search and call_tool - n3, n6"]
    R["541k tokens -> 15k at 1,180 tools,<br/>and the curve is roughly <b>flat</b><br/>as the catalog grows 24x - n9, n10"]
    E["so names and descriptions become<br/><b>ranking features</b>, and the first<br/>tuning pass is <b>editorial</b> - n19, n13"]

    C --> X --> S --> T --> R --> E

    style S fill:#dcfce7,stroke:#15803d,color:#14532d
    style E fill:#e8f0fc,stroke:#4338ca,color:#312e81

This is a reframing diagram, not an architecture diagram, and the last box is the payload rather than the token saving. The crux is that once retrieval sits between the agent and its tools, tool names and descriptions stop being documentation and become ranking features - which relocates the first tuning pass from engineering to editing. It is drawn as one descent with the crossover marked because the reframe is conditional: below ten or fifteen tools a manifest is simply better, and the whole argument only starts above that line. Notice that the flatness of the curve says more than the 36x headline, since flatness is a claim about how this scales rather than about one measurement.

Synthesized from n3, n6, n9, n10, n13 and n19.

An "agent-first data stack" turns out not to be a data stack at all. It is a documentation layer wrapped around an unchanged one. The figure captioned "LangChain's data stack architecture" shows a stock ELT pipeline - Fivetran and Airbyte and Segment into BigQuery, dbt on top, reporting at the end - with no agent, no semantic model and no feedback loop anywhere in it (d1). Everything that makes the stack "agent-first" lives in a second figure, and all of it is prose.

The transferable move is that a column definition stops being a description and becomes an instruction. account_status: The status of the account. becomes a paragraph that spells out each lifecycle value in business terms and then issues an imperative: "For customer reporting, filter to Active unless the analysis explicitly includes churned or prospective accounts" (n3). That is not documentation. That is a default policy stored in metadata, where the agent meets it at exactly the moment it matters.

The loop that maintains it inverts what a data team is for. Observability over agent conversations shows where context is missing. The team then writes the missing context. So the loop's output is never an answer to a user but a write back into the store (n7, n8).

Read the results with both hands. Every number here measures adoption - 2,200 conversations, 40x throughput, 100% migration in six weeks - while the article's thesis is about trustworthiness, which it never measures (d4). The authors concede it and file evals under "next" (n10). Deep research supplies both halves the source lacks. Schema documentation is measured to help, and far more on real warehouses (+16pp) than on public benchmarks (+2pp). The enterprise text-to-SQL setting closest to this stack tops out around 65.6%. The mechanism is well corroborated. The result is not.

flowchart TB
    C["The article's claim:<br/>'a big architectural shift'"]

    subgraph SHOWN["What the architecture figure actually shows - d1"]
        direction TB
        P1["Fivetran, Airbyte, Segment"]
        P2["BigQuery, dbt, reporting"]
        P3["no agent, no semantic model,<br/>no feedback loop"]
        P1 ~~~ P2 ~~~ P3
    end

    subgraph REAL["Where the agent-first-ness actually lives - all of it prose"]
        direction TB
        R1["column definitions rewritten<br/>as instructions - n3"]
        R2["five context stores, one per<br/>question asked - n2"]
        R3["a loop whose output is a write<br/>to the store, never an answer - n8"]
        R1 ~~~ R2 ~~~ R3
    end

    C --> SHOWN
    C --> REAL
    SHOWN --> V["the plumbing was never the problem"]
    REAL --> W["the deliverable is English,<br/>and it costs three permanent people"]

    style SHOWN fill:#fdeaea,stroke:#dc3545,color:#7f1d1d
    style REAL fill:#e8f4ea,stroke:#28a745,color:#14532d

This is a claim-versus-artifact diagram, not an architecture diagram, and the two columns are the same system described by two different parts of the same article. The crux is that an "agent-first data stack" turns out to be a documentation layer wrapped around an entirely unchanged stack, and the strongest evidence for that is the article's own figure, which is captioned as the architectural shift and contains none of the things that make the stack agent-first. The columns are drawn as siblings under one claim rather than as before-and-after because they are simultaneous: both are true descriptions of the same company on the same day. What follows from the shape is the cost line on the right, since a deliverable made of prose is bounded by how fast people can write it rather than by anything you can provision. Synthesized from n2, n3, n8 and divergence d1.

An organisation that wants agents for twelve business units has two bad options - twelve teams building twelve stacks, or one shared agent that must be trusted to keep twelve datasets apart - and this document is Google's answer to picking neither. The answer is that the tenancy boundary should be the cloud platform's own coarsest one, a project per business unit, with the agent's authority bounded by policy on the principal rather than by care in the code. That single decision turns out to buy three things at once: cross-tenant access becomes structurally impossible, one unit's incident stays inside that unit, and one unit's traffic spike cannot starve another. The document's real lesson is in its second half, where every cost-saving alternative it offers is the same trade in different clothing - give a piece of isolation back and re-enforce it in software you now have to write - and the sharpest instance is a cost recommendation that quietly deletes the tenant-local PII filter the security section calls essential. Vendor reference architecture, T2, no measurement of any kind: read it as a well-argued shape, never as a result.

flowchart TB
    F{"Twelve business units<br/>need agents"}
    O1["twelve teams build twelve stacks<br/><i>silos, duplicated ops, governance gaps</i>"]
    O2["one shared agent keeps twelve<br/>datasets apart by being careful<br/><i>one prompt away from failing</i>"]
    A["Neither. Make the tenant boundary the<br/>platform's own coarsest one:<br/>a project per business unit, with the agent's<br/>authority capped on the <b>principal</b>"]
    P1["cross-tenant access becomes<br/>structurally impossible"]
    P2["one unit's incident stays<br/>inside that unit"]
    P3["one unit's spike cannot<br/>starve another"]
    C["and every cheaper variant the document offers<br/>gives one of these back, to be re-enforced<br/>in software you now have to write - n14"]

    F --> O1
    F --> O2
    F --> A
    A --> P1 --> C
    A --> P2 --> C
    A --> P3 --> C

    classDef bad fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    classDef good fill:#dcfce7,stroke:#15803d,color:#14532d
    class O1,O2 bad
    class A,P1,P2,P3 good
    class C bad

This is a decision diagram, not an architecture diagram, and it draws the note's thesis rather than the system. The crux is that one tenancy decision buys three separate guarantees at once, and the whole second half of the document consists of ways to sell them back individually. It is drawn as a rejected fork feeding a single answer that fans into three payoffs and then reconverges on one warning, because the reconvergence is the argument: a reader who takes the architecture and skips the alternatives has read the easy half. The closing red box is the sharpest thing in the source and the reason it is worth reading at all, since the cost section quietly deletes a control the security section calls essential. Synthesized from n2, n4, n13, n14 and divergence d2.

karpathy/autoresearch gives a coding agent one editable Python file, five minutes of GPU time per experiment, one protected metric, and an instruction never to stop; overnight it runs about a hundred experiments and keeps the ones that improve the number. The interesting object is not the language-model training code - it is the containment design, which is ten files, no framework, and no agent code at all. What the repo actually teaches is which four things you must freeze before an agent can be trusted to change everything else: the editable surface, the resource budget, the metric's units, and the holdout (n1-n4). It also teaches, unusually honestly, where that design leaks: the protected score is printed by the file the agent rewrites (n5), and the accept rule is a bare comparison with no notion of run-to-run variance - which is why the fifteenth and final "improvement" in the author's own published run is a change of random seed (n11). Read it as a worked example of building an unattended optimizer, and read the results chart as a warning about what such a loop will confidently bank.

flowchart TB
    subgraph FR["Frozen before the loop starts, and it holds"]
        direction TB
        A["the editable surface<br/>one file, train.py"]
        B["the budget<br/>wall-clock seconds, not steps or tokens"]
        C["the metric's units<br/>bits per byte, at a fixed sequence length"]
        D["the holdout<br/>pinned inside the read-only file"]
        A ~~~ B ~~~ C ~~~ D
    end

    subgraph OP["Never frozen, and both failures are here"]
        direction TB
        E["who prints the score<br/>the file the agent rewrites - n5"]
        G["what counts as an improvement<br/>a bare comparison, no variance - n11"]
        E ~~~ G
    end

    FR --> R["~100 experiments overnight,<br/>ten files, no framework, no agent code"]
    OP --> R
    R --> S["15 kept improvements, and the last<br/>one is a change of random seed"]

    style FR fill:#e8f4ea,stroke:#28a745,color:#14532d
    style OP fill:#fdeaea,stroke:#dc3545,color:#7f1d1d
    style S fill:#fdeaea,stroke:#dc3545,color:#7f1d1d

This is a containment diagram, not an architecture diagram, and it sorts the repository by one question: was this decided before the agent started running? The crux is that every property this design gets right is something frozen in advance, and both places it fails are places where nothing was frozen at all, which is why the failures are not bugs and cannot be patched without adding a fifth freeze. The two columns are drawn as peers rather than as a design and its caveats because they are the same kind of object, and the closing box is the author's own published result rather than a criticism of it: a loop with no variance model banked a random seed as an improvement, exactly as the right-hand column predicts. Synthesized from n1-n5 and n11.

Every agent in this brain retrieves something before it acts, whether that is a memory of a past session or a document from a knowledge base. AgentPoison shows that the retrieval step is an attack surface in its own right, and that attacking it is cheaper than attacking anything else in the system. The method optimises a short trigger phrase so that any query containing it lands in a private, tightly clustered corner of the retriever's embedding space, where the attacker has already placed a handful of malicious records (n3). No weights are touched and no training is run (n2). The costs that matter are the ones that are almost zero: a single poisoned record and a one-token trigger are enough to reach roughly 62% and 79% retrieval success respectively, while benign accuracy stays above 90% (n5). The trigger it produces reads like ordinary text, so a human reviewing the store would not flag it (n8). Read this as the threat model that three vendor memory sources in this brain designed without.

flowchart TB
    R["every agent retrieves something<br/>before it acts"]
    S["so the <b>retriever</b> is an attack surface,<br/>and the cheapest one in the system"]
    M["optimise a short trigger so any query<br/>containing it lands in a private corner<br/>of the embedding space - n3"]
    N["<b>no weights touched, no training run</b> - n2"]
    C["one poisoned record -> ~62%<br/>one trigger token -> ~79%<br/>benign accuracy stays above 90% - n5"]
    H["and the trigger reads like ordinary text,<br/>so a human reviewing the store<br/>would not flag it - n8"]

    R --> S --> M --> C
    M --> N
    C --> H

    style S fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    style H fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d

This is a cost diagram, not an attack diagram, and every number in it is chosen because it is small. The crux is that this attack is cheap in exactly the dimensions defenders monitor - record count, trigger length, accuracy impact and human readability - which is what makes it a threat model rather than a demonstration. It is drawn with the costs and the stealth on the same branch because they are one property: an attack that needs a single innocuous-looking record is invisible to volume anomaly detection and to review alike. Read this as the threat model that three vendor memory sources in this brain designed without.

Synthesized from n2, n3, n5 and n8.

S17 established that an LLM reading untrusted content cannot distinguish data from instructions, and that no filter fixes it. CaMeL's answer is to stop trying to make the model safe and instead build a system in which an unsafe model cannot cause an unsafe action (n1). A Privileged LLM sees only the trusted user query and writes a program; a Quarantined LLM parses untrusted data, holds no tools, and can return only schema-conforming values (n3, n4). A custom interpreter runs that program, tags every value with its provenance and permitted readers, and checks a Python security policy at every tool call (n5, n6, n7). Untrusted data therefore never reaches control flow, and the actions it can influence are bounded by policy rather than by the model's judgement. It solves 77% of AgentDojo tasks against an undefended 84%, cutting successful attacks from roughly 100 to 0-1, for 2.82x input tokens (n9, n10, n12). Two things to hold onto: the benchmark is the authors' own (d1), and the paper spends its last five pages showing where its own design breaks (n15, n16, n18).

flowchart TB
    A["<b>the abandoned goal</b><br/>make the model able to tell<br/>data from instructions"]
    B["<b>the replacement goal</b><br/>build a system in which an unsafe model<br/>cannot cause an unsafe action - n1"]
    C["untrusted data never reaches<br/>control flow"]
    D["and the actions it can influence are<br/>bounded by <b>policy</b> rather than<br/>by the model's judgement"]
    E["77% of tasks against an undefended 84%,<br/>attacks from ~100 down to 0-1,<br/>for 2.82x input tokens - n9, n10, n12"]

    A -.->|"give up on this"| B --> C --> D --> E

    style A fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    style B fill:#dcfce7,stroke:#15803d,color:#14532d

This is a goal-substitution diagram, not an architecture diagram, and the dashed edge is the entire contribution. The crux is that CaMeL concedes the problem S17 established rather than attacking it, and then makes the concession harmless by moving the security boundary out of the model entirely. It is drawn as one abandoned goal feeding a replacement because the design only makes sense once you accept that the first box is unachievable; a reader still hoping for a better classifier will read the architecture as overkill. The bottom box is deliberately a single node holding both the benefit and the price, since quoting either without the other misrepresents the trade.

Synthesized from n1, n9, n10 and n12.

Stanford's CS329A opens by arguing that a model can be made to improve itself, and the mechanism is plainer than the name suggests. Sample the model many times instead of once, keep the answers that survive a check, and feed those answers back as training data (n5). The interesting part is not that this works but where it stops working. Every turn of the loop needs something that can tell a good answer from a bad one, and that something is scarce outside math, code and other rule-based domains, which is why the lecture calls verification the field's bottleneck (n6). Read this as a map of a research area rather than as a result, because the source is lecture 1 of a course and its headline chart measures something weaker than its title claims (d1). The most useful thing it leaves you with is a question to ask of any self-improving system: what checks the output, and who wrote the checker?

flowchart TB
    S["sample the model many times<br/>instead of once"]
    K["keep the answers that<br/><b>survive a check</b>"]
    F["feed those back as<br/>training data - n5"]
    L["the loop turns"]
    Q{"but every turn needs something that can<br/>tell a good answer from a bad one"}
    V["and that is scarce outside math, code<br/>and other rule-based domains - n6"]
    A["so the question to ask of any<br/>self-improving system is:<br/><b>what checks the output,<br/>and who wrote the checker?</b>"]

    S --> K --> F --> L --> Q --> V --> A

    style Q fill:#e8f0fc,stroke:#4285f4,color:#1a3a6b
    style A fill:#dcfce7,stroke:#15803d,color:#14532d

This is a bottleneck diagram, not a method diagram, and the interesting part is where the chain stops rather than how it turns. The crux is that self-improvement is mechanically simple and gated entirely on verification, so the loop's reach is set by the domain rather than by the model. It is drawn as a cycle that runs into a question because the mechanism genuinely does work and the constraint genuinely does bind; drawing either alone would misrepresent the lecture. The terminal box is what to carry away, and it is a question rather than a finding because this is lecture 1 of a course and a map of a research area rather than a result.

Synthesized from n5 and n6.

You can buy accuracy at inference time instead of at training time, by sampling a model many times instead of once, and the returns are lawful enough to budget against - coverage follows c = exp(a·k^b) in the number of samples k, across models from 70M to 70B parameters (n3). The lecture then spends its best twenty minutes demolishing the naive reading of that result. Sampling gets you a set containing a right answer; it does not get you the right answer, and every practical way of picking one out plateaus after roughly ten to fifty samples while the set keeps improving (n10). That distance is the generation-verification gap, and it decides where this whole technique is worth anything. The most useful idea here is the reframe at the end: stop trying to select the best candidate and start synthesizing one from all of them, which beats even a perfect oracle selector (n25).

flowchart TB
    S["sample the model k times<br/>instead of once"]
    C["<b>coverage</b> climbs lawfully:<br/>c = exp(a·k^b), from 70M<br/>to 70B parameters - n3"]
    G["but coverage is a property of the <b>set</b>.<br/>You still have to pick one answer."]
    P["and every practical selector plateaus<br/>after roughly 10-50 samples while<br/>the set keeps improving - n10"]
    D["<b>the generation-verification gap</b>,<br/>which decides where any of<br/>this is worth paying for"]
    F["so stop <b>selecting</b> a candidate<br/>and start <b>synthesizing</b> one -<br/>which beats a perfect oracle - n25"]

    S --> C --> G --> P --> D --> F

    style C fill:#dcfce7,stroke:#15803d,color:#14532d
    style P fill:#e8f0fc,stroke:#4285f4,color:#1a3a6b
    style F fill:#dcfce7,stroke:#15803d,color:#14532d

This is a limits diagram, not a technique diagram, and the middle of the chain is where the lecture spends its best twenty minutes. The crux is that sampling buys a set containing a right answer and never buys the right answer, so the headline scaling law and the practical ceiling are measuring two different things. It is drawn as one descent because the argument is a single walk from an attractive result to the constraint that governs it and then out the other side; branching would suggest the gap is one consideration among several rather than the thing that decides whether the technique pays. The last box is the reframe worth taking away, and it is the only move here that gets past the plateau rather than optimising within it.

Synthesized from n3, n10 and n25.

The moment an LLM reads content it did not author, the content and the instructions arrive through the same channel, and the model has no mechanism for telling them apart. This paper names the consequence: processing retrieved data is analogous to executing arbitrary code (n1). An attacker therefore does not need an account, a session, or any interface to your system. They need only to place text somewhere your agent is likely to read - a web page, an email, a package's documentation - and the retrieval step does the rest (n2). The authors demonstrate six threat classes on real deployed products, including Bing Chat on GPT-4 and GitHub Copilot (n3, n4), and the demonstrations that should worry an engineer most are the ones borrowed from classical malware: a prompt that forwards itself to your contacts (n5), one that writes itself into the agent's long-term memory and re-poisons a later session (n6), and one that fetches fresh instructions from the attacker's server on every request (n7). Read it for the taxonomy and the framing, and read d1 first: this paper proves feasibility on named systems and reports no success rate for anything.

flowchart TB
    R["the model reads content<br/>it did not author"]
    C["content and instructions arrive<br/>through the <b>same channel</b>"]
    E["<b>processing retrieved data is analogous<br/>to executing arbitrary code</b> - n1"]
    A["so the attacker needs no account,<br/>no session and no interface"]
    P["only text somewhere your agent<br/>is likely to read - n2"]

    R --> C --> E --> A --> P

    style E fill:#f8b4b4,stroke:#c1121f,color:#7f1d1d
    style P fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d

This is a consequence diagram, not an attack tree, and the whole paper is the middle box unpacked. The crux is that retrieval is not a data operation but an execution one, which relocates the security question from who can talk to your system to what your system is willing to read. It is drawn as a single descent because nothing branches: each step follows from the one above with no design choice available, which is why the paper's framing survived even as the specific products it tested changed. The terminal box is the practical consequence and the reason the threat model in section 2 pointed the wrong way for years.

Synthesized from n1 and n2.

Four sources into this brain's security material, every efficacy number has come from whoever was making the claim. AgentDojo is the field's reference benchmark and the first thing here built to settle such arguments rather than to win one. It runs a user task and an attacker task in the same stateful tool-calling environment and scores them separately - 97 user tasks, 629 security test cases, 70 tools, four applications (n1, n2). Its most transferable design decision is that utility is checked by deterministic functions rather than an LLM judge, for an adversarial reason: an attack strong enough to hijack the agent might also hijack the evaluator (n3). Three findings matter. Agents fail more than a third of these tasks with no attacker present at all (n4); more capable models are easier to attack, because weak models fail at the attacker's goal too (n6); and attack success is a property of the application, not the model - 92% on Slack, 0% on some Travel tasks (n8). On defences, the simplest isolation mechanism wins: a tool filter drops attack success to 7.5%, and its failure mode is stated exactly - it breaks when the task's own tools suffice for the attack, 17% of cases (n12, n13). Read d1 first: this shares two authors with S18, so it cannot validate CaMeL.

flowchart TB
    P["four sources in, and every efficacy<br/>number came from whoever<br/>was making the claim"]
    B["a benchmark built to <b>settle</b> such<br/>arguments rather than to win one"]
    D["utility checked by <b>deterministic functions</b>,<br/>never an LLM judge - because an attack<br/>strong enough to hijack the agent<br/>might hijack the evaluator too - n3"]
    F1["agents fail a third of these tasks<br/>with <b>no attacker present</b> - n4"]
    F2["more capable models are <b>easier</b><br/>to attack - n6"]
    F3["attack success is a property of the<br/><b>application</b>, not the model:<br/>92% on Slack, 0% on some Travel - n8"]

    P --> B --> D
    B --> F1
    B --> F2
    B --> F3

    style D fill:#dcfce7,stroke:#15803d,color:#14532d

This is a provenance diagram, not a benchmark description, and the top box is why this source matters more than its numbers do. The crux is that the design decision and the findings both follow from taking adversarial conditions seriously: the evaluator is deterministic because a judge is attackable, and the three findings are all things a vendor benchmarking its own product would have no reason to surface. It is drawn with the green node separated from the findings because that decision is the transferable part - it generalises to any adversarial evaluation, whether or not you ever run this benchmark. Read d1 first: this shares two authors with S18, so it cannot validate CaMeL.

Synthesized from n1, n3, n4, n6 and n8.

Schmidhuber's Gödel machine rewrites its own code only when it can prove the change beneficial, which is why nobody has built one. The DGM's move is to swap the proof for empirical evidence on a benchmark (n1), and then to handle everything that follows from being able to be wrong. It improves its own codebase - self-improvement is the coding task it is measured on (n2) - and keeps every agent it has ever produced in an archive, selecting parents by score and by how little explored they are (n3, n4). Over 80 iterations that takes SWE-bench from 20.0% to 50.0% and Polyglot from 14.2% to 30.7% (n7), and two ablations show both components are load-bearing: freeze the meta-agent and progress plateaus, keep only the latest agent and it plateaus lower (n8). What it discovers is unglamorous - better file editing, patch ranking, retry on empty patches (n10) - and it transfers to held-out models, benchmarks and languages (n11). The section to read for your build is §5: the builders state that if your benchmark does not capture every property you care about, the loop amplifies whatever it does not measure (n12). That is S19's V-S5, said by the people running the loop.

flowchart TB
    G["Schmidhuber's machine rewrites its own code<br/>only when it can <b>prove</b> the change good"]
    W["which is why nobody built one"]
    S["swap the proof for <b>empirical evidence</b><br/>on a benchmark - n1"]
    C["and then handle everything that follows<br/>from being able to be <b>wrong</b>"]
    A["an <b>archive</b> of every agent ever produced,<br/>parents chosen by score <i>and</i> by how<br/>little explored they are - n3, n4"]
    R["20.0% -> 50.0% on SWE-bench<br/>over 80 iterations - n7"]

    G --> W --> S --> C --> A --> R

    style S fill:#dcfce7,stroke:#15803d,color:#14532d
    style C fill:#e8f0fc,stroke:#4285f4,color:#1a3a6b

This is a substitution diagram, not an architecture diagram, and the box after the swap is where the real work is. The crux is that replacing proof with evidence is one line of design and produces a system that can be wrong, so every subsequent choice - the archive, the exploration term, the frozen components - exists to survive being wrong rather than to improve anything. It is drawn as a single descent because the paper reads that way: one substitution, then its consequences, then the numbers. The result is unglamorous by design, and section 9 carries the builders' own warning about what the loop does to whatever the benchmark fails to measure.

Synthesized from n1, n3, n4 and n7.

Prompt injection needs its payload present every time it fires. Memory poisoning needs one successful write. This paper is the first systematic account of how that write happens: four channels through which content reaches an agent's long-term memory, of which three are decided by the model's own judgement rather than by any command (n2), and nine structural vulnerabilities that make them exploitable (n3). The finding that matters is about detectability. A prompt injection carries an explicit override, so its intent is usually recoverable from raw text; a memory-poisoning payload can be stored because it looks like a valid fact, policy or past experience (n1). Against that, four production prompt-injection detectors give incomplete coverage, none achieves both high recall and low false positives, and retraining on memory-poisoning data made the strongest one slightly worse (n10, n11). The result a builder should not skip is V-S5: in an agent with autonomous skill refinement, a poisoned skill is not static - the loop treats every error-free execution as validation and optimises the adversarial procedure over time (n4). Read d1 first: the benchmark hands the payload to the agent rather than routing it through a real tool call.

flowchart TB
    PI["<b>prompt injection</b><br/>payload must be present<br/>every time it fires"]
    MP["<b>memory poisoning</b><br/>one successful write - n1"]
    D["and the payload is stored because<br/>it <i>looks like</i> a valid fact,<br/>policy or past experience"]
    E["so there is no explicit override<br/>to recover from the text"]
    F["four production detectors give incomplete<br/>coverage, and retraining on memory-poisoning<br/>data made the strongest one <b>worse</b> - n10, n11"]

    PI -.->|"the shift"| MP --> D --> E --> F

    style MP fill:#f8b4b4,stroke:#c1121f,color:#7f1d1d
    style F fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d

This is a threat-model diagram, not an architecture diagram, and the dashed edge is where the whole paper lives. The crux is that the property making memory poisoning cheap is the same property that makes it undetectable: a payload accepted because it resembles legitimate content leaves nothing for a text classifier to find. It is drawn as one shift cascading rather than as a comparison table because the detection result at the bottom is not a separate finding, it is a consequence of the definition at the top. The red terminal is the part a builder should not read past: retraining the best available defence on exactly this attack class made it slightly worse.

Synthesized from n1, n2, n10 and n11.

This is the defence most teams actually ship, and it costs almost nothing - S18 measures it at 1.06x input tokens against CaMeL's 2.82x. Spotlighting transforms untrusted input so its provenance is continuously visible to the model, then tells the model about the transformation (n2). Three variants, and the ordering matters: delimiting halves attack success and the authors recommend against it because an adversary who learns your system prompt writes their own delimiters (n4); datamarking - interleaving a marker token throughout the text - drops attack success from ~50% to 3.1%, and costs nothing measurable on the underlying task (n5, n6); encoding gives the best number and only works on high-capacity models, wrecking accuracy on weaker ones (n7). Two things make this worth reading beyond the recipe. The adversary section is genuinely good - assume the system prompt has leaked, therefore randomise the marker (n9) - and the discussion contains the best cross-domain framing in this brain's security set (n12). Read d3 first: every experiment is document summarization or Q&A, with no agent and no tools.

flowchart TB
    P["make the <b>provenance</b> of untrusted input<br/>continuously visible to the model,<br/>then tell the model about it - n2"]
    D["<b>delimiting</b><br/>halves attack success -<br/>and the authors recommend <b>against</b> it - n4"]
    M["<b>datamarking</b><br/>~50% -> <b>3.1%</b>, and costs nothing<br/>measurable on the task - n5, n6"]
    E["<b>encoding</b><br/>best number, and wrecks accuracy<br/>on weaker models - n7"]
    C["1.06x input tokens,<br/>against CaMeL's 2.82x"]

    P --> D
    P --> M
    P --> E
    M --> C

    style D fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    style M fill:#dcfce7,stroke:#15803d,color:#14532d

This is a pricing diagram, not a technique diagram, and the ordering of the three variants is the content. The crux is that the cheapest defence in this brain has a variant its own authors tell you not to use and a variant that is nearly free, and telling them apart is the entire value of reading the paper rather than the abstract. It is drawn with all three hanging off one idea because they are the same transformation at different strengths, and with the cost attached only to datamarking because that is the one worth deploying. Read d3 first: every experiment is document summarization or Q&A, with no agent and no tools.

Synthesized from n2, n4, n5, n6 and n7.

The MCP 2026-07-28 specification deletes the initialize handshake and the Mcp-Session-Id header, so any server instance can serve any request and a plain round-robin load balancer becomes sufficient (n3). The change is real and the mechanism is clean, and you can watch it happen field by field by diffing the article's two payloads. What the article calls statelessness is better understood as state relocation, because the same state turns up in three new homes: on the wire in a _meta block sent with every request, in the client as a serialized requestState blob, and in your own application as a task store the article's headline says you no longer need (n10, d2). Each relocation has a bill, and the article prices none of them. The one that matters most is the second, because decoding the article's own example shows server execution state travelling through the client as unsigned plaintext while it guards a file deletion (n8, d1).

flowchart TB
    B["before: one state owner<br/>server memory, keyed by Mcp-Session-Id"]
    H["delete the handshake - n3"]
    W["<b>to the wire</b><br/>a _meta block on every<br/>request, forever"]
    C["<b>to the client</b><br/>requestState, unsigned plaintext<br/>in the article's own example - n8"]
    A["<b>to your application</b><br/>a task store, which is Redis, four sections<br/>after the headline says you do not<br/>need Redis - n9, d2"]
    S["State is conserved.<br/>Three new owners, three bills,<br/>and the article prices none of them - n10"]

    B --> H
    H --> W --> S
    H --> C --> S
    H --> A --> S

    classDef cost fill:#fce8e6,stroke:#ea4335,color:#7f1d1d
    class C,A,S cost
    style H fill:#e8f0fe,stroke:#4285f4,color:#1a3a6b

This is a conservation diagram, not an architecture diagram, and the claim it makes is arithmetic rather than architectural. The crux is that what the article calls statelessness is state relocation, so the correct question is never whether the state is gone but who is holding it now and what they pay. It is drawn as one owner fanning into three because the announcement presents a deletion and the deletion is real, while the three destinations appear scattered across the article with nothing adding them up. The middle branch is the one to act on: server execution state travelling through the client as unsigned plaintext, while it guards a file deletion. Synthesized from n3, n8, n9, n10 and divergences d1 and d2; the conservation framing is this brain's.

Every agent framework teaches you the loop, and almost none of them teach you what happens to a message on the way in and on the way back out. This article walks one boring task through a real open-source agent and finds that the interesting engineering is entirely in the boundaries: routing identity is not conversation identity, session state is not prompt context, a tool schema is not an authorization, and "the agent succeeded" is not one fact but eight of them in a row. The single most useful thing in it is a cell in a table rather than a sentence in the prose, and it says that the guarantee stopping two turns from mutating one conversation is held in memory - so it is process-local, and it does not survive a restart. Read it as the operational half of everything this brain already holds about agent loops.

flowchart TB
    MSG["inbound message"] --> SESS["session"]
    SESS --> CTX["context"]
    CTX --> LOOP["model and tool loop"]
    LOOP --> PERS["persistence"]
    PERS --> DEL["delivery"]

    SESS -.-> C1["routing identity is not<br/>conversation identity"]
    CTX -.-> C2["session state is not<br/>prompt context"]
    LOOP -.-> C3["a tool schema is not<br/>an authorization"]
    LOOP -.-> C4["transcript order is not<br/>side-effect order"]
    PERS -.-> C5["session identity is not<br/>the execution workspace"]
    DEL -.-> C6["committed is not<br/>delivered"]

    classDef collapse fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    class C1,C2,C3,C4,C5,C6 collapse

Read the spine downward as the article's own one-line summary of what a message becomes, and each red node as a distinction the note exists to defend. The crux is that every failure worth naming here comes from writing two objects as one. It is drawn against the spine rather than as a list of pairs because each collapse bites at a particular stage, and knowing where it bites is what tells you which component owns the fix - a routing bug is not repaired by changing the model, and a delivery problem must not be allowed to run a destructive tool twice. A tidier shape, six pairs in a table, would let a reader memorise the distinctions without ever learning where they apply, which is the failure this whole note is arguing against. Spine quoted from the source; the collapses are synthesized from n1, n7, n10, n14, n16 and n17.

Seven benchmarks now measure whether a model can find a security flaw and turn it into a working attack, and this article walks through all of them. The useful thing it gives you is not the leaderboard, it is the anatomy underneath: a sandboxed target, an information dial that sets difficulty, tools, and a grader that scores the outcome because the method is unbounded. What the article does not do is confront its own numbers, and once you put its figures side by side the pattern is hard to miss. Every headline percentage in this field is a description of a configuration rather than a capability, and each dial in that configuration is worth more than the gap between one model and the next. Decomposing the task raised the difficulty ceiling elevenfold. Swapping the scaffolding took one model from 3 of 40 networks to 37. Turning the vendor's safety filters back on took another from 120 exploits to zero. And beneath all of it sits the number the prose never mentions, printed inside one embedded chart: offensive capability, denominated in simulated stolen dollars, is doubling roughly every 1.3 months.

flowchart TB
    N["the number you are about to quote<br/>'17.5% on Cybench'"]

    D1["information given<br/>zero-day to one-day<br/>10% to 12.5% - n9"] --> N
    D2["task decomposition<br/>unguided to subtask-guided<br/>11 min to 2h03 ceiling - n6"] --> N
    D3["scaffolding around the model<br/>ExpertPromptShell to Incalmo<br/>3 of 40 to 37 of 40 - n19"] --> N
    D4["safeguards<br/>enabled to disabled<br/>0 to 120 exploits - n25"] --> N
    D5["attempt budget<br/>single shot to best-of-8<br/>never stated in prose - n8"] --> N

    N --> F["and underneath every dial,<br/>the capability itself doubles<br/>every ~1.3 months - n23"]

    classDef dial fill:#fef3c7,stroke:#b45309,color:#78350f
    classDef num fill:#e0e7ff,stroke:#4338ca,color:#312e81
    classDef floor fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    class D1,D2,D3,D4,D5 dial
    class N num
    class F floor

Read the yellow boxes as the five dials this article turns without ever lining them up, each labelled with the swing it produced in the article's own evidence, all of them feeding the single blue number a reader would otherwise quote out of context. The crux is that the configuration moves the score further than the model does, so a cybersecurity capability figure means nothing until you know all five settings. It is drawn as dials converging on one output rather than as a ranked list because the dials are not alternatives and you do not get to pick one, since every published number has a setting for all five whether or not the author disclosed it. The red floor is separate because it is the one thing no configuration choice affects, and because it is the finding a reader is least likely to leave with, existing as it does only inside an image. Synthesized from n6, n8, n9, n19, n23 and n25; the dials are this brain's framing and are not the article's.

This brain already holds the pattern for LLM-maintained knowledge bases, as S8, and has never seen anyone run one. This talk is that missing half. Ben Holmes read Karpathy's gist, built it, and put the working parts on a projector - the skill file, the tag registry, the generated entity page, the scheduled job - which turns a set of eighteen needs-check assertions into something you can watch operate. Two things make it worth twenty minutes. The mechanisms he had to invent that the pattern does not mention are the ones that decide whether it survives past a month, and there are four of them. And the system quietly breaks the pattern's single most load-bearing rule - raw sources are immutable - which turns out not to be sloppiness but a genuine correction, because the rule as written cannot be implemented and the rule as practised can.

flowchart TB
    P["S8: the pattern<br/>(a gist, 2026-04)"] --> Q{"can it actually<br/>be run?"}
    Q --> R["S26: one instance<br/>(a vault, 2026-07)"]

    R --> N1["idempotence stamp<br/><i>makes sweeps incremental</i>"]
    R --> N2["controlled vocabulary<br/><i>stops taxonomy sprawl</i>"]
    R --> N3["scheduled unattended run<br/><i>removes the human trigger</i>"]
    R --> N4["per-directory schema<br/><i>one job, many wikis</i>"]

    R --> D["<b>and it violates<br/>'raw is immutable'</b>"]
    D --> D2["the rule that survives:<br/><b>one declared writer per layer</b>"]

    R --> Z["<b>but measures nothing</b><br/><i>instantiability, not efficacy</i>"]

    classDef add fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef div fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    classDef warn fill:#fef3c7,stroke:#b45309,color:#78350f
    class N1,N2,N3,N4 add
    class D,D2 div
    class Z warn

Green is what the instance adds, red is where it contradicts its own source, and amber is the ceiling on all of it. The crux is that the interesting content of an implementation is exactly the part the pattern did not specify, which is why the four green nodes matter more than the faithful reproduction above them. It is drawn as one fan-out rather than as the talk's five-stage pipeline because the pipeline is S8's and reproducing it here would say that the value of this source is its agreement, when the value is its residue. The amber node hangs off the instance rather than sitting at the bottom as a caveat, because it qualifies every green node individually and none of them survives being quoted without it. Synthesized from n1, n5, n6, n10, n12, n16 and d1.

GitHub's MCP server is the largest one anybody has published operating data about, at roughly 7.34 million tool calls a week, and the talk is a year of things going wrong at that scale [n17]. Open contribution filled it to 101 tools and made agents measurably worse at using GitHub, so the team built three elegant opt-in fixes - grouped toolsets, dynamic tool discovery, and a semantic tool search prototype - and everyone used the default settings [n4]. The reductions that finally landed were the ones nobody had to opt into, which is a governance lesson wearing a context-window costume. The sharper finding sits underneath it. Having spent the first half of the talk trying to build a per-user filter on the tool surface, the team found one already sitting in the credential: a token's scopes are a free, correct, zero-configuration filter, and turning auth into a context mechanism solved the two problems the rest of the talk was about [n15].

flowchart TB
    P["101 tools arrived by contribution<br/>agents got worse at using GitHub"]

    subgraph U["Fixes that need the user to act"]
        direction TB
        T1["Grouped toolsets"]
        T2["Dynamic tool discovery"]
        T3["Semantic tool search"]
        T1 ~~~ T2 ~~~ T3
    end

    subgraph N["Fixes that need nobody to act"]
        direction TB
        D1["A smaller default<br/>49 pct fewer tools"]
        D2["Tailored responses<br/>77 to 86 pct fewer output tokens"]
        D3["Intent encoded in the tool<br/>instead of an error returned"]
        D4["Scope filtering<br/>the credential already knows"]
        D1 ~~~ D2 ~~~ D3 ~~~ D4
    end

    P --> U
    P --> N
    U --> X["Reached almost nobody"]
    N --> Y["Reached the whole user base"]

    style U fill:#3a2020,stroke:#a04040,color:#fff
    style N fill:#1f3320,stroke:#4a9e5c,color:#fff
    style X fill:#5a1f1f,stroke:#a04040,color:#fff
    style Y fill:#1c4025,stroke:#4a9e5c,color:#fff

This is a delivery diagram, not an architecture diagram, and the axis it sorts on is who has to act for the fix to reach a user. The crux is that the two columns contain roughly equally clever engineering and only one of them shipped value, because the red column's entry cost is a JSON edit and the green column's is nothing. Notice that the fourth green box is the odd one out and is the note's real payload: scope filtering is not a smaller version of a tool list, it is a different source of truth for what the tool list should be. It is drawn green because the user does nothing to get it, which is exactly the property the red column lacked.

Synthesized from n2, n4, n5, n6, n8 and n15.