source

Tool search: Finding the right tool at the right time

Lisa Brown Jaloza - Command Line (Microsoft)

TL;DR#

A tool catalog stops being a schema-management problem and becomes a search problem, and the crossover is around ten to fifteen tools. Microsoft Foundry's Toolbox replaces the full tools/list manifest with exactly two meta-tools - tool_search(query, limit) and call_tool(name, arguments) - and keeps the rest of the catalog indexed but never listed (n3, n6). On ToolRet (44,000+ tools) that cuts context from 541k tokens to 15k at 1,180 tools, a 36x reduction, and the figure shows something the prose undersells: the tool-search curve is roughly flat as the catalog grows 24x (n9, n10). The reframing is the real payload - tool names and descriptions become ranking features, so the first tuning pass is editorial, not algorithmic (n19, n13).

flowchart TB
    C["a tool catalog as a<br/><b>schema-management</b> problem"]
    X["crossover at roughly<br/>ten to fifteen tools"]
    S["a tool catalog as a<br/><b>search</b> problem"]
    T["two meta-tools replace the manifest:<br/>tool_search and call_tool - n3, n6"]
    R["541k tokens -> 15k at 1,180 tools,<br/>and the curve is roughly <b>flat</b><br/>as the catalog grows 24x - n9, n10"]
    E["so names and descriptions become<br/><b>ranking features</b>, and the first<br/>tuning pass is <b>editorial</b> - n19, n13"]

    C --> X --> S --> T --> R --> E

    style S fill:#dcfce7,stroke:#15803d,color:#14532d
    style E fill:#e8f0fc,stroke:#4338ca,color:#312e81

This is a reframing diagram, not an architecture diagram, and the last box is the payload rather than the token saving. The crux is that once retrieval sits between the agent and its tools, tool names and descriptions stop being documentation and become ranking features - which relocates the first tuning pass from engineering to editing. It is drawn as one descent with the crossover marked because the reframe is conditional: below ten or fifteen tools a manifest is simply better, and the whole argument only starts above that line. Notice that the flatness of the curve says more than the 36x headline, since flatness is a claim about how this scales rather than about one measurement.

Synthesized from n3, n6, n9, n10, n13 and n19.

The 1-minute version#

This article covers a preview feature in Microsoft Foundry that stops handing an agent its tool catalog and starts making the agent search for it. The mechanism is small enough to describe in one sentence, and the interesting object is not the mechanism at all. It is the reframing the authors arrive at on the way, which is that a tool catalog past a certain size is an information-retrieval problem wearing a schema-management costume (n19). To see why that reframing is earned rather than decorative, start with the cost the catalog is already imposing.

That cost is the tool manifest itself. When a client connects, tools/list hands the model every tool definition up front, and those definitions stay resident in the model's context on every turn - names, descriptions, JSON schemas, argument definitions and nested parameters, all of it present before the user has asked for anything (n1). The property worth fixing on is not the size but what the size is a function of. It tracks what is connected, not what the task needs, so a catalog of 1,180 tools costs 541k tokens per turn whether the request touches one of them or none (n9). At which point the obvious response is to attach fewer tools.

The reason that response fails is what makes the problem hard, and it is a distribution argument rather than a budget one. The tools you would drop are the rarely used ones, and rarely used tools are disproportionately the ones that rotate a credential, recover a failed deployment or apply a compliance exception (n16). Dropping them is cheap on every ordinary turn and catastrophic on the one turn that needed them. In other words, the catalog cannot be trimmed to the common case, because the value of the long tail is concentrated exactly in the moments the common case does not cover.

Suppose instead you keep every tool and simply pay less for it. This is the engineer's first reflex and it is already the default here, because prompt caching was enabled in the baseline and cached tokens are roughly 90% cheaper. The article is unusually careful about what that buys. Cached tokens are "not free", and cached context "still competes for the model's attention" (n2). The manifest therefore keeps degrading the model's behaviour at a discount, which is worse than it sounds, because the discount removes the one signal that would have told you the manifest was a problem. You paid less to poison the well.

The idea, then, is to stop listing the catalog and start indexing it. Foundry exposes exactly two meta-tools in tools/list - tool_search(query, limit) for describing the capability you want, and call_tool(name, arguments) for invoking whatever comes back - while everything else in the catalog stays indexed but never listed (n3, n6). The natural objection is that one tool should have been enough, since a search that returns a tool definition ought to let the model call it. The reason there are two is a runtime constraint rather than a design preference. Many runtimes guard against the model calling a tool that never appeared in tools/list, so a retrieved tool is not yet a callable tool, and a registered proxy has to carry the dispatch (n4).

How it works in practice pushes the difficulty out of the protocol and into the metadata. Because the toolbox sits at the aggregator layer, remote MCP servers, OpenAPI tools, A2A integrations and native tools all reach the model through one index, and no new protocol primitive was needed anywhere (n6). What the index actually contains is tool names, descriptions and argument names, which is why benchmarking found the failures were editorial rather than algorithmic. Tool descriptions written by implementers say "get", "create", "manage" and "REST API", while users search for "analytics", "dashboard" and "reporting" (n13). The fix is an index-only field, additional_search_text, that is indexed for retrieval, invisible to the model, and leaves the upstream schema untouched (n14). Above that sits a pinned head of tools that are never retrieved at all, because some capabilities should never have to be rediscovered (n16).

What it costs splits cleanly into a half the source measures well and a half it measures and then walks away from. On the token side the result is strong and the figure carries it: context falls from 541k to 15k at 1,180 tools, and the tool-search curve is roughly flat while the baseline climbs across a 24x increase in catalog size (n9, n10). On the retrieval side the same benchmark reports Recall@10 between 39% and 46%, against a limit that defaults to five (n11). Those two numbers belong in the same sentence and the article never puts them there, which is the single most important thing to carry out of this note.

That leaves the question of how far to trust any of it. This is a T2 vendor post about the vendor's own preview product, and it is better evidenced than that class usually manages, because it runs a public benchmark, states an honest baseline and reports a category where its own method loses. Three caveats survive that credit. The head-to-head is not one experiment, since two of its three rows are borrowed from another paper under a protocol that may not match the self-run (d2). The metadata-tuning percentages are method-free self-report with no dataset and no absolute baselines (n15). And the source never confronts its own headline recall number against its own default shortlist of five.

The same argument, compressed for reference rather than for reading:

Key claims#

• The tool manifest is resident per-turn context scaling with the catalog, not the task. n1
• Prompt caching is a price cut, not an attention cut. n2 ⚠️ single-leg
• Two meta-tools replace the manifest; the catalog stays indexed and never listed. n3 n6
• call_tool exists because runtimes reject unlisted tools. n4
• 541k -> 15k at 1,180 tools (36x), roughly flat across a 24x catalog increase. n9 n10
• Recall@10 of 45.99 / 39.56 / 41.36 - a tuned sparse pipeline competitive with a GPU cross-encoder in two of three categories. n11 n12
• Descriptions become ranking features; the first tuning pass is editorial. n13 n19
• An index-only field separates the searchable surface from the model-facing one. n14
• Pin the head, retrieve the tail. n16 n17

What you will learn, and in what order#

flowchart TB
    subgraph A["A. The tax you are already paying"]
        S1["1 - The manifest is<br/>resident context"]
        S2["2 - Caching cuts the price,<br/>not the attention"]
    end
    subgraph B["B. The mechanism, deliberately small"]
        S3["3 - Two tools instead<br/>of a hundred"]
        S4["4 - Why it needs no<br/>new protocol"]
    end
    subgraph C["C. Does it work? Two different questions"]
        S5["5 - Tokens: measured,<br/>and the figure leads"]
        S6["6 - Retrieval: where the<br/>source is weakest"]
    end
    subgraph D["D. The finding that outlives the product"]
        S7["7 - Tool curation is<br/>an IR problem"]
        S8["8 - Pin the head,<br/>retrieve the tail"]
    end
    A --> B --> C --> D
    S1 --- S2
    S3 --- S4
    S5 --- S6
    S7 --- S8

    style C fill:#e8f0fc
    style D fill:#fbf1dc

This is a reading-order diagram about the note rather than about the product, in four movements, each holding two of the eight numbered sections below. Only two of the movements are coloured, and the colours mark the two things worth carrying away rather than the two things that are longest. Blue marks where the evidence lives, and it is split in half because the source answers one of its two questions well and the other one badly. Amber marks what survives even if the product does not, which is a reframing about metadata that generalises far past tool catalogs.

The crux is that deferring the manifest converts catalog size from a per-turn cost into an indexing cost, and buys a retrieval risk the source measures and then never discusses.

Movements A and B are short, and their shortness is doing argumentative work rather than saving your time. The mechanism genuinely is two tools and no new protocol, and that smallness is itself one of the source's claims (n5). A reader who already believes that a resident manifest is expensive can move through A quickly, at the cost of the caching argument in section 2, which is the one place the obvious objection gets answered properly.

Movement C is where to slow down, and splitting it in two is deliberate. The token result and the retrieval result arrive in the same post, in the same voice, with the same air of measurement, and they have very different evidential quality. Reading them as one number is the mistake this source invites, and section 6 exists to make that hard to do.

Movement D is separated from the rest because it is the part you would still want after switching vendors. Section 7 carries the finding that generalises, and section 8 carries the design shape that tells you where to apply it. If you read only two sections, read 6 and 7, because one is where the source is weakest and the other is where it is most useful.

Synthesized roadmap of this note - not from the source.

Movement A - the tax, and why the obvious relief does not remove it#

flowchart TB
    M["the full tools/list manifest sits in<br/>context on <b>every single turn</b>"]
    T["names, descriptions, JSON schemas,<br/>argument definitions - before the<br/>question is even asked"]
    C["2. caching makes it ~90% cheaper"]
    A["and cached context still competes<br/>for the model's <b>attention</b>"]

    M --> T
    C --> A

    style A fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d

This is a cost diagram, and the distinction in the last box is the movement's whole job. The crux is that caching lowers the price and not the cost: the tokens get cheaper and they still occupy the window and still dilute attention. It is drawn as two separate chains because the relief and the problem are genuinely independent - a team that has enabled prompt caching has addressed the invoice and not the mechanism. That distinction is what makes the rest of the note necessary rather than an optimisation, and it is the sentence most likely to be waved away in a design review.

Synthesized from n1 and n2.

1. The tax you are already paying, and it is charged per turn#

"Every tool you give an agent is both a capability and a distraction." (§intro)

The mechanism behind that sentence is unglamorous. tools/list hands the client every tool definition up front, and those definitions are then resident in the model's context on every turn. The article describes the result as "thousands of tokens of names, descriptions, JSON schemas, argument definitions, and nested parameters before you've asked anything useful" (n1, §intro).

At first glance this reads as a size problem, and size problems have familiar answers. The property that actually matters is subtler, and it is what the cost is a function of. It tracks what is connected, not what the task needs. Attach a hundred-tool MCP server because you occasionally want one tool from it, and you pay for the other ninety-nine on every request, forever, including on the requests that touch no tool at all.

Which raises the objection any engineer reaches for first, and it deserves a straight answer before anything else is built.

2. Caching lowers the price, not the cost#

flowchart LR
    T["the tool manifest, resident<br/>on every turn"]
    P["<b>price</b>: ~90% cheaper<br/>with prompt caching"]
    C["<b>cost</b>: unchanged -<br/>it still occupies the window and<br/>still competes for attention"]
    T --> P
    T --> C
    style P fill:#dcfce7,stroke:#15803d,color:#14532d
    style C fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d

This is a distinction diagram, and the two words are doing all the work. The crux is that caching addresses the invoice and not the mechanism, so a team that has enabled it has bought a discount rather than a fix. It is drawn as one input with two divergent outcomes because they are genuinely independent effects of the same fact, and conflating them is the single most common way this whole line of argument gets dismissed in a design review. Attention is the resource that does not cache, and it is the one the next four sections are about.

Synthesized from n2. The price-versus-cost framing is this brain's.

Prompt caching was already on in the baseline, because it is the Azure OpenAI default, and cached tokens are roughly 90% cheaper. The article does not overclaim from that, and this is the sentence to keep:

Cached tokens are "not free, and cached context still competes for the model's attention." (n2, §The default agent tax) ⚠️ single-leg - prose only.

Background, supplied - and this is exactly where the brain's own measured evidence lands. Prompt caching works by reusing an unchanged prompt prefix, so the provider skips recomputing it and bills a fraction. Nothing about that changes what the model attends over. The tokens are still in the window, still consuming the n² attention budget, still positioned where "lost in the middle" effects apply. This brain records degradation tracking what is in the window, not what it cost to put there (brain/claims.md claims 22 and 27, measured across 18 models). A cached manifest is a fully-priced attention liability at a 90% discount on the invoice.

The consequence is that price and cost have come apart. Caching pays down the invoice and leaves the behavioural cost untouched, so the manifest has to not be in the window at all. But you cannot simply delete it either, because the model has to find out what it can do somehow.

Movement B - the mechanism#

flowchart TB
    A["3. expose exactly <b>two</b> meta-tools:<br/>tool_search(query, limit)<br/>and call_tool(name, arguments) - n3"]
    B["the rest of the catalog stays<br/><b>indexed but never listed</b> - n6"]
    Q{"4. does this need a new<br/>protocol primitive?"}
    N["no - the two meta-tools are<br/>ordinary tools, so every existing<br/>client works unchanged"]

    A --> B --> Q --> N

    style N fill:#dcfce7,stroke:#15803d,color:#14532d

This is a mechanism diagram, and section 4's answer is why the design is interesting rather than merely sensible. The crux is that the whole reframe is implemented inside the existing protocol, because a search endpoint is itself just a tool - so nothing about the client, the transport or the spec has to change. It is drawn ending on the negative answer because that is the load-bearing property: a design requiring a protocol extension would be a proposal, and one that does not is something you can deploy this week. The indexed-but-never-listed line is the trick in five words.

Synthesized from n3, n6 and n7.

3. Two tools instead of a hundred#

• What it teaches: the whole mechanism in two rows. Row 01 is the baseline - every definition, up front, resident. Row 02 is the replacement - tool_search(query, limit) hits a toolbox index and returns "top 5 definitions returned in full", then call_tool(name, arguments) dispatches. n3 §Two tools instead of a hundred
• Corroborated by: the prose describing the model "describing the capability it needs" and receiving a small set of matching definitions.

The figure answers a question without saying so, and it is worth asking out loud. Why are there two meta-tools rather than one? Search alone looks sufficient, since a search that returns a full tool definition has handed the model everything it needs to make the call itself.

Because "if a tool wasn't registered in the original tools/list, many runtimes will guard against the model calling it directly as an unknown tool. call_tool gives the framework a registered, policy-aware dispatch path." (n4, §Two tools instead of a hundred)

That is the most reusable protocol fact in the source, and it arrives sideways, as an implementation constraint rather than a finding. A tool that was retrieved rather than listed is not yet callable, so a registered proxy has to carry the call on its behalf. The constraint then hands the platform something it wanted anyway, which is a single dispatch point that every call passes through, so policy applies in one place instead of being scattered across a catalog.

⚠️ Note the hedge, because it bounds how far you may take this. The source says "many runtimes", not "the protocol". No runtime is named and no spec version is cited, so treat it as a portability constraint you will probably hit rather than a rule you can cite.

The dashed box in that figure is worth a second look as well. Remote MCP servers, OpenAPI, A2A and native tools all sit behind one index, described as "indexed, never listed" (n6). One index across four heterogeneous sources is a strong hint about where this mechanism lives, and it explains why the whole thing needed no protocol change at all.

4. Why it needs no new protocol primitive#

• What it teaches: the toolbox is itself consumed as an MCP server. "Connect to this toolbox using MCP and call it from your agent code", at an endpoint of the form .../toolboxes/{name}/versions/{version}, via MCPStreamableHTTPTool with a bearer token injected per request - with four ordinary MCP servers attached inside it. n7 ⚠️ figure-only - the prose never states that the toolbox is an MCP server.
• And tool search is a plain toggle on an otherwise ordinary toolbox, with no protocol configuration anywhere in the pane (n5).

This is what makes the mechanism unremarkable, in the good sense of the word. An aggregating MCP server is free to expose whatever tools it likes and to service them however it wants, because the client only ever sees the aggregator. Replacing a hundred-tool catalog with a search index therefore needed no new protocol primitive, no model-specific feature, and no shared ranking semantics across providers (n5).

💡 Aggregating (or proxying) MCP server - a server whose tools are drawn from other servers rather than implemented locally. Because a client sees only the aggregator's tools/list, the aggregator can add, hide, rename or index what is behind it without any client or downstream server changing.

Composition is the leverage point, and it is the transferable lesson here. Everything this product achieves comes from sitting between client and servers. Any protocol with a proxy layer has this affordance available, and it is worth asking of any integration standard you adopt: can something sit in the middle and change the shape of what I see?

So the mechanism is cheap to build and cheap to adopt. Whether it works is two questions rather than one, and they have very different answers.

Movement C - does it work, in two halves#

flowchart TB
    A["5. <b>tokens</b>: 541k -> 15k at 1,180 tools,<br/>and the curve is roughly flat<br/>as the catalog grows 24x - n9, n10"]
    B["6. <b>retrieval</b>: does it return the<br/><b>right</b> tool? This half is<br/>largely unexamined"]
    C["so one half is measured well<br/>and the other is asserted"]

    A --> C
    B --> C

    style A fill:#dcfce7,stroke:#15803d,color:#14532d
    style B fill:#fbf1dc,stroke:#b45309,color:#78350f
    style C fill:#fbf1dc,stroke:#b45309,color:#78350f

This is an evidence diagram, not a results tour. The crux is that the question splits cleanly in two and only one half has numbers behind it: saving tokens is worthless if the search returns the wrong tool, and that is the half nobody measured. It is drawn as two branches converging on an asymmetry because reading the movement as a single "does it work" produces false confidence - the token figure is genuinely strong and it answers a question nobody was really worried about. Section 6 is where a reader deciding whether to adopt this should spend their attention.

Synthesized from n9, n10 and n12.

5. Does it save tokens? Yes, and the figure says more than the prose#

• What it teaches: measured on ToolRet (44,000+ tools, 7,000 queries) - "more than 60%" saved at 50 tools, "above 97%" at 1,000, with the endpoint 541k baseline versus 15k with tool search at 1,180 tools, a 36x reduction. n9 §The savings were real
• Corroborated by: the prose stating the same trend directionally.

The figure carries something the prose never says, and it is the more interesting half of the result (n10). The tool-search series is roughly flat, staying inside a band of about 15k to 50k tokens across the whole sweep, while the baseline climbs from around 40k to 541k. "Savings scale with size" is true and it understates what the chart shows.

The transferable claim is not "tools got cheaper". It is that catalog size stops being a context-budget decision at all - it becomes an indexing cost, paid once, off the per-turn path. That is a change in the shape of the cost function, not a discount on it.

⚠️ One caveat the source never mentions is visible in the same chart. The baseline curve has a discontinuity, jumping from about 160k to about 360k between roughly 500 and 550 tools, which is more than a doubling for a catalog increase of about 10%, after which the gentle climb resumes. A clean single-variable ablation should not produce a step like that. It does not threaten the headline, which is measured at the right-hand end of the sweep, but the curve is not the smooth sweep the prose implies.

That is the first of the two questions, and it is the one the source answers well. The second is where this source is weakest.

6. Does it retrieve the right tool? This is the unexamined half#

flowchart TB
    Q["does it return the <b>right</b> tool?"]
    A["if yes: the token saving<br/>is a pure win"]
    B["if no: you have saved tokens and<br/>broken the agent, which is worse<br/>than paying the tax"]
    M["and this is the half<br/>nobody measured"]

    Q --> A
    Q --> B
    Q --> M

    style B fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    style M fill:#fbf1dc,stroke:#b45309,color:#78350f

This is a gap diagram, not a criticism of the product. The crux is that retrieval quality is not a secondary concern here but the thing the entire design depends on, and a false negative is strictly worse than the problem being solved - a tool that exists and cannot be found is a tool you do not have. It is drawn with both outcomes shown because the asymmetry matters: the upside is a cost saving and the downside is a capability loss, and those are not comparable quantities. Anyone evaluating this should measure recall on their own catalog before believing the token figure means anything.

Synthesized from n12. The asymmetry argument is this brain's.

The measure here is Recall@10 on ToolRet, reported across three slices of the benchmark (n11, Figure 3). Two definitions make the numbers legible before we walk them.

💡 Recall@k - the share of queries where the correct item appears anywhere in the top k. Silent about its rank within those k, and silent about everything below.

💡 Cross-encoder reranker - a model scoring query and candidate together rather than comparing precomputed vectors. More accurate and far more expensive, because nothing can be indexed ahead of time: every candidate needs a forward pass at query time.

Take the three methods in order of what they cost to run. BM25s is classical sparse retrieval with no tuning, and it recalls 24.62% on web, 28.23% on code and 32.39% on customized, which is roughly the floor that plain lexical matching buys you. The BGE-reranker-v2-gemma cross-encoder sits at the other end, recalling 45.94%, 38.23% and 49.43%, and it pays for that with a GPU forward pass per candidate at query time. Foundry's tuned sparse pipeline lands at 45.99%, 39.56% and 41.36%, which is a dead heat with the cross-encoder on web, a small win on code, and a loss of about eight points on customized (n11, Figure 3).

The same three rows, for reference:

The claim the source draws from that is genuinely useful. A tuned sparse lexical pipeline matched a GPU cross-encoder in two of three categories without paying for the GPU at serving time (n12), and that is a real data point against the reflex that quality retrieval requires neural reranking.

Two things weaken it, and to the source's credit it states the first itself.

⚠️ It is not one experiment (d2). The BM25s and BGE columns are marked "from [1]", meaning they were lifted from another paper, while the tool-search column is a self-run that deliberately left out the benchmark's instruction string in order to avoid an extra LLM call at serving time. Whether [1] omitted that string too is never stated. The columns may therefore not be like-for-like, and the direction of the resulting bias is unknown, so read the head-to-head as suggestive rather than as a measurement.

⚠️ The second weakness is the bigger one, and it concerns the absolute level rather than the comparison. Recall@10 in the low forties means the right tool is outside the top ten for more than half of queries, and the limit parameter defaults to five. The article closes with "Smaller is not better if the right capability disappears. The shortlist has to be good", and it never places that sentence beside its own table.

This brain's reading, not the source's claim: when the retrieved items are capabilities, a miss does not return a worse answer - it removes an option, and the model may never learn the option existed. That is a different failure mode from a bad search result, and it is unmeasured here.

So the retrieval half is the weak half, which makes the next question the productive one. Why does retrieval fail here at all? The answer is the best thing in the article.

Movement D - the reframe, and what to do with it#

flowchart TB
    R["7. tool curation is an<br/><b>information-retrieval</b> problem - n19"]
    N["so names and descriptions are<br/><b>ranking features</b>, and the first<br/>tuning pass is editorial - n13"]
    P["8. and the practical shape is<br/><b>pin the head, retrieve the tail</b>"]
    W["which is a decision about your traffic<br/>distribution rather than about<br/>your catalog size"]

    R --> N --> P --> W

    style N fill:#e8f0fc,stroke:#4338ca,color:#312e81
    style W fill:#dcfce7,stroke:#15803d,color:#14532d

This is a consequence diagram, and it is the payload of the note. The crux is that reclassifying tool curation as retrieval hands you a whole discipline's worth of technique, and the first thing that discipline says is that your text is a feature rather than documentation. It is drawn as a straight consequence chain because each step follows without a design choice: if retrieval is the frame, then descriptions rank; if descriptions rank, then editing them is tuning; and if some tools are always needed, then pinning them beats retrieving them. The last box is what turns the note into something you can act on without adopting the product.

Synthesized from n13, n19 and n20.

7. The real finding: tool curation is an information-retrieval problem#

What broke was not the ranker.

Benchmarking "revealed that tool search failed when tool descriptions were uneven, sometimes capturing implementation detail instead of user intent vocabulary" - "get", "create", "manage", "REST API". (n13, §Tuning the search space) ⚠️ single-leg.

Their example is exact, and it is worth sitting with for a moment. A tool named execute_query, whose description truthfully says that it "runs a query against the configured database", has to be found by users who type "analytics", or "dashboard", or "SQL", or "reporting", or "warehouse". Not one of those words appears in the tool's own documentation. Accurate and unsearchable are perfectly compatible.

The fix is an index-only alias field (n14, corroborated prose-versus-code):

"execute_query": {"pin": True, "additional_search_text": "SQL database analytics reporting dashboard queries"} - indexed for retrieval, "isn't visible to models in MCP responses", and "no changes are made to the original tool schema of the source MCP server".

Three consequences follow, and they get steadily less comfortable. First, retrieval vocabulary can be tuned without spending any model tokens, because the field never enters the context window. Second, a third-party server can be tuned for your users' vocabulary without forking it, since the upstream schema is untouched. Third, and this is the one nobody in the source mentions, an invisible field now steers which capability the agent is offered, leaving no trace in the context that either the model or a human reviewer can inspect.

⚠️ The self-reported gains attached to this section are the weakest evidence in the source, and they should not travel. Hit rate improved "by about 56%", end-to-end accuracy "by about 55%", recovering to "within about 4% of the full-catalog baseline" (n15), with no dataset named, no absolute baselines, and no statement of whether the percentages are relative or points. They are also not the ToolRet run reported above, so two evaluation regimes are blended inside one section. Do not quote them.

The generalisation survives all of that, and it is the thing the authors say surprised them:

"We thought we were solving a token-cost problem; we were also building a search product" (n19, §intro). At small scale a catalog looks like schema management; at larger scale tool names and descriptions become ranking features.

This is the third instance of one pattern this brain now holds (claim 93). A skill's description is its trigger and causes 50%+ of failures (S5). A column's description becomes a default policy the agent applies (S11). A tool's description becomes a ranking feature. Metadata written for a human to skim gets promoted to a control surface the moment a model reads it - and the incumbent human-facing vocabulary is the dominant failure mode every time.

If descriptions are ranking features, then the remaining question is which tools should be subject to ranking at all.

8. Pin the head, retrieve the tail#

flowchart TB
    D["your tool traffic is a<br/>distribution, not a set"]
    H["the <b>head</b>: a handful of tools used<br/>constantly -> pin them, always resident"]
    T["the <b>tail</b>: hundreds used rarely<br/>-> index them, retrieve on demand"]
    R["so the design question is about your<br/><b>traffic shape</b>, not your catalog size"]

    D --> H --> R
    D --> T --> R

    style R fill:#dcfce7,stroke:#15803d,color:#14532d

This is a strategy diagram, and it is the most portable thing in the note. The crux is that all-or-nothing is a false choice: pinning the head removes the retrieval risk exactly where it would hurt most, and retrieving the tail captures nearly all the token saving. It is drawn from a distribution rather than from a catalog because that is the reframe - two teams with identical tool counts can need opposite designs if one has a flat usage profile and the other a steep one. This also softens section 6's problem without solving it, since the tools most likely to be mis-retrieved are the ones you pinned.

Synthesized from n20.

The last piece is a distribution argument, and it is the shape worth keeping (n16, §Search is for the long tail). The catalog splits in two, and the two halves reach the model by different routes.

The head is the agent's core contract, meaning its policy tools, its frequent data access, and what the source calls "capabilities the model should never have to rediscover". Those are pinned, so they are always present in the manifest and never retrieved. The long tail is everything else, and the source's own examples of it are pointed: rotate a credential, recover a failed deployment, apply a compliance exception, inspect an audit trail. Those are retrieved on demand.

The tail is the interesting half, and the reason is not the obvious one. The reflex reading says that rare tools matter less, so hiding them behind a search is cheap. The reverse is true. Rare tools are disproportionately the emergency ones, so the tail is exactly where a retrieval miss is most expensive, which is what turns this from an optimisation into a design decision. You are choosing which capabilities the agent is allowed to forget it has.

Background, supplied. This is a Pareto split, and the useful part of naming it is that Pareto shapes tell you where to spend differently, not where to spend less. The head justifies hand-curation because it is small and always paid for. The tail justifies machinery because it is large and rarely paid for. Applying one strategy to both is the mistake.

Pinning turns out to carry a second job as well (n17). The source notes that "deterministic pinning keeps the prompt prefix stable, which preserves prompt-cache behavior", which returns us to section 2 from the other side. Prefix stability is a context design constraint rather than a billing detail, because it decides where in the window a dynamic list is allowed to sit.

⚠️ There is a tension inside that same paragraph, though (d3). The toolbox auto-pins frequently used tools per user after a warmup period, and stale entries age out. A pin set that is per-user, warmed up and aged is by construction neither identical across users nor stable across days, which sits badly beside the word "deterministic". Both statements hold only if "deterministic" means manual pins alone, and the text never says so.

The adoption heuristic closes the argument. The authors suggest tool search is worth testing above 10-15 tools, or when different tasks need different subsets of the catalog, or when one agent serves many workflows (n18, ⚠️ single-leg, and a vendor's own adoption guidance for its preview product).

Diagram (mental model)#

flowchart LR
    M["Model"] -->|"tool_search(query, limit)"| IDX[("Toolbox index<br/>name / args / descriptions<br/>+ additional_search_text")]
    IDX -->|"top-k definitions<br/>default k = 5"| M
    M -->|"call_tool(name, args)"| DISP{{"Registered dispatcher<br/>policy applies here"}}
    DISP --> CAT[("The catalog<br/>MCP / OpenAPI / A2A / native<br/>INDEXED, NEVER LISTED")]
    PIN["Pinned head<br/>always in the manifest"] -->|"resident, never retrieved"| M
    CAT -.->|"a miss removes a<br/>CAPABILITY, silently"| GAP(("Recall@10<br/>39-46%"))

    style IDX fill:#e8f0fc
    style PIN fill:#cfe8cf
    style GAP fill:#fbf1dc

Solid arrows are what happens inside a single turn. Green marks what is always in context, which is the pinned head, and blue marks what is fetched on demand. The amber circle is not a component at all. It is the measured risk the source reports and then does not discuss, drawn in because leaving it out would make the design look tidier than the evidence does.

The crux is that two arrows leave the model instead of one, and the second arrow exists only because a retrieved tool is not a callable tool.

Three shape decisions are doing the teaching here. call_tool is drawn as a dispatcher rather than a passthrough, because the registered-proxy requirement (n4) starts as a constraint and ends as a single policy point you would otherwise have had to build yourself. The pinned box bypasses the index entirely and reaches the model directly, which is the visual form of the point in section 8: pinning is not "ranked highly", it is not retrieved at all, and that is precisely what keeps the prefix cacheable. The dotted arrow runs to the gap rather than back to the model, and that choice is deliberate, because on a retrieval miss nothing arrives and nothing is what the model sees. A diagram that routed the miss back into the loop would be claiming a recovery path the source never describes.

Synthesized from n3, n4, n6, n11, n14, n16 - a redrawing of fig_tool-search-figure.png with the measured recall gap added, which the source's own figure does not show.

💡 Terms#

What to distrust in this note#

• T2 vendor post about its own preview product - but better evidenced than that class usually is, which is worth saying plainly: it runs a public benchmark (ToolRet, 44,000+ tools), states an honest baseline, and reports a category where it loses to the neural reranker.
• The head-to-head is not one experiment (d2). Two of the three rows are lifted from another paper, and the self-run deliberately omitted the benchmark's instruction string. Whether the borrowed baselines did too is never stated, so the direction of the bias is unknown.
• The metadata-tuning percentages are method-free (n15): no dataset, no absolute baselines, no relative-versus-points, and not the ToolRet run. The weakest evidence in the source, and it sits next to the strongest.
• The source never confronts its own headline recall number. Recall@10 of 39-46% against a default shortlist of five is the central unexamined gap, and combining it with the closing "the shortlist has to be good" is this brain's reading, not a claim the article makes.
• fig_image-6.png carries n7 alone - the prose never says the toolbox is itself an MCP server, so that fact is figure-only.
• A printed code sample that cannot run (d1): four different names for one API inside one section, raising NameError as printed. A useful reminder that a vendor code sample is evidence of intent, not of a working API.
• The "Background, supplied" blocks are mine - caching versus attention, Pareto strategy, Recall@k and cross-encoder semantics. Uncited by construction.

Open questions#

• What happens on a retrieval miss? The measured Recall@10 says it happens for more than half of queries at k=10, and the default shortlist is 5. No recovery path is described - does the model re-search, widen, or simply proceed without the capability? The highest-value question here.
• Is the unregistered-tool guard a spec rule or a client convention? n4 says "many runtimes" and names none. The answer decides whether the two-proxy pattern is a workaround or the intended shape, and it is cheap to research.
• How much of this survives leaving the tool-catalog setting? The corpus is short, structured, curated and writable - the friendliest possible conditions for sparse retrieval, and the reason the editorial fix works at all. You cannot rewrite someone else's documents to be more searchable.
• What does an aggregating server owe the servers behind it? Error and auth propagation, version skew, downstream schema changes under the index, and who is accountable for a call the aggregator dispatched - all ignored.
• additional_search_text is an unexamined trust surface. An index-only field invisible to the model decides which capability the agent is offered, and on an aggregating server its author may be neither the tool's author nor the agent's owner. See agent-security.md.

Feeds these topics#

• ../../brain/topics/mcp.md - the cost model of tools/list, the unregistered-tool guard, and aggregation.
• ../../brain/topics/rag.md - the retrieval numbers, sparse versus neural, and editorial-before-algorithmic.
• ../../brain/topics/context-engineering.md - the manifest as a measured budget line item, and caching as a price cut that is not an attention cut.
• ../../brain/topics/agents.md - pin the head, retrieve the tail; descriptions as index entries.

Presentation narrative#

A talk track for a team whose agent has outgrown its tool manifest, derived entirely from the gated nodes above. The token measurements are strong and public; the retrieval-quality half is essentially unmeasured, which the fifth slide addresses rather than skips.

Slide 1 - You are already paying this tax, and caching did not remove it#

The full tool manifest sits in context on every single turn - names, descriptions, JSON schemas and argument definitions, before the question is even asked. At 1,180 tools that is 541k tokens of resident context.

Most teams have already reached for prompt caching, and it is worth being precise about what that bought. Caching lowers the price by roughly 90% and does not lower the cost. The tokens still occupy the window and still compete for the model's attention, and attention is the resource that does not cache. That distinction is the one most likely to be waved away, and everything after this slide depends on it.

flowchart LR
    T["manifest, resident every turn"]
    P["price: ~90% cheaper with caching"]
    C["cost: unchanged - window<br/>and attention"]
    T --> P
    T --> C
    style C fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d

This is a distinction slide, not a cost model. The crux is that a discount is not a fix: the two branches are independent consequences of the same fact, and conflating them is how this whole argument gets dismissed in a design review. Attention is the resource that does not cache, and it is what the next five slides are about [n2].

Slide 2 - Above ten or fifteen tools, this stops being schema management and becomes search#

That crossover is the whole claim, and it is worth stating as a threshold rather than a direction. Below it a manifest is simply better: listing everything is cheap, exact and needs no retrieval to go right. Above it the manifest is a resident tax that scales with your catalog rather than your task.

The mechanism follows immediately. Expose exactly two meta-tools - tool_search(query, limit) and call_tool(name, arguments) - and keep the rest of the catalog indexed but never listed [n3, n6].

This is the swap, drawn. The crux is "indexed but never listed" - the catalog does not shrink, it stops being resident [n3, n6].

Slide 3 - It needs no new protocol primitive, which is why you could ship it this week#

The two meta-tools are ordinary tools, so every existing client works unchanged. No spec extension, no transport change, no client update.

That property is easy to skim past and it is what separates this from a proposal. A design requiring a protocol change is a roadmap item that depends on other people; a design implemented inside the existing primitives is a decision your team can make alone. For anyone evaluating the idea rather than the vendor, this is the reason it is worth taking seriously as a pattern.

This is what it looks like as a product decision. The crux is that it is a toggle - which is the visible consequence of needing no new primitive [n7].

Slide 4 - The token result is strong, and the flat curve says more than the 36x#

541k tokens down to 15k at 1,180 tools [n9]. That is the headline, and the shape of the curve is the more useful finding.

The tool-search series stays roughly flat while the catalog grows twenty-fourfold [n10]. A ratio is a measurement at one point; flatness is a claim about how the thing scales. It means the design's cost is governed by how many tools you retrieve per turn rather than by how many exist, which is what makes a large catalog viable at all.

This is the result, and the flat line is the part to quote. The crux is that the two series have different shapes, not just different values [n9, n10].

Slide 5 - The other half of the question is essentially unmeasured#

Saving tokens is worthless if the search returns the wrong tool, and that half has no numbers behind it. This is where anyone deciding whether to adopt the pattern should spend their attention.

The asymmetry matters. If retrieval works, the token saving is a pure win. If it does not, you have saved tokens and broken the agent, which is worse than paying the tax - a tool that exists and cannot be found is a tool you do not have. Those two outcomes are a cost saving and a capability loss, and they are not comparable quantities.

So the concrete recommendation before adopting is to measure recall on your own catalog with your own query distribution. The published token figure tells you nothing about whether that will hold.

flowchart TB
    Q["does it return the right tool?"]
    A["yes -> token saving is a pure win"]
    B["no -> you saved tokens and<br/>broke the agent"]
    Q --> A
    Q --> B
    style B fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d

This is the unexamined half. The crux is that the downside is a capability loss rather than a cost, and the two outcomes are therefore not comparable quantities you can trade off. It is drawn as a single question with both branches shown because presenting only the upside is what makes the token figure look decisive when it settles nothing [n12].

Slide 6 - The real finding is that your tool descriptions are now ranking features#

Reclassifying tool curation as an information-retrieval problem is the payload, and it hands you a whole discipline's worth of technique [n19]. The first thing that discipline says is that your text is a feature rather than documentation.

That makes the first tuning pass editorial rather than algorithmic [n13]. Before anyone touches an embedding model, someone rewrites tool names and descriptions so they rank well against the queries your agents actually produce. That is cheap, it is unglamorous, and it is where the gains are.

And the practical shape is not all-or-nothing. Pin the head and retrieve the tail: a handful of tools used constantly stay resident, hundreds used rarely get indexed. That removes the retrieval risk exactly where it would hurt most while capturing nearly all the saving - and it makes the design question about your traffic distribution rather than your catalog size.

flowchart TB
    D["tool traffic is a distribution"]
    H["head: pin, always resident"]
    T["tail: index, retrieve on demand"]
    D --> H
    D --> T
    style H fill:#dcfce7,stroke:#15803d,color:#14532d

This is the strategy, and it is the portable part. The crux is that two teams with identical tool counts can need opposite designs, because the shape of the traffic decides rather than the size of the catalog. It is drawn from a distribution rather than a list for exactly that reason, and it softens the previous slide's risk without removing it, since the tools most likely to be mis-retrieved are the ones you would have pinned [n20].

Key takeaway message#

A tool manifest is resident context charged on every turn, and prompt caching lowers its price without lowering its cost, because attention does not cache. Somewhere around ten to fifteen tools the problem stops being schema management and becomes search, at which point two meta-tools replace the manifest and the catalog stays indexed but never listed - inside the existing protocol, so every client works unchanged. Context falls from 541k tokens to 15k at 1,180 tools and, more importantly, the curve is flat as the catalog grows. The unmeasured half is whether retrieval returns the right tool, and its failure is a capability loss rather than a cost, so measure recall on your own catalog first. The finding worth carrying is that tool names and descriptions have become ranking features, which makes the first tuning pass editorial. Pin the head, retrieve the tail.