source

How we built LangChain's agent-first data stack

Emily Hawkins (data team, LangChain)

TL;DR#

An "agent-first data stack" turns out not to be a data stack at all. It is a documentation layer wrapped around an unchanged one. The figure captioned "LangChain's data stack architecture" shows a stock ELT pipeline - Fivetran and Airbyte and Segment into BigQuery, dbt on top, reporting at the end - with no agent, no semantic model and no feedback loop anywhere in it (d1). Everything that makes the stack "agent-first" lives in a second figure, and all of it is prose.

The transferable move is that a column definition stops being a description and becomes an instruction. account_status: The status of the account. becomes a paragraph that spells out each lifecycle value in business terms and then issues an imperative: "For customer reporting, filter to Active unless the analysis explicitly includes churned or prospective accounts" (n3). That is not documentation. That is a default policy stored in metadata, where the agent meets it at exactly the moment it matters.

The loop that maintains it inverts what a data team is for. Observability over agent conversations shows where context is missing. The team then writes the missing context. So the loop's output is never an answer to a user but a write back into the store (n7, n8).

Read the results with both hands. Every number here measures adoption - 2,200 conversations, 40x throughput, 100% migration in six weeks - while the article's thesis is about trustworthiness, which it never measures (d4). The authors concede it and file evals under "next" (n10). Deep research supplies both halves the source lacks. Schema documentation is measured to help, and far more on real warehouses (+16pp) than on public benchmarks (+2pp). The enterprise text-to-SQL setting closest to this stack tops out around 65.6%. The mechanism is well corroborated. The result is not.

flowchart TB
    C["The article's claim:<br/>'a big architectural shift'"]

    subgraph SHOWN["What the architecture figure actually shows - d1"]
        direction TB
        P1["Fivetran, Airbyte, Segment"]
        P2["BigQuery, dbt, reporting"]
        P3["no agent, no semantic model,<br/>no feedback loop"]
        P1 ~~~ P2 ~~~ P3
    end

    subgraph REAL["Where the agent-first-ness actually lives - all of it prose"]
        direction TB
        R1["column definitions rewritten<br/>as instructions - n3"]
        R2["five context stores, one per<br/>question asked - n2"]
        R3["a loop whose output is a write<br/>to the store, never an answer - n8"]
        R1 ~~~ R2 ~~~ R3
    end

    C --> SHOWN
    C --> REAL
    SHOWN --> V["the plumbing was never the problem"]
    REAL --> W["the deliverable is English,<br/>and it costs three permanent people"]

    style SHOWN fill:#fdeaea,stroke:#dc3545,color:#7f1d1d
    style REAL fill:#e8f4ea,stroke:#28a745,color:#14532d

This is a claim-versus-artifact diagram, not an architecture diagram, and the two columns are the same system described by two different parts of the same article. The crux is that an "agent-first data stack" turns out to be a documentation layer wrapped around an entirely unchanged stack, and the strongest evidence for that is the article's own figure, which is captioned as the architectural shift and contains none of the things that make the stack agent-first. The columns are drawn as siblings under one claim rather than as before-and-after because they are simultaneous: both are true descriptions of the same company on the same day. What follows from the shape is the cost line on the right, since a deliverable made of prose is bounded by how fast people can write it rather than by anything you can provision. Synthesized from n2, n3, n8 and divergence d1.

The 1-minute version#

This article covers how a company of roughly 290 people put a data agent in front of its own warehouse for the whole staff, and what it had to write down before that could work. The interesting object here is not the pipeline underneath, which is entirely ordinary. It is the layer of English wrapped around it, and the first question worth asking is what all that English is actually for.

The answer starts with a failure that does not announce itself. An agent pointed at a company's warehouse will happily write valid SQL and return a technically correct answer based on the wrong business interpretation, which is the failure that looks exactly like success. Before the agent existed, three people held the context that would have prevented it, and every question queued behind them. So the problem is not that nobody can write the query. It is that the knowledge which makes a query mean the right thing lives in a small number of heads.

To see why that is hard rather than merely tedious, compare it to a coding agent. When a coding agent guesses wrong, a test fails or the process throws, and the wrongness surfaces on its own. When a data agent forgets that churned customers are normally excluded, it does not error or hedge. It returns a confident number that is wrong by exactly the size of your churn, and hands it to somebody who will put it in a board deck. There is no traceback for a wrong assumption, which is what makes context a safety property here and not just a quality one.

At first glance the fix looks like a plumbing problem, and the naive attempt is to hand a competent model warehouse credentials and the schema. It collapses in four separate ways (n1). The agent queries accounts_legacy rather than accounts, because both exist and nothing ranks them. It computes ARR by a textbook formula rather than yours, because your formula was never written anywhere machine-readable. It writes correct SQL that answers a subtly different question, because words like "customer" and "active" carry local meaning. And it reaches for an asset that "looks relevant but is not the best source", because trust is not a property of any single table (n6). Notice that none of those four are fixed by re-plumbing. The article calls its migration "a big architectural shift" and then publishes an architecture diagram containing no agent, no semantic model and no feedback loop (d1), which is the strongest evidence in the piece that the plumbing was never the problem.

The idea, then, is to make implicit context explicit, and to decompose that context by the question each store answers rather than by the tool that happens to hold it (n2). One store says what the data is. A second says what a metric means. A third says how the business works. A fourth says which source to trust. A fifth says how a number is actually computed. The rule that falls out is the portable part, and it doubles as a stopping condition: a layer that cannot name a question only it answers is a duplicate, and duplicated context is worse than missing context because the two copies drift.

How it works in practice is less exotic than the framing suggests, because almost all of it is prose. A column definition stops being a description and becomes an instruction, so account_status: The status of the account. becomes a paragraph naming each value's business meaning and then issuing an imperative - "For customer reporting, filter to Active unless the analysis explicitly includes churned or prospective accounts" (n3). Context that fits no schema field becomes a versioned markdown document, which the author herself calls "like skills for the data agent" (n5). The whole arrangement then stays alive on a loop whose demand signal is the agent's own conversations, triaged from symptom to layer, and whose output is a write back to the context store and never an answer to a user (n7, n8). The one guardrail on it is a writer restriction on the trust flag, because "if everything is endorsed, the signal stops being useful" (n6).

What that costs is three permanent people, and the reason is older than the technology. The LLM collapsed the encoding cost of expert knowledge, since the target formalism is now English prose rather than production rules. It left the elicitation cost completely untouched, because somebody still has to sit with the GTM team and find out what they mean by "pipeline" (F5, Feigenbaum 1977). That is a 45-year-old diagnosis arriving on schedule, and it is why this is a standing headcount rather than a project.

How far you should trust the result is the part to read most carefully. Every number the article reports measures adoption while its thesis is about trustworthiness, which is measured nowhere, and the authors concede exactly that and file evals under "next" (d4, n10). The headline 40x compares mismatched units and should never be quoted bare (d3). What weight this note carries comes from the research pass rather than from the article, because the mechanism is externally corroborated and the result is not.

The same argument, compressed for reference rather than for reading:

Key claims#

How to read this note#

This is written as a ramp from zero, in the order an architect would walk a new engineer through the problem. First comes the problem itself, before any technology. Then the machinery underneath, so the vocabulary is not in the way. Then the obvious approach and how it fails. Only then is the design derived one piece at a time, so that it feels inevitable rather than arbitrary. If you already work with warehouses daily, skip Part 1.

One honesty rule for this note. AGENTS.md scopes a LEARNING.md to a single question - what did this source teach? - and a ramp needs foundations the source never taught. So Part 1 is marked scaffolding and is uncited by construction: it is background I am supplying, not knowledge from this source. Everything from Part 2 onward carries a node ID (n1, d1) or an external reference (F1), and where a conclusion is mine rather than the article's, it says so.

flowchart TB
    P0["Part 0 - the problem,<br/>before any technology"]
    P1["Part 1 - foundations<br/><i>scaffolding, uncited - skip if you<br/>work with warehouses daily</i>"]
    P2["Part 2 - the naive agent,<br/>and precisely how it fails"]

    subgraph CORE["The derivation - the reusable part"]
        direction TB
        P3["Part 3 - the five stores,<br/>derived one residual question at a time"]
        P4["Part 4 - one question,<br/>traced end to end"]
        P5["Part 5 - a definition is a prompt"]
        P3 --> P4 --> P5
    end

    subgraph LIVE["Keeping it alive - where the costs are"]
        direction TB
        P6["Part 6 - the second-order problems"]
        P7["Part 7 - how would you know<br/>any of this works?"]
        P6 --> P7
    end

    P8["Part 8 - what you would build first"]

    P0 --> P1 --> P2 --> CORE --> LIVE --> P8

    style CORE fill:#e8f4ea,stroke:#28a745,color:#14532d
    style LIVE fill:#fff4e5,stroke:#b45309,color:#78350f
    style P1 fill:#eef2ff,stroke:#4338ca,color:#312e81

This is a reading-order diagram about the note rather than about LangChain, and the colours mark three different kinds of material rather than three topics. Green is the derivation and it is what transfers to a company that is not this one, so a reader with limited time should spend it there. Blue is scaffolding, supplied by this brain and cited to nothing by construction, and it is the one part that is safe to skip outright. Amber is where the article stops being a design description and starts being an operating report, which is also where its evidence gets thinnest. Part 8 sits outside every grouping because it is the only part that tells you what to do rather than what is true, and it carries the note's one direct disagreement with another source in this brain.

Generated from the structure of this note - a diagram the article does not contain.

Part 0 - The problem, before any technology#

Forget agents for a moment. A company of about three hundred people accumulates data in a dozen systems, which is to say the billing platform, Salesforce, the product's own database, and whatever the marketing team signed up for last year. Someone in GTM then asks a perfectly ordinary question:

"How is pipeline looking this month?"

To answer it you have to know four separate things, and notice that only one of them is about SQL. You have to know where that data physically lives, across those dozen systems. You have to know what the fields mean, so that acct_st = 'A' reads as an active paid contract rather than an active trial. You have to know what this particular person means by "pipeline", which is a company-specific definition that differs between the weekly GTM report and the sales forecast. And you have to know which existing report is the trustworthy one, out of the four that all compute something pipeline-shaped. As a checklist:

1. Where that data physically lives, across those dozen systems.
2. What the fields mean - that acct_st = 'A' means an active paid contract and not an active trial.
3. What this person means by "pipeline", which is a company-specific definition that differs between the weekly GTM report and the sales forecast.
4. Which existing report is the trustworthy one, out of the four that all compute something pipeline-shaped.

Only one kind of person holds all four at once, and that is the data analyst. So every question queues behind them. LangChain describes exactly this state, and it is the entire motivation for what follows: "answering them usually required a data team member to translate the question, find the right model, write or adjust the query, validate the result, and send back an answer" - a team that at the time "was just one person" [S11 §Where we started].

Hold onto this framing, because it is the one that makes the rest make sense: the bottleneck is not query-writing. It is that the four things above live in one person's head and nowhere else. An agent that writes SQL beautifully but has none of those four solves the wrong problem. Before we can see why, though, it is worth knowing what the machinery underneath actually looks like.

Part 1 - Foundations (scaffolding, not from the source)#

⚠️ This part is uncited background I am supplying so the rest reads. Skip if you already work with warehouses. Nothing here is a claim from the source.

Essentially every modern company data stack has the same shape:

sources  ->  ingestion  ->  warehouse (raw)  ->  transformation  ->  warehouse (prod)  ->  reporting

💡 ELT - extract, load, transform. Land the raw data in the warehouse first, then transform it in place with SQL. Older ETL transformed before loading. ELT won because warehouse compute got cheap and keeping the untouched raw copy is worth a lot when a definition turns out to be wrong.

Three of those boxes are worth naming properly. The warehouse, here BigQuery, is a database built for scanning large tables analytically rather than for serving an application, and you will hear "the warehouse" used loosely to mean the whole storage layer. Ingestion tools such as Fivetran, Airbyte and Segment are managed connectors that copy Salesforce, the billing system and event streams into the warehouse on a schedule, and nobody writes these by hand any more. dbt is the transformation layer, where you write SELECT statements and dbt manages their dependency order, materialises the results as tables or views, and holds their documentation and tests alongside the SQL in a git repo. The important part for us is that a dbt project is source code, so it is versioned, reviewable, and readable by anything that can read a repo.

💡 Model (dbt sense) - one SELECT statement that produces one table. Not a machine-learning model. Data people say "model" constantly and mean this.

💡 Grain - what one row of a table represents. "One row per customer per month" is a grain. Getting the grain wrong is how you accidentally sum a value four times. It is the single most common source of quietly wrong analytics.

💡 Semantic layer - a declarative definition of metrics and entity relationships (ARR, pipeline, active usage) sitting above the physical tables, so "revenue" resolves to one agreed calculation instead of being re-derived per query. Long-standing BI infrastructure; LookML shipped the idea in 2012.

That leaves the consumption end. A dashboard is a fixed set of charts someone built in advance, and a notebook is an open workspace where you write queries ad hoc. The gap between the two is the whole self-service problem, because dashboards answer only anticipated questions while notebooks require you to know SQL and the schema already.

Here is LangChain's actual instance of that generic shape, which is the article's own diagram:

This is the whole board, and it is worth studying for ten seconds before moving on [S11 visuals/fig2_data-stack-architecture.png, captioned "LangChain's data stack architecture"]. Every box is a named product doing an unremarkable job. There is no agent in it. No semantic model, no workspace guides, no endorsements, no feedback loop. All of those appear only in a different figure, which you will meet in Part 6.

Keep that absence in mind for the rest of this note. The article calls this migration "a big architectural shift", and the diagram it publishes to illustrate the shift is a pipeline that would have looked identical in 2022 (d1). The finding, stated more plainly than the source states it: what changed was the reporting tool at the right-hand edge and the amount of English written around the warehouse. The plumbing did not move. If you came looking for a new data architecture for the agent era, the honest answer on this evidence is that there is not one yet.

That is the entire board. Now put an agent on it.

Part 2 - The naive agent, and precisely how it fails#

Hand a competent model warehouse credentials and the schema, meaning table names, column names and types. It will write syntactically valid, executable SQL. This works well enough in a demo that many teams stop here, which is why it is worth being precise about what breaks.

The source names four failure modes, and they are worth separating because they fail differently [S11 §intro, n1]. The first is that the agent queries accounts_legacy instead of accounts, since both exist, both look plausible, and nothing in a bare schema ranks them. The second is that it computes ARR by a textbook formula rather than yours, because the agreed formula was never written down anywhere machine-readable. The third is subtler, because the agent writes correct SQL that answers a subtly different question, and it does that because words like "customer" and "active" carry local meaning the schema cannot carry. The fourth is that it picks an asset that "looks relevant but is not the best source" [n6], and it cannot do better because trust is not a property of any single table. The four, compressed:

Now the one that should worry you most, and the reason this whole topic exists rather than being a prompt-engineering footnote:

A query that returns the right number for the wrong population is indistinguishable from a correct answer.

If the agent forgets that churned customers should normally be excluded, it does not error, return null, or hedge. It returns a confident number that is wrong by exactly the size of your churn, and hands it to someone who will put it in a board deck. There is no traceback for a wrong assumption. Compare this to a coding agent, whose bad output usually fails a test or throws.

The generalisation to carry out of this note: the danger of a domain agent scales with how plausible its wrong answers look, not with how often it is wrong. That is what makes context a safety property here and not just a quality one. Which raises the question the rest of the source answers, namely what exactly has to be written down.

Part 3 - Deriving the context layer, one residual question at a time#

flowchart TB
    Q1{"the agent picked the<br/>wrong table"} --> S1["1. table and column definitions<br/><i>what the data is</i>"]
    S1 --> Q2{"it picked the right table<br/>and the wrong formula"}
    Q2 --> S2["2. the semantic model<br/><i>what a metric means</i>"]
    S2 --> Q3{"right formula, and it still<br/>answered a different question"}
    Q3 --> S3["3. workspace guides<br/><i>how the business works</i>"]
    S3 --> Q4{"two assets both look right"}
    Q4 --> S4["4. the trust signal<br/><i>which source to believe</i>"]
    S4 --> Q5{"the number still does not<br/>match the board deck"}
    Q5 --> S5["5. computation lineage<br/><i>how the number is actually made</i>"]

    style S5 fill:#e8f4ea,stroke:#28a745,color:#14532d

This is a derivation diagram, not a taxonomy, and the diamonds matter more than the boxes. The crux is that each store exists because of a specific failure the previous four cannot explain, which is what makes five the right number rather than an arbitrary one and gives you a stopping condition you can apply to your own stack. It is drawn as an alternating failure-and-answer chain because the article presents these as a list, and a list hides which items are load-bearing: a reader handed five boxes cannot tell that dropping the third one reintroduces a failure the other four are blind to. The rule that falls out is the portable part and it doubles as a test - a layer that cannot name a question only it answers is a duplicate, and duplicated context is worse than missing context because the copies drift apart silently.

Synthesized from n2. The residual-question framing is this brain's; the article lists the five.

This is the heart of the source, and it is much more useful derived than listed. The method is simple. Give the agent what we have so far, ask "what can it still not know?", and let each answer name the next store. The five that fall out are exactly the five in the article's own figure [S11 §How we think about context, visuals/fig3_feedback-loop.png, n2].

Start: the agent has the schema. Table names, column names, data types.

Residual 1: what does this column actually mean?

The type says account_status is a string. Nothing says which strings, or what they signify. So the agent must guess that 'Churned' implies a former paying customer rather than a cancelled trial.

→ Store 1: table and column definitions, living in dbt and surfaced through the warehouse.

This is the one layer that cannot be skipped, and the reason is that it is the only layer that grounds vocabulary. Everything above it assumes the words already mean something fixed.

Residual 2: what is "ARR"?

Now every column is documented. A user asks for ARR. The agent looks for an arr column and finds either nothing or three different ones. ARR is not a column. It is a calculation over several tables at a particular grain with particular filters, and your company has agreed on one version of it.

Notice why the previous store cannot help. No amount of documenting columns individually produces an agreed formula, because a metric is a fact about a combination of columns, and there is nowhere in a per-column store to write a fact about a combination.

→ Store 2: the semantic model, defining metrics and how models relate.

Residual 3: how does this company actually operate?

Metrics are now defined. A user asks for "pipeline for the weekly GTM report", and GTM's weekly definition excludes renewals while the sales forecast includes them. The same gap opens in several other places. Somebody has to say which dashboards count as canonical for a metric. Somebody has to say how product usage should be read across deployment types. And somebody has to say when a question should be escalated to a human [S11 §Capturing business context, n5].

None of this is a fact about data. It is a fact about the organisation, and there is no schema slot for "on Mondays we exclude renewals".

→ Store 3: workspace guides - prose documents, managed in a GitHub repo that syncs into the tool, so they are versioned and reviewed like code.

The author names the family herself: "These would be like skills for the data agent" [n5]. She is not citing anyone; the word simply fits. Markdown, versioned, reviewed on change, injected as context, steering how to do something rather than stating a fact. That is a skill, and this is the first instance in this brain of the pattern outside a coding context.

Residual 4: which of these five ARR assets is the real one?

Business processes are now documented. The agent goes looking for ARR and finds four tables and two dashboards that all compute something ARR-shaped, some stale, one superseded, one canonical. Every one of them is individually well documented.

This is the subtle one. Documentation of each asset can never resolve it, because canonical-ness is a property of the relationship between assets, not of any asset. So you need a layer whose entire job is ranking.

→ Store 4: endorsements - a trust flag marking the canonical asset [S11 §Endorsements, n6].

Residual 5: why is this number what it is?

The agent now picks the endorsed ARR dashboard and returns a figure. The user says "that does not match my spreadsheet." Answering that requires the joins, filters and CASE statements that produced the number.

And there is a sharper reason this layer must exist. A definition says what a column means today; only the SQL says how it actually got that way, and definitions drift from code. When prose and implementation disagree, the implementation is what ran.

→ Store 5: the dbt repo itself as a context source, so the agent "can inspect the underlying model logic, understand joins and transformations, and trace how a field is produced" [S11 §GitHub provides deeper implementation context].

The five, and the rule to take away#

The design rule: sort context by the question it answers, not by the tool that stores it. A layer that cannot name a question only it can answer is a duplicate - and duplicated context is worse than missing context, because the two copies drift and nothing tells you which is stale.

That rule is portable. Swap "table" for "API endpoint" and "dashboard" for "internal wiki page" and the five residuals reappear unchanged for any agent over any proprietary domain. Nothing in this decomposition is actually about data, which is why this brain filed it under context engineering rather than opening a new topic (ADR-0014). Derived one at a time the five feel inevitable, so it is worth watching them all fire at once.

Part 4 - One question, traced end to end#

flowchart TB
    Q["'How many active customers<br/>did we add last quarter?'"]
    A["definitions: which table holds<br/>accounts, and which is legacy"]
    B["semantic model: what 'added'<br/>counts as, and from which date"]
    C["guides: that 'customer' excludes<br/>internal and trial accounts"]
    D["trust signal: which of two<br/>candidate tables is endorsed"]
    E["lineage: how the number was<br/>computed the last time"]
    R["a number somebody can<br/>put in a board deck"]

    Q --> A --> B --> C --> D --> E --> R

    F["remove any one of these and the<br/>agent still returns a confident number,<br/>wrong by exactly the amount you omitted"]
    A -.-> F
    B -.-> F
    C -.-> F
    D -.-> F
    E -.-> F

    style F fill:#fdeaea,stroke:#dc3545,color:#7f1d1d
    style R fill:#e8f4ea,stroke:#28a745,color:#14532d

This is a trace diagram, not a pipeline, and the dashed edges carry the teaching rather than the solid ones. The crux is that every store is consulted on a single ordinary question, and omitting any one of them produces not an error but a plausible wrong answer, which is the failure mode that makes context a safety property here rather than a quality one. It is drawn with one shared failure node because drawing five separate failures would suggest they are distinguishable in practice, and the whole problem is that they are not: there is no traceback for a wrong assumption, so all five omissions look identical from the outside, like a confident number arriving on time.

Synthesized from n2, n3 and n6. The traced request is this brain's construction, following the contract's "trace one concrete instance end to end".

Watch all five fire on a single realistic question:

"What was ARR from active customers last quarter?"

Walk it a step at a time. The agent first has to know which tables hold accounts and subscriptions, and the column definitions answer that. Without them it queries the legacy table. Next it needs the agreed ARR formula and its grain, which is the semantic model's job. Without it the agent invents a plausible formula, and the number comes out wrong by the monthly-to-annual conversion. Then it has to know that "active" excludes churned and prospect accounts, which comes from the column definition's default rule, and without that the answer silently includes churned revenue. It also has to know whether "last quarter" is fiscal or calendar, which is a workspace guide fact, and getting it wrong moves the answer by a month invisibly. It then has to decide which ARR asset is canonical, which is what the endorsement is for, or else it takes the stale dashboard. And finally, when the user objects, it has to explain why the number disagrees with their spreadsheet, which only the dbt repo can support. Without that the agent cannot explain itself and the user stops trusting it. The same trace, compressed:

Step 3 is the one to stare at. The agent was never asked to think about churned accounts. The exclusion happened because someone wrote a filtering rule into a column description, and the model read it while doing something else. That is context doing its job invisibly, and it is also why you cannot tell from a correct answer whether your context layer is working. Which becomes Part 7's problem. First, though, look closely at the sentence that did the work.

Part 5 - A definition is a prompt#

Here is the weak version, which is what most companies have:

account_status: The status of the account.

Before reading on, list what is missing. This is worth thirty seconds, because the gap is the whole lesson.

The article's strong version [S11 §How we define the data models, n3]:

account_status: The current lifecycle status of the account in Salesforce. Active means the
customer has an active paid contract. Churned means the customer previously had a paid contract
that has ended. Prospect means the account has not yet become a customer. For customer reporting,
filter to Active unless the analysis explicitly includes churned or prospective accounts.

Four things happen in that paragraph, and only the first is documentation. It names Salesforce as the system of record, which is provenance. It then enumerates the allowed values and defines each in business terms rather than restating the label. Having done that, it says what the distinction implies for analysis, which is interpretation guidance. And then it does something different in kind, because it issues a default policy - filter to Active unless told otherwise - which is an imperative addressed to the agent and stored in a metadata field. The four, compressed:

1. Provenance - names Salesforce as the system of record.
2. Enumeration with meaning - the allowed values, defined in business terms rather than restating the label.
3. Interpretation guidance - what the distinction implies for analysis.
4. A default policy - filter to Active unless told otherwise. An imperative. An instruction to the agent, embedded in a metadata field.

Point 4 is the move. The stated purpose is to prevent "a technically correct answer based on the wrong business interpretation", which is Part 2's dangerous failure, defused at the only point where it can be defused cheaply.

Why this is a pattern and not a tip#

It would be easy to file that as a documentation tip, and the reason not to is that this brain has now seen the same move three times in three unrelated domains. A skill's description turns out to be its trigger, and gets the blame for more than half of skill failures (S5, claim 43). A tool's name and description turn out to be ranking features, so the first tuning pass on a tool catalogue is editorial rather than algorithmic (S10, claim 88). And a column's description turns out to be a default policy the agent applies (S11, n3). The three, side by side:

• A skill's description is the trigger, and causes 50%+ of skill failures (S5, claim 43).
• A tool's name and description are ranking features, so the first tuning pass is editorial rather than algorithmic (S10, claim 88).
• A column's description is a default policy the agent applies (S11, n3).

In each case a field written for a human to skim was quietly promoted to a control surface the model acts on. And in each case the incumbent vocabulary is the failure, whether that is implementer shorthand such as "get", "manage" and "REST API" in S10, or "the status of the account" here. Both were written for a reader who already knows the answer.

Metadata written for humans underperforms as metadata written for models, and nobody notices until an agent reads it. This is claim 93, and it is the most robust thing in this note.

It is also, unusually for this note, measured (F1). Column descriptions are worth +20% accuracy on completely uninformative column names (arXiv:2408.04691, T3, BIRD-Bench). On a real warehouse with genuinely ambiguous names, query-derived descriptions moved execution accuracy 36% to 52%, where the same intervention bought only +2pp on a public benchmark (MotherDuck, T2).

💡 Execution accuracy (EX) - the standard text-to-SQL metric: the fraction of generated queries whose result set matches the gold query's. Measures the answer, not the phrasing.

The eight-fold gap between those two numbers is itself a lesson. Benchmark schemas have distinct, clean column names, whereas real warehouses have acct_st, acct_status and account_state sitting side by side. Documentation pays where names are ambiguous, which is where every real company lives and no public benchmark does (claim 94). In other words, a context intervention that looks marginal on a benchmark may be decisive in your system, and neither number transfers. That is the design settled. What remains is everything that goes wrong once it is running.

Part 6 - The second-order problems#

You now have a design. Everything below is what goes wrong once it is running, which is the part that separates a demo from a system. Start with the question that determines whether it can exist at all.

Who writes all this, and forever?#

Someone has to sit with GTM and find out what they mean by pipeline. That work has a name and a 45-year-old diagnosis (F5). It is knowledge engineering, and the constraint is Feigenbaum's knowledge acquisition bottleneck (1977). Expert systems were limited not by inference but by the human labour of eliciting and encoding expertise, in "a very painstaking way that reminds one of cottage industries".

💡 Knowledge acquisition bottleneck - the limiting factor in a knowledge-based system is the human labour of extracting expert knowledge and encoding it usably, not the system's reasoning power. Identified in 1977 and never solved, only made cheaper.

What changed since 1977 is the target formalism: English prose instead of production rules, which collapses the encoding cost. The elicitation cost has not moved at all. That is precisely why this stack costs three permanent people rather than one project, and it is the second independent historical precedent for claim 72, the first being Bush's Memex in 1945. Two separate literatures, 1945 and 1977, reached the same conclusion, and S11 is what it looks like when someone pays the bill. Paying it buys you a context layer, which immediately raises the question of what stops that layer degrading.

What stops the trust signal becoming noise?#

Endorsements only work while they are scarce. The article states the principle in one line worth memorising [S11 §Endorsements, n6]:

"If everything is endorsed, the signal stops being useful."

That is a general property of trust signals, not a data detail. A signal carries information only in proportion to what it excludes, which is why a code-owners file, a "verified" badge and a @deprecated annotation all decay the moment everyone can apply them. So endorsement gets a writer restriction, meaning only the data team may set it, and endorsed assets need review before changes ship. The transferable form of that is short. If you add a trust flag to any agent-facing store, decide who may write it before you decide what it means.

Prior art makes the point stronger, and it also improves on the answer (F4). Microsoft's Power BI has shipped endorsement for years, with attribution and search-priority effects, and it has two tiers where this source has one. The lower tier, Promotion, may be applied by any content owner or by anyone with workspace write access. The higher tier, Certification, may be applied only by an admin-defined reviewer group, and only if an admin has enabled the feature at all. Side by side:

The two-tier split is the better answer to the same saturation problem, because the cheap tier absorbs the volume of "this is good, use it" so the scarce tier stays scarce without making its gatekeepers a bottleneck. LangChain has re-derived Certification and not yet re-derived Promotion. That a hyperscaler's governance feature and a three-person data team arrived at the same primitive independently, years apart, is better evidence that trust signals are structurally necessary than either instance alone. Scarcity keeps the signal meaningful, but something else has to keep the content true.

What keeps it current?#

The demand signal is the agent's own conversations, and the article turns that into a triage rule mapping symptom to layer [S11 §How we improve the system, n7]. When people keep asking similar questions, the fix is to build a dashboard. When the agent repeatedly struggles with a metric, the semantic model needs clarifying. When questions need internal business context, someone writes a workspace guide. And when the agent reaches for the wrong source, the fix is in the endorsements or the dbt docs. As a lookup:

That still sounds like unbounded human work, which is where external evidence helps, because the first draft can be machine-written. MotherDuck mined descriptions from query history, tracking how often an identifier appears, in which clauses, and alongside which others, for ~$0.50 per warehouse, and that is where the +16pp of Part 5 came from (F2). A separate study found "Common Queries" the highest-yield metadata component of all. Two unrelated teams found usage to be the best signal for what to document. Only the genuinely ambiguous cases still need the expert, which is exactly where the same paper found LLM generation falls down. Put the demand signal, the stores and the humans on one picture and the shape of the whole system becomes visible.

The loop, and the human the diagram forgets#

What it teaches: the source's own drawing of the maintenance loop, and the object the rest of this section is about. Corroborated by: the surrounding prose describing the loop [n7, n8]. Shown here because the criticism below is about what this figure omits, and a reader cannot check that against a figure they have not seen.

flowchart LR
    subgraph CTX["Context layer (all of it prose)"]
        direction TB
        D["Table + column defs<br/>what the data is"]
        S["Semantic model<br/>what a metric means"]
        G["Workspace guides<br/>how the business works"]
        E["Endorsements<br/>which source to trust"]
        R["dbt repo<br/>how it is computed"]
    end

    CTX --> A["Data agent<br/>answers questions"]
    A --> U["Users<br/>Hex, Slack, CLI, MCP"]
    A --> O["Observability<br/>gaps, warnings, repeated topics"]
    O --> H{{"Data team<br/>reviews and decides"}}
    H -->|"writes"| CTX

    style H fill:#7a2828,stroke:#e06666,color:#fff
    style CTX fill:#1f3a54,stroke:#6fa8dc,color:#fff
    style A fill:#2a4a2a,stroke:#93c47d,color:#fff

Read it clockwise from the blue box. Blue is the context layer, which is the five prose stores and no code at all. Green is the running agent. The red diamond is a human, and it is the only human in the picture. Context feeds the agent. The agent's behaviour is observed. Those observations reach a person, and that person writes back into the context layer. Users do receive answers, but notice where they sit, which is outside the cycle that improves anything. The crux is that the output of a working data-agent loop is not answers, it is context, and the only thing that closes the loop is a human writing prose.

Three choices give it that shape, and each is worth taking in turn. First, the loop terminates in the context box rather than at the user, which is what converts the data team's job from a queue of requests into maintenance of a shared artifact (n8). A queue never compounds and an artifact does, and that single property is the whole return on the investment. Second, observability sits between the agent and the human rather than beside them. Without it the team is guessing which of five stores to improve, and the triage table above is only executable because the conversations are logged (n7). Finally, and this is the choice that matters most, the red diamond is drawn here because the source's own architecture diagram omits it. In fig3 the feedback arrow runs from usage trends straight back into dbt with no review step, and no human appears anywhere (d2), while the prose insists that only the data team may endorse and that endorsed assets need review before changes go live.

Suppose you removed that diamond. You would get a system that rewrites its own trust signals from usage, which is a self-reinforcing loop with no ground truth. The agent's most-used source becomes its most-endorsed source becomes its most-used source. In short, the one control that makes this design safe is the one the diagram leaves out.

This diagram is synthesized from n2, n7, n8 and d2, and the red diamond in particular is this brain's correction of fig3 rather than a shape the source drew. Having noticed that the human is missing from the picture, it is worth asking how firmly that human is attached to the system at all.

The gate that is a convention rather than a constraint#

The answer is a design smell you will meet elsewhere. Users are told that "agent responses should be treated with judgment", and that the data team "should always be looped in when questions need validation", with reminders shared in Slack. One listed workspace-guide topic is "when a question should be routed to the data team for validation", so the routing rule is written for the agent to relay, not enforced by the system (n12).

That works at 290 people with a visible data team on Slack. It is not obvious it survives 3,000, and the failure would be silent, because nobody ever sees the escalation that did not happen. Silent failure is the theme of the next part too, and it is where the evidence gets thin.

Part 7 - How would you know any of this works?#

You would run evals. This stack has none, and the authors say so [S11 §Evaluating context changes, n10]:

"Next, we want to start leveraging evals, which will help us understand whether context changes are improving agent responses. Today, we can look at usage patterns, warnings, and qualitative feedback... This will make context management feel more like software development. We can make a change, test it, and build more confidence before rolling it out broadly."

That last sentence is the article conceding S5's thesis unprompted. S5's title is literally "Don't Ship Skills Without Evals" (claim 46: an instruction artifact without an eval is an unfalsifiable change), and workspace guides are skills, by the author's own description (n5). So this stack shipped five layers of instruction artifacts to an entire company with no eval on any of them, and it appears to be working. Both things are true at once, and the honest reading is that nobody knows which layer is load-bearing.

What is measured instead is adoption, and the shape of those numbers is worth seeing (n9). There were roughly 2,200 conversations in 30 days, from about a third of the company, at 23 per user per month, and the company was fully off the old BI tool within six weeks. The figures are internally consistent, since 2,200 divided by 23 gives about 96 users, which matches a third of roughly 290 people.

Two cautions and then the real one.

The 40x should never be quoted bare (d3). Its numerator is agent conversations, while its denominator is the request volume a three-person team "could field directly", an estimate that was never measured and that back-solves to about 55 requests a month. A conversation is not a request. Four follow-ups are not four answered questions.

The second caution is smaller but points the same way, because "100% migration in six weeks" measures that people stopped opening the old tool and nothing about whether the answers they now get are right. Which is the real one, d4. Every reported figure measures adoption while the thesis is trust. The article argues that without context "the answers are harder to trust" and that "the reliability of a data agent comes from the context", and reliability is the one property never measured.

Generalise this, because you will see it constantly (claim 100): agent throughput is cheap to count and agent correctness is not, so reported agent ROI is composed almost entirely of the measurable half. Not dishonesty - a structural bias in what instrumentation makes easy. When a deployment reports only volume, read the missing correctness number as expensive, not as good.

How wrong could it be? The task does have a public benchmark (F3). Spider 2.0 is 632 enterprise text-to-SQL problems over BigQuery and Snowflake databases with 1,000+ columns, which is the same warehouse class. The best model scored 17.0% at publication, against 91.2% for the same model class on the older Spider 1.0. Public leaderboard entries have since climbed steeply, but the ordering is the signal, and the dbt-based setting, the closest analogue to this stack, is the hardest, at 65.6% among tuned, purpose-built commercial systems.

That does not say LangChain's agent is inaccurate. It says the task has a real, currently binding error rate, worst in exactly this configuration, and that 40x volume at an unknown accuracy is an open risk rather than a rhetorical caveat.

If you were going to fix that with a single experiment, the obvious candidate is the one nobody has built (F6). Ablate the endorsement flags, hold everything else fixed, and measure whether source selection degrades. Endorsement-style signals are widely shipped and independently re-derived, and yet every documented effect is on human discovery, whether that is badges, sort order or search priority. Whether a trust flag changes a model's choice is assumed by everyone and demonstrated by no one.

Part 8 - What you would build first#

Sequencing, in the order the evidence supports:

1. Fix the foundations before adding layers. "A semantic model is most useful when it sits on top of solid data modeling... If the data model is confusing to humans, it will be confusing to agents too" (n4). Context layers compose downward and cannot repair the layer beneath them. A semantic model over incoherent grains encodes the incoherence.
2. Cover the head, not the tail. "Start with the questions people ask most often... if you can cover roughly 80% of the questions people ask" (n11).
3. Generate the tail's first draft from the query log rather than deferring it entirely (F2). It costs about $0.50 per warehouse, and it reserves the humans for genuinely ambiguous columns.
4. Add trust flags only once you actually have duplicate assets, and decide the writer restriction in the same breath. Consider two tiers rather than one.
5. Do not add a store you cannot name a question for. That is the rule from Part 3, used here as a stopping condition. One study found metadata gains flattening past three or four components, and this stack runs five, so whether the fifth pays for its maintenance is unknown.
6. Instrument before you optimise. The triage table is only executable because conversations are logged, which is the same "log first" precondition S1 puts at the base of all evals (claim 1).

Step 2 deserves a second look, because a previous source in this brain says the opposite.

The head/tail rule, and why S10 says the opposite#

S10's prescription is to retrieve the long tail and pin the head (claim 90), which is the reverse of covering the head and deferring the tail. Both are right, and reconciling them gives you a rule worth more than either. S10's binding cost is tokens, and indexing the tail is nearly free once the index exists, so the tail gets retrieved while the head gets pinned, because pinning protects what must never be missed. S11's binding cost is human authorship instead. Every tail item there is a definition someone writes, reviews and maintains forever, so the tail gets deferred.

Same distribution, opposite prescription, because the scarce resource differs. The question is never "head or tail" - it is "what runs out first, context window or people".

The evidence, weighed#

Read this before citing anything above.

What rescues it is that the deep-research pass found the mechanism measured by people with no stake in it, namely a T1 benchmark, a T1 product doc, a T3 preprint, a T2 first-party study and a 1977 result. Claims 93 and 95 are the first in this brain to reach corroborated on independent external evidence rather than two internal legs.

So the note is good for several things. The five-store derivation works as a checklist for any agent over a proprietary domain. The weak-versus-strong definition is the clearest illustration here of metadata becoming a control surface. The endorsement guardrail generalises into a rule about trust signals of any kind. And the triage table is a usable operational routine. It is not good for deciding whether any of it works.

One small observation. MCP appears twice in this article, both times as an unremarked bullet in a list of ways to reach the agent, beside Slack and the CLI. There is no mechanism, no version and no design consequence. Per ADR-0012 that is a mention, not a source, and it does not advance brain/topics/mcp.md. But the manner is mildly interesting, because MCP has become boring, listed beside Slack by someone with no interest in the protocol.

Open questions#

• Does an endorsement flag actually change an agent's source selection? Nobody has measured it (F6). Documented effects are on human discovery only. The cheapest high-value experiment this brain has surfaced, and exactly the eval n10 admits is missing.
• What is the actual accuracy of this stack? Unmeasured by the source; the closest public proxy leaves roughly a third of enterprise dbt questions wrong (F3).
• How many context stores earn their maintenance? One study found metadata gains flattening past three or four components (F2, weak T4/T5 evidence). This stack has five. Ablation is the method that answers it (claim 47).
• Does the social human gate hold at scale? n12 and d2: the escalation rule is written for the agent to relay, not enforced. Fine at 290 people; the failure at 3,000 would be silent.

Feeds these topics#

• brain/topics/context-engineering.md - the primary home. Contributes the decomposition of a context layer by the question each store answers rather than by the tool holding it (n2), the duplicate test that falls out of it, the definition-as- instruction move (n3), and context as a safety property rather than a quality one, since a missing assumption produces a confident wrong number rather than an error.
• brain/topics/rag.md - the trust signal as a retrieval-time ranking input, and its saturation failure: an endorsement flag needs an access-controlled writer because "if everything is endorsed, the signal stops being useful" (n6).
• brain/topics/evals.md - the adoption-versus-trustworthiness gap (d4, n10), which is this brain's cleanest instance of a source measuring the thing that is easy to measure while claiming the thing that is not.
• brain/topics/skills.md - versioned prose documents that the author herself calls "like skills for the data agent" (n5), a non-coding instance of the family.

Presentation narrative#

A talk track for a room deciding whether to put an agent in front of its own warehouse, derived from the gated nodes above. It does not claim the approach works: the source never measures that, and what weight it carries comes from the deep-research pass.

Slide 1 - The failure mode is a correct query answering the wrong question#

An agent pointed at your warehouse will write valid SQL and return a technically correct answer based on the wrong business interpretation, which is the failure that looks exactly like success. Compare it to a coding agent, because the contrast is the whole argument. When a coding agent guesses wrong, a test fails or the process throws, and the wrongness surfaces by itself. When a data agent does not know that churned customers are normally excluded, nothing errors and nothing hedges. It returns a confident number that is wrong by exactly the size of your churn, and hands it to somebody who puts it in a board deck.

The leadership significance is that this is a safety property rather than a quality one, because there is no traceback for a wrong assumption: the usual signal you rely on to catch a bad agent is structurally absent. Before this system existed, three people held the context that would have prevented the error, and every question queued behind them [n1].

flowchart LR
    Q["'how many active customers<br/>did we add last quarter?'"]
    S["five context stores consulted"]
    R["a number for the board deck"]
    Q --> S --> R
    F["omit any one and you get a<br/><b>confident wrong number</b>,<br/>not an error"]
    S -.-> F
    style F fill:#fdeaea,stroke:#dc3545,color:#7f1d1d

This is a failure-mode slide, not a pipeline. The crux is the dashed edge: there is no traceback for a wrong assumption, so all five omissions look identical from outside, like a confident number arriving on time. That is what makes context a safety property here rather than a quality one.

Synthesized from n1, n2.

Slide 2 - It was never a plumbing problem, and the article's own figure proves it#

The article calls its migration "a big architectural shift" and then publishes an architecture diagram containing no agent, no semantic model and no feedback loop [d1]. That figure is a stock ELT pipeline: Fivetran and Airbyte and Segment into BigQuery, dbt on top, reporting at the end. Everything that makes the stack agent-first lives in a second figure and in prose.

I want to be careful about how that is read, because it sounds like a criticism and it is really the strongest evidence in the piece. The gap between the caption and the drawing is what tells you the plumbing was never the thing that changed. What changed is the reporting tier and the English written around the warehouse. For a room evaluating a similar project, the question this reframes is the budget question: you are not buying infrastructure, you are commissioning documentation, and those have very different cost curves.

This is the article's own figure, captioned by its authors as the big architectural shift. The crux is what is absent: no agent, no semantic model, no feedback loop - a stock ELT pipeline [d1]. It is the strongest evidence in the piece that the plumbing was never what changed.

Slide 3 - Five context stores, each earning its place by a failure the other four cannot explain#

The useful decomposition is by the question each store answers, not by the tool that happens to hold it [n2]. One store says what the data is, a second what a metric means, a third how the business works, a fourth which source to trust, and a fifth how a number is actually computed. Presented as a list that is five boxes and a reader cannot tell which are load-bearing. Derived as a chain it becomes inevitable: the agent picked the wrong table, so you need definitions; it then picked the right table and the wrong formula, so you need a semantic model; it then used the right formula and still answered a different question, so you need guides.

The rule that falls out is the portable part: duplicated context is worse than missing context, because two copies drift apart and nothing tells you which one the agent read.

flowchart TB
    A["wrong table"] --> S1["definitions"]
    S1 --> B["wrong formula"] --> S2["semantic model"]
    S2 --> C["different question"] --> S3["workspace guides"]
    S3 --> D["two assets look right"] --> S4["trust signal"]
    S4 --> E["still off"] --> S5["computation lineage"]
    style S5 fill:#e8f4ea,stroke:#28a745,color:#14532d

This is a derivation slide, not a taxonomy. The crux is that each store exists because of a failure the previous ones cannot explain, which is what makes five the right number and gives you a stopping condition: a layer that cannot name a question only it answers is a duplicate.

Synthesized from n2; the article presents the five as a list.

Slide 4 - A column definition stops being a description and becomes an instruction#

This is the single most transferable move in the article, and it fits on one line. account_status: The status of the account. becomes a paragraph that spells out what each lifecycle value means in business terms and then issues an imperative: "For customer reporting, filter to Active unless the analysis explicitly includes churned or prospective accounts" [n3]. That is not documentation. It is a default policy stored in metadata, positioned so the agent meets it at exactly the moment it matters.

Notice where the instruction lives: not a system prompt, not a retrieval corpus, not a wrapper, but the schema field the agent already reads to do its job. It cannot be skipped and competes with nothing for context budget. The claim is single-leg on the article's content, and it is where external evidence is strongest: schema documentation is measured to help, far more on real warehouses than on public benchmarks.

flowchart TB
    A["<code>account_status: The status of the account.</code>"]
    B["a paragraph naming each value's<br/>business meaning, then an imperative:<br/><i>'for customer reporting, filter to Active<br/>unless the analysis explicitly includes<br/>churned or prospective accounts'</i>"]
    C["a default <b>policy</b>, stored where the<br/>agent already has to look"]
    A --> B --> C
    style C fill:#e8f4ea,stroke:#28a745,color:#14532d

This is a before-and-after on one field. The crux is where the instruction lives: not a system prompt, not a retrieval corpus, but the schema field the agent already reads to do its job, so it cannot be skipped and competes with nothing for context budget.

Synthesized from n3; measured externally by F1.

Slide 5 - The loop's output is a write back to the store, never an answer to a user#

Observability over the agent's own conversations becomes the demand signal for what to document, and that inverts what the data team is for [n7, n8]. Conversations show where context is missing, the team writes the missing context, and the deliverable is a change to the store rather than a report to a person. The one guardrail is a writer restriction on the trust flag, because if everything is endorsed the signal stops being useful [n6].

The cost of this is three permanent people, and the reason is older than the technology. The model collapsed the encoding cost of expert knowledge, since the target formalism is now English prose rather than production rules. It left the elicitation cost completely untouched, because somebody still has to sit with the go-to-market team and find out what they mean by "pipeline". That is a 45-year-old diagnosis arriving on schedule, and it is why this is standing headcount rather than a project.

This is the source's own loop, and what it omits is the point. The crux is that usage trends route straight back into the context layer with no human in the path, and the human is the part that costs three salaries [n12]. The encoding cost of expert knowledge collapsed; the elicitation cost did not [F5].

Slide 6 - Adopt the mechanism, and do not believe the results, because nobody measured them#

Every number this article reports measures adoption while its thesis is about trustworthiness, which it never measures at all, and the authors concede exactly that and file evals under "next" [d4, n10]. Two thousand two hundred conversations, forty times the throughput, a hundred percent migration in six weeks: all of those describe how much the thing is used, and none of them describes whether its answers are right. The headline forty-times figure compares mismatched units and should never be quoted bare [d3].

So the verdict is pilot rather than adopt, and the split is unusually clean. The mechanism is externally corroborated by the research pass and is worth borrowing today. The result is not corroborated by anything, including the article. What would change that is the one thing the authors say they have not built: an eval that scores answers rather than counting conversations. If you run this pattern, build that first, because without it you will be measuring the same thing they did and concluding something they did not.

flowchart TB
    M["what is measured<br/>2,200 conversations, 40x throughput,<br/>100% migration in six weeks"]
    T["what is claimed<br/>trustworthiness"]
    G["nothing connects them,<br/>and the authors concede it - d4, n10"]
    M --> G
    T --> G
    style G fill:#fdeaea,stroke:#dc3545,color:#7f1d1d

This is an evidence slide, not a summary. The crux is that every number describes how much the thing is used and none describes whether its answers are right. The 40x compares mismatched units and should never be quoted bare [d3].

Synthesized from d3, d4, n9, n10.

Key takeaway message#

An agent-first data stack is a documentation layer wrapped around a stack you already have, and the article's own architecture figure proves it by containing no agent. The transferable move is that a column definition becomes an instruction, placed where the agent already looks. Decompose by the question each store answers, with the duplicate test as the stopping condition. The cost is three permanent people, because the model collapsed the encoding cost of expert knowledge and left elicitation untouched. The boundary is firm: everything reported measures adoption and correctness is measured nowhere, so pilot with an answer-quality eval attached.