source

Defending Against Indirect Prompt Injection Attacks With Spotlighting

Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, Emre Kıcıman - all Microsoft

TL;DR#

This is the defence most teams actually ship, and it costs almost nothing - S18 measures it at 1.06x input tokens against CaMeL's 2.82x. Spotlighting transforms untrusted input so its provenance is continuously visible to the model, then tells the model about the transformation (n2). Three variants, and the ordering matters: delimiting halves attack success and the authors recommend against it because an adversary who learns your system prompt writes their own delimiters (n4); datamarking - interleaving a marker token throughout the text - drops attack success from ~50% to 3.1%, and costs nothing measurable on the underlying task (n5, n6); encoding gives the best number and only works on high-capacity models, wrecking accuracy on weaker ones (n7). Two things make this worth reading beyond the recipe. The adversary section is genuinely good - assume the system prompt has leaked, therefore randomise the marker (n9) - and the discussion contains the best cross-domain framing in this brain's security set (n12). Read d3 first: every experiment is document summarization or Q&A, with no agent and no tools.

flowchart TB
    P["make the <b>provenance</b> of untrusted input<br/>continuously visible to the model,<br/>then tell the model about it - n2"]
    D["<b>delimiting</b><br/>halves attack success -<br/>and the authors recommend <b>against</b> it - n4"]
    M["<b>datamarking</b><br/>~50% -> <b>3.1%</b>, and costs nothing<br/>measurable on the task - n5, n6"]
    E["<b>encoding</b><br/>best number, and wrecks accuracy<br/>on weaker models - n7"]
    C["1.06x input tokens,<br/>against CaMeL's 2.82x"]

    P --> D
    P --> M
    P --> E
    M --> C

    style D fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    style M fill:#dcfce7,stroke:#15803d,color:#14532d

This is a pricing diagram, not a technique diagram, and the ordering of the three variants is the content. The crux is that the cheapest defence in this brain has a variant its own authors tell you not to use and a variant that is nearly free, and telling them apart is the entire value of reading the paper rather than the abstract. It is drawn with all three hanging off one idea because they are the same transformation at different strengths, and with the cost attached only to datamarking because that is the one worth deploying. Read d3 first: every experiment is document summarization or Q&A, with no agent and no tools.

Synthesized from n2, n4, n5, n6 and n7.

The 1-minute version#

This article covers a 2024 Microsoft paper proposing the cheap, prompt-level defence against indirect prompt injection. The first thing to establish is why it belongs in a set otherwise made of attacks and expensive structural defences, because on evidential quality it is the weakest source here.

The problem it addresses is the one S17 named, restated in security vocabulary by an unrelated team: the LLM cannot distinguish code from data, because it operates on "boundary-less streams of tokens" (n1). That independent restatement is itself worth something - it is Microsoft in 2024 reaching the same diagnosis as Saarland and CISPA in 2023.

Why the obvious fix fails is measured, and it is the fix nearly everyone tries first. Adding defensive instructions to the system prompt - do not obey instructions found in the document - has "almost no added benefit" on GPT-3.5-Turbo, moving attack success from about 60% to about 58% (n3). Telling a model to be careful is not a control.

The idea is to stop instructing and start transforming. If the untrusted text is altered in a way the model can perceive continuously - not just at its boundaries - then provenance becomes a property of the text rather than a claim in the prompt. That is spotlighting, and its three instantiations differ in how much of the text they touch.

How it works in ascending order of strength. Delimiting wraps the input in special tokens and halves attack success, and the authors disown it in the same paper: an adversary who has your system prompt writes the closing delimiter themselves (n4). Datamarking interleaves a marker token throughout the body - In^this^manner^Cosette^traversed^the^labyrinth - which drops attack success to 3.1% on GPT-3.5-Turbo and 0.00% on Text-003 (n5). Encoding transforms the whole input to base64, giving the lowest attack rates of all (n7).

What it costs splits cleanly by variant, and this is the practical payload. Datamarking costs nothing measurable - accuracy on SQuAD, sentiment, WiC and BoolQ is unchanged (n6). Encoding costs a great deal on any model that cannot decode fluently: GPT-4 is unaffected, GPT-3.5-Turbo falls from ~0.85 to ~0.58 on SQuAD (n7). So the recommendation is datamarking by default, encoding only with a frontier model and per-use-case validation, delimiting never (n8).

How far to trust it needs two caveats and one piece of credit. Nothing here is agentic - every experiment is summarization or Q&A with no tools and no environment state, and the only variant anyone has since tested against an actual agent is the one these authors disown (d3). And the authors do not know why it works (n11), which is precisely the property S18 objects to in the whole heuristic family. The credit is that they say both of those things themselves, and then supply the analogy that explains their own ceiling.

The narrative above is the argument. The table below is the same argument compressed for someone returning to check one row.

Key claims#

• The LLM cannot distinguish code from data, restated independently of S17 by a different team (n1).
• Telling the model to ignore injected instructions barely works - roughly 60% to 58% attack success (n3, fig3_delimiters.png). The measured failure of the thing everyone tries first.
• Delimiting halves attack success and its own authors recommend against it, because an adversary with the system prompt writes the delimiters themselves (n4).
• Datamarking drops attack success from ~50% to 3.1% (GPT-3.5-Turbo) and to 0.00% (Text-003) in summarization (n5, fig4_datamarking.png).
• And it costs nothing measurable on the underlying task across four benchmarks (n6, fig7_datamarking_no_task_cost.png). The finding that makes it deployable.
• Encoding gives the lowest attack rates and requires a frontier model - GPT-4 unaffected, GPT-3.5-Turbo's accuracy collapsing (n7, fig8_encoding_task_cost.png).
• Design against an adversary who has your system prompt: randomise the marker token and its positions, giving a 1/N^k guess (n9). A reversible encoding is exploitable - with ROT13 the attacker writes text whose ROT13 image is the attack (n10).
• The authors cannot explain why spotlighting works (n11), which is exactly what makes a security guarantee impossible.
• The telecom analogy: spotlighting is in-band signalling, and the real answer is out-of-band (n12). In-band multi-frequency stopped accidental interference and was defeated intentionally by phone phreaking; the fix was a separate channel. LLMs are worse off than early telephony, and the authors call the out-of-band analogue infeasible with current architectures - a year before S18 built one at the program level.

What you will learn, and in what order#

flowchart TB
    subgraph M1["Movement 1 - the problem, and the answer everyone tries"]
        S1["1. Why the cheap defence deserves a note"]
        S2["2. Telling the model to be careful"]
    end
    subgraph M2["Movement 2 - three transformations, priced"]
        S3["3. Delimiting, and why they disown it"]
        S4["4. Datamarking: works, and free"]
        S5["5. Encoding: best number, real cost"]
    end
    subgraph M3["Movement 3 - designing against a real adversary"]
        S6["6. Assume your system prompt has leaked"]
    end
    subgraph M4["Movement 4 - the ceiling (the payload)"]
        S7["7. The telecom analogy"]
        S8["8. Where this sits among your defences"]
    end
    M1 --> M2 --> M3 --> M4
    style M4 fill:#f8b4b4,stroke:#c1121f,stroke-width:2px

Four movements, and this is the one note in the security set where the last movement is the payload. Movement 1 is short. Movement 2 is the recipe, and a reader who wants only the practical answer can stop after section 4 with "use datamarking, randomise the token". Movement 3 is the best piece of defensive engineering in the paper and generalises past this technique. Movement 4 is why this source earns its place in a brain that already holds two stronger defences: section 7 is the authors explaining the limit of their own method through fifty-year-old telephony, and section 8 is where this note places all four defences you now hold on one axis.

Movement 1 - the problem, and the answer everyone tries#

flowchart TB
    C["1. the cheap defence deserves a note<br/>because it is the one most teams<br/>actually ship - 1.06x tokens"]
    T["2. and the first thing everyone tries is<br/>telling the model to be careful"]
    R["which helps, and offers<br/>no guarantee at all"]

    C --> T --> R

    style R fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d

This is a framing diagram, not a design. The crux is that this note exists because of deployment reality rather than strength: it is the weakest family of defence in this brain and the one running in the most production systems. It is drawn ending on the absence of a guarantee because that is the honest frame for everything after it - the variants differ in how much they help, and none of them converts into a bound. Section 2 closes off defensive instructions specifically, which is what makes the transformations in Movement 2 read as engineering rather than as more of the same.

Synthesized from n1 and n3.

1. Why the cheap defence deserves a note#

On evidential quality this is the weakest source in this brain's security set - a vendor preprint with no venue, no code and no dataset. It is here for three reasons.

It is what teams actually deploy. Delimiters and marked-up untrusted blocks are in production prompt templates everywhere, usually without anyone having read a measurement of whether they work.

It is the cost anchor. S18 measures spotlighting at 1.06x input tokens and 0.98x output against CaMeL's 2.82x. Any argument about whether structural defence is worth its price is really an argument against this baseline.

And it names its own ceiling better than its critics do, which is section 7.

The problem statement is worth pausing on because of who wrote it. In security vocabulary, the LLM "is not able to distinguish code from data", where code means the system instructions you wrote and data means anything you did not control, and this is "a structural limitation of LLMs since they operate on boundary-less streams of tokens" (n1).

That is S17's central claim, reached independently. S17 was Saarland and CISPA in early 2023; this is Microsoft in early 2024, no author overlap. Two unrelated groups converging on processing data is equivalent to executing code is worth more than either statement alone.

So the diagnosis is agreed. The question is what to do about it, and the first answer everyone reaches for turns out to be measurable.

2. Telling the model to be careful#

Before any technique, the paper measures the thing that costs nothing and requires no thought: add a line to the system prompt telling the model not to obey instructions found in the document.

What it teaches: three bars for GPT-3.5-Turbo. Baseline attack success sits at about 0.60. Adding defensive instructions moves it to about 0.58. Adding delimiters on top brings it to about 0.30. Corroborated by: §4.2 and §5.1, p3-4 (n3, n4).

Read the first two bars against each other and the lesson is blunt. Instructing the model not to be injected is very nearly a no-op, and the paper's phrasing is that the addition "has almost no added benefit" for GPT-3.5-Turbo (n3). Text-003 does better and still leaves "a significantly high ASR remaining".

The reason is worth stating because it explains the whole design that follows. A defensive instruction is another instruction in the same undifferentiated stream - it competes with the injected one on equal terms, and there is no mechanism making yours authoritative. You cannot fix a channel problem with more traffic on the channel.

Which is why the third bar is different in kind. It is not another instruction; it is a change to the input itself.

Movement 2 - three transformations, priced#

flowchart TB
    Q{"how visible can you make<br/>provenance, and at what cost?"}
    D["3. <b>delimiting</b>: halves attack success,<br/>and the authors recommend against it -<br/>an adversary who learns your system<br/>prompt writes their own delimiters - n4"]
    M["4. <b>datamarking</b>: interleave a marker<br/>throughout. ~50% -> 3.1%, and no<br/>measurable task cost - n5, n6"]
    E["5. <b>encoding</b>: the best number, and it<br/>wrecks accuracy on weaker models - n7"]

    Q --> D
    Q --> M
    Q --> E

    style D fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    style M fill:#dcfce7,stroke:#15803d,color:#14532d

This is a comparison diagram, and the ordering matters more than any single number. The crux is that the strongest-looking variant and the disowned one bracket the one you should actually deploy, so a reader who takes only the headline figure will pick wrong twice. It is drawn as one question with three answers because they are the same transformation at three strengths rather than three techniques. Datamarking is the recommendation: it does the most per unit of cost, and its cost is close enough to zero that the usual security-versus-utility argument does not arise.

Synthesized from n4, n5, n6 and n7.

3. Delimiting, and why they disown it#

The first transformation is the obvious one: wrap the untrusted input in special tokens and tell the model never to obey instructions between them.

It works, roughly halving attack success from 60% to 30% (n4). And in §5.4 the authors recommend against using it, which is unusual enough to be worth reading closely.

The adversary model they adopt is the right one: assume your entire system prompt has leaked. Under that assumption delimiting collapses immediately, because the attacker knows which tokens close your block. They write the closing delimiter, then their instructions, and their text is now outside the region you told the model to distrust. The paper's own words: "we do not recommend using delimiting in practice, but include it here for comparisons".

Note what makes this failure structural rather than a matter of choosing better tokens. The defence marks boundaries, so it depends on the attacker not being able to forge a boundary. Anything that lives only at the edges of untrusted text has that weakness.

The fix follows from naming the weakness: stop marking the edges.

4. Datamarking: the one that works, and is free#

Datamarking interleaves a marker token throughout the body of the text rather than at its ends. Replace every whitespace with a signifier, so "In this manner Cosette traversed the labyrinth of" becomes In^this^manner^Cosette^traversed^the^labyrinth^of, and tell the model in the system prompt that this transformation has occurred.

What it teaches: baseline attack success of roughly 0.49 on Turbo 3.5 and 0.42 on Text-003 falls to about 0.03 and 0.00 with datamarking. Corroborated by: §5.1, p4 (n5).

The improvement over delimiting is large and the reason is the one section 3 set up. A forged boundary buys the attacker nothing when provenance is a property of every token. For the injected text to look like trusted input it would have to be marked the same way throughout, which requires knowing the marker.

That leaves the question any defence has to answer: what does it cost the thing you were trying to do?

What it teaches: four benchmark pairs, and the marked bars track the baseline bars within noise on all four. Corroborated by: §5.2, p5 (n6).

Nothing measurable. The model reads mangled text as competently as clean text, which is surprising the first time you see it and is the finding that makes datamarking deployable. The paper's framing is that datamarking "provides the model with an adequate cue so that it can distinguish blocks of text, while also not obscuring the text in any impactful way".

So one technique halves attacks and is disowned, and the next one nearly eliminates them for free. The third is where the trade-off returns.

5. Encoding: the best number, and the real cost#

Encoding transforms the input with a standard algorithm - base64 in the paper's example - so the untrusted block is not natural language at all. It produces the lowest attack rates of the three: 0.0% on summarization and 1.8% on Q&A (n7).

Then the cost arrives, and it is not uniform across models.

What it teaches: two panels. GPT-4 shows encoded and unencoded accuracy nearly identical across SQuAD, Sentiment, WiC and BoolQ. GPT-3.5-Turbo shows a collapse - SQuAD from ~0.85 to ~0.58, Sentiment from ~0.92 to ~0.50. Corroborated by: §5.2, p5 (n7).

The mechanism is exactly what it looks like: the model has to decode before it can reason, and only a high-capacity model decodes fluently enough that reasoning survives. The authors are direct - encoding "should only be used with the highest-capacity models" and task performance "should be validated in a use case specific way".

Which yields a recommendation with an unusual shape, and it is the practical takeaway (n8). Datamarking by default, because it is nearly free and large. Encoding only on a frontier model, with per-use-case validation. Delimiting not at all. Note that the strongest technique is the one with a capability prerequisite - your defence budget here is denominated in model quality, not tokens.

Three techniques, priced. The section that makes the paper worth keeping is what happens when you assume the attacker has read all of this.

Movement 3 - designing against a real adversary#

flowchart TB
    A["assume your system prompt<br/><b>has leaked</b>"]
    D["then a fixed delimiter is<br/>known to the attacker"]
    R["so <b>randomise the marker</b> - n9"]
    G["which is the best piece of defensive<br/>engineering in the paper, and it<br/>generalises past this technique"]

    A --> D --> R --> G

    style A fill:#e8f0fc,stroke:#4338ca,color:#312e81
    style G fill:#dcfce7,stroke:#15803d,color:#14532d

This is a threat-modelling diagram, not a parameter choice. The crux is the assumption in the first box: designing as though the system prompt is already public converts a defence that works until somebody looks into one that works after they have. It is drawn as a short forced chain because the randomisation is not a clever idea in itself - it is the only thing that survives the assumption, and the assumption is the contribution. This is a movement of one section and it earns the space, since the same move applies to any defence with a secret in it.

Synthesized from n9.

6. Assume your system prompt has leaked#

The adversary analysis in §5.4 is the best engineering in the paper, and both of its findings generalise past spotlighting.

First, static datamarking has a hole, and the authors find it themselves. If the marker is interleaved by replacing whitespace, then "it is easy to imagine an attack string which contains no spaces. This attack would then not be interleaved with the marking token at all" (n9). An unmarked payload sitting inside marked text is precisely the signal the defence was built to prevent.

The fix is to stop making the marker predictable. Choose it dynamically - a random k-gram from a character set of size N, regenerated per invocation, with the system prompt updated to match - and randomise the interleaving positions too rather than tying them to whitespace. An adversary then has a 1/N^k chance of guessing, and a leaked system prompt is stale the moment it leaks.

The transferable move is worth naming, because this brain has seen it before. The defence stops depending on a secret the attacker might learn and starts depending on a value that changes faster than they can learn it. S3's OAuth material carries the same shape: make the untrusted leg carry only material that is useless by the time anyone steals it.

Second, a reversible encoding is an own goal, and the ROT13 example is the cleanest illustration of a general trap (n10). If the attacker knows you apply ROT13 before showing text to the model, they do not need to defeat the encoding - they write text whose ROT13 image is the attack, and your own defensive transformation renders it into plaintext for them. The paper's phrasing is that "our system would then (inadvertently) transform it into the plaintext attack".

So the requirement is a transformation the attacker cannot control, which is why base64 with dynamic elements is offered rather than a simple substitution cipher. Generalised: a defensive transformation must be one-way with respect to the attacker's ability to choose its input.

Both fixes are good, and both are still fixes within an approach whose limits the authors then describe better than anyone else in this brain's security set.

Movement 4 - the ceiling#

flowchart TB
    T["7. the authors explain their own limit<br/>through fifty-year-old telephony:<br/>this is an <b>in-band</b> signal - n12"]
    I["a marker travelling in the same channel<br/>as the data can, in principle,<br/>be forged by the data"]
    S["8. so it sits in the <b>behavioural</b> class,<br/>below structural defences and above<br/>detection - this brain's taxonomy"]

    T --> I --> S

    style I fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    style S fill:#e8f0fc,stroke:#4338ca,color:#312e81

This is a limits diagram, and its provenance is unusual. The crux is that the authors supply the ceiling on their own method, and they do it by naming a fifty-year-old failure mode from telephony rather than by hedging. It is drawn ending in the taxonomy because that is where this note places the finding: in-band signalling is exactly what separates a behavioural defence from a structural one, and the classification is this brain's synthesis rather than any source's. This is the movement that earns the note its place beside two stronger defences.

Synthesized from n12; the taxonomy in section 8 is this brain's.

7. The telecom analogy#

Section 6 of the paper opens with an admission and then earns it back.

The admission: "While the data presented here seem to indicate that this intuition is correct, we lack a clear understanding of why spotlighting actually helps" (n11). A defence that works for unexplained reasons cannot offer a guarantee, which is precisely S18's objection to the whole heuristic family.

Then comes the analogy (n12).

Background, supplied. Early telephone networks were single-channel: the control signals that routed a call and the voice data of the conversation travelled over the same wire, in the same frequency space. In-band multi-frequency signalling was the first improvement - control tones were placed at frequencies that human speech rarely produced, which stopped conversations from accidentally triggering call control. It did not stop anyone doing it on purpose: phone phreaking was exactly that, generating the control tones by hand to seize free long-distance calls. The eventual fix was out-of-band signalling (Signalling System 6 and 7), where control information travels on a physically separate channel that carries no voice at all. This block is background I am supplying and is uncited by construction.

The authors then map it, and the mapping is unflattering to their own field. The LLM situation is worse than early telephony. In telecom, control and data at least occupied different frequencies within the shared medium. In an LLM, "all tokens are treated roughly equally by the model with no ability to distinguish disparate blocks of text" - as if, in their words, control tones "were transmitted in a frequency space that overlapped with typical frequencies of human voices".

Spotlighting is therefore in-band signalling. Datamarking and encoding push untrusted tokens "into a different region of representation space, thus reducing interference", which is exactly what multi-frequency did. And the authors draw the consequence against themselves: it "helps to create separation but is not perfectly secure against intentional interference".

The conclusion they reach is the one worth carrying out of this note: what is needed is an out-of-band analogue for LLMs, where control tokens are passed on a separate channel the model treats differently by construction. And they say it is not available: "With current architectures of common language models, however, this is not feasible in any straightforward way."

Read that sentence against S18, published a year later. CaMeL does not change the model architecture either - S18 agrees that is not available. What it does is move the separation up a level, into a program: the trusted control flow lives in code the planner wrote before any untrusted byte existed, and untrusted data enters as a typed value that can never become an instruction. That is an out-of-band analogue built where one was actually feasible, and this paper named the requirement before anyone met it. Neither paper cites the other. (The connection is this brain's.)

8. Where this sits among your defences#

flowchart TB
    D["<b>detection</b><br/>classify the input<br/><i>fails: weak-signal payloads<br/>carry no anomaly</i>"]
    B["<b>behavioural</b><br/>mark provenance, ask the model<br/>to honour it<br/><i>fails: in-band, no guarantee</i>"]
    S["<b>structural</b><br/>constrain what a value may do<br/><i>fails: bounded by the 17%, and by<br/>what has no data-flow consequence</i>"]
    A["the axis is <b>what the defence<br/>asks of the model</b>"]

    A --> D
    A --> B
    A --> S

    classDef weak fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
    class D weak
    style B fill:#fbf1dc,stroke:#b45309,color:#78350f
    style S fill:#dcfce7,stroke:#15803d,color:#14532d

This is a placement diagram, not a ranking, and the axis is the content. The crux is that the three classes are distinguished by how much they need the model to cooperate: detection asks it to be correct about inputs, behavioural asks it to honour a convention, and structural asks it for nothing. It is drawn as one axis fanning into three rather than as a ladder because the failure modes are not degrees of the same weakness - each class fails for its own reason, and knowing which reason applies is what tells you whether stacking two of them helps. This taxonomy is this brain's synthesis and none of the four sources draws it.

Synthesized across S18, S19, S20 and this source. The axis is this brain's.

With four defensive sources now in hand, they sort onto one axis, and the axis is what the defence asks of the model. (This taxonomy is this brain's synthesis; none of the four sources draws it.)

Three readings follow.

The classes fail for unrelated reasons, so they compose. S18 says CaMeL "can and should be used in conjunction with other defenses to deliver defense in depth", and at 1.06x tokens there is no budgetary argument against adding datamarking underneath a structural defence.

But do not mistake composition for coverage. All three classes act on the input path, and S19's whole subject is the write path into persistent memory - where a payload sits, unmarked and undetected, until a later session retrieves it as trusted knowledge. Datamarking marks untrusted text while it is untrusted; nothing here survives a memory write.

And the agentic gap is specific enough to state as a fact (d3). Every experiment in this paper is document summarization or Q&A - no tools, no environment state, no multi-step planning. The one variant anyone has since tested against a real agent is delimiting, which S20 evaluated "following Hines et al." and found only modestly effective - and which these authors explicitly recommend against. Datamarking and encoding, the two techniques actually recommended here, have never been evaluated against a tool-calling agent by anyone in this brain's evidence. For a defence this widely deployed, that is a strange hole.

Diagram (mental model)#

flowchart LR
    SYS["System prompt<br/>+ description of<br/>the transformation"] --> CTX["Context window"]
    UNTRUSTED[("Untrusted input<br/>document, page, tool output")] --> T{"Spotlighting<br/>transformation"}
    T -->|"delimiting<br/>marks edges only"| W1["&lt;&lt; text &gt;&gt;"]
    T -->|"datamarking<br/>marks every token"| W2["In^this^manner^Cosette"]
    T -->|"encoding<br/>changes the alphabet"| W3["SW4gdGhpcyBtYW5uZXI="]
    W1 --> CTX
    W2 --> CTX
    W3 --> CTX
    CTX --> M{"Model<br/>decides what to honour"}
    M --> OUT["Output"]
    style M fill:#f8b4b4,stroke:#c1121f,stroke-width:3px
    style UNTRUSTED fill:#fff3cd,stroke:#856404

Read it left to right. Yellow is untrusted input, and the three branches are the three variants in ascending order of how much of the text they touch. The red diamond is where the security decision is taken.

The crux is that the decision still happens inside the model: spotlighting changes what the model sees, never what it is permitted to do - which is why its own authors call it in-band signalling.

The shape is worth comparing directly with S18's diagram, where the equivalent red box sits outside the model, at the policy check on a tool call. That is the whole difference between the behavioural and structural classes, and it is visible as a position in the drawing rather than as a claim about efficacy. A defence whose decision point is inside the untrusted component can be argued with; a defence whose decision point is outside it cannot.

Note also what the three branches share. All of them widen the gap between trusted and untrusted tokens in representation space, and none of them creates a boundary. Delimiting comes closest to drawing one and is exactly the variant that fails, because a boundary an attacker can forge is not a boundary. That is the telecom analogy stated as a picture: more frequency separation, still one channel.

Provenance: synthesized from n2, n4, n5, n7, n12. The paper gives worked system prompts for each variant and draws no architecture diagram at all.

💡 Terms#

What to distrust in this note#

The source is a vendor preprint with no venue, no code and no dataset (d1). All six authors are Microsoft, and the related work cites the compromise of Bing Chat as motivation - so this is the vendor of a compromised product publishing its mitigation. Note the timing too: S17 found that Bing Chat filtered its chat channel and not its retrieval channel, thirteen months before this. The mechanism may be sound and the evidential class is S12's, not S20's.

The headline is a best-case composite (d2). ">50% to below 2%" takes the best technique on the best model for the best task. Datamarking on GPT-3.5-Turbo Q&A is 8.0%, and delimiting - the variant most teams actually implement - is roughly 30%.

Nothing here is agentic, and this is the most important limitation (d3). Every experiment is document summarization or Q&A: no tools, no environment state, no planning. The two recommended techniques have never been evaluated against a tool-calling agent in any source this brain holds.

The authors cannot explain why it works (n11), and say so. That is honest and it is also disqualifying for anyone wanting a guarantee, which is S18's objection to the entire heuristic family - an objection these authors independently reach through the telecom analogy.

The three-class taxonomy in section 8 is this brain's synthesis, not the paper's. None of the four defensive sources draws it, and the placement of each defence in it is my reading.

Open questions#

• Does datamarking survive in an agentic setting? (d3) The single most obvious experiment this source implies, and nobody in this brain's evidence has run it. S20 provides the environment and evaluated only the disowned variant.
• Does datamarking compose with a structural defence, and at what cost? At 1.06x tokens there is no budget argument against layering it under CaMeL, and no measurement of whether the combination helps.
• Why does it work? (n11) The authors' own open question. An interpretability answer would turn a heuristic into something that could carry a bound.
• Does the dynamic marking token defeat an adaptive attacker? (n9) The 1/N^k argument assumes the attacker must guess the token. An attacker who can observe any output influenced by marked text may be able to infer it, and no adaptive evaluation was run.
• What is the out-of-band analogue at the model level? (n12) The authors name the requirement and call it infeasible. S18 met it at the program level a year later; whether an architecture could meet it at the token level - a genuinely separate control channel - remains open, and it is the most interesting question this source raises.

Feeds these topics#

• brain/topics/agent-security.md - the cheap defence the topic was missing, its measured effect and cost, and the in-band/out-of-band framing that places every defence here on one axis.
• brain/topics/context-engineering.md - provenance as a property of the tokens themselves rather than of the prompt structure. This note has held "which tokens reach the model" as the question; here the answer is that how they are written carries security meaning independent of what they say.

Presentation narrative#

A talk track for a team that has to ship a prompt-injection defence this quarter, derived entirely from the gated nodes above. One boundary governs every number here and is on the last slide: every experiment is document summarization or Q&A, with no agent and no tools.

Slide 1 - This is the defence most teams actually ship, and it costs almost nothing#

Spotlighting runs at 1.06x input tokens, against CaMeL's 2.82x. That price difference is why it is worth a serious look even though it is the weakest family of defence in this brain.

The idea is one sentence: transform untrusted input so its provenance is continuously visible to the model, then tell the model about the transformation [n2]. It is not asking the model to detect anything. It is making the boundary between your instructions and somebody else's data impossible to lose track of mid-document.

This is the baseline and the obvious answer. The crux is that telling the model to be careful helps and guarantees nothing - which is the honest frame for everything that follows [n3].

Slide 2 - The variant with the best-known name is the one the authors tell you not to use#

Delimiting halves attack success, and the authors recommend against it [n4]. That is unusual enough to be worth pausing on, and the reason is a threat-model argument rather than a performance one.

An adversary who learns your system prompt writes their own delimiters. Since delimiting depends on a token the attacker must not know, it works exactly until somebody looks - and system prompts leak routinely. So a defence measured as effective in the lab has a failure condition that is a matter of time rather than of probability.

flowchart LR
    D["a fixed delimiter"] --> L["the system prompt leaks"]
    L --> F["the attacker writes<br/>their own delimiters"]
    style F fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d

This is a failure-condition slide. The crux is that the failure is not probabilistic but scheduled - it happens whenever the prompt leaks, which is a when rather than an if [n4].

Slide 3 - Datamarking is the one to deploy, and it is close to free#

Interleaving a marker token throughout the untrusted text drops attack success from roughly 50% to 3.1%, and costs nothing measurable on the underlying task [n5, n6]. That is the recommendation, and the second half is what makes it easy to sell internally.

Most security decisions are a trade against utility, and this one substantially is not. The task accuracy comparison holds across SQuAD Q&A, IMDB sentiment and two SuperGLUE benchmarks, which is enough breadth to believe the null result rather than to suspect the benchmark was chosen.

This is the cost side, and the finding is the absence of a gap. The crux is that the bars are the same height - which is what turns a security measure into an easy decision [n6].

Slide 4 - Encoding gives the best number and only works on strong models#

It produces the best attack-success figure and wrecks accuracy on weaker models [n7]. So the strongest-looking variant is model-dependent in a way that matters if you run a mixed fleet or expect to downgrade for cost.

The general shape is worth naming for a room evaluating any defence. A technique that consumes model capability to work will look excellent on your frontier model and quietly fail on the cheaper one you switch to next quarter. Datamarking does not have that property, which is a second reason to prefer it over the better headline.

This is the model dependence, in two rows. The crux is the difference between the top and bottom panels - the same defence, two models, opposite verdicts [n7].

Slide 5 - Assume the system prompt has leaked, and randomise the marker#

This is the best piece of defensive engineering in the paper and it generalises well past this technique [n9]. Design as though your system prompt is already public, and a fixed marker becomes obviously wrong while a randomised one survives.

What engineers should take from this is the assumption rather than the parameter. Any defence with a secret in it inherits the question of what happens when the secret is out, and most defences answer it by not asking. The authors ask it explicitly and their recommendation falls out in one step.

This is the defence working. The crux is that these numbers assume a randomised marker - the same mechanism with a fixed one is measuring an adversary who has not looked yet [n5, n9].

Slide 6 - The authors name their own ceiling, and it decides where this sits#

Spotlighting is an in-band signal, and the authors explain the limit through fifty-year-old telephony [n12]. A marker travelling in the same channel as the data can in principle be forged by the data, which is precisely the failure that took telephone signalling out of band decades ago.

That places it. Across the four defensive sources this brain now holds, the axis is what the defence asks of the model. Detection asks the model to be correct about inputs and fails on weak-signal payloads. Behavioural, which is this one, asks the model to honour a convention and offers no guarantee. Structural asks the model for nothing and is bounded instead by the 17% of tasks whose own tools suffice for the attack. That taxonomy is this brain's synthesis; none of the four sources draws it.

So the decision is: deploy datamarking now, because it is nearly free and it is a real reduction, and do not let it occupy the slot a structural defence should have. And read every figure here knowing that no experiment involved an agent or a tool call [d3], which is exactly the setting the numbers would be used to justify.

flowchart TB
    A["what does the defence<br/>ask of the model?"]
    D["detection: be correct<br/>about inputs"]
    B["behavioural: honour<br/>a convention"]
    S["structural: nothing"]
    A --> D
    A --> B
    A --> S
    style B fill:#fbf1dc,stroke:#b45309,color:#78350f
    style S fill:#dcfce7,stroke:#15803d,color:#14532d

This is where spotlighting sits. The crux is that the amber class is the one that depends on model compliance, which is why it is worth deploying and not worth relying on [n12].

Key takeaway message#

Spotlighting makes the provenance of untrusted input continuously visible and then tells the model about it, at 1.06x input tokens against CaMeL's 2.82x. Of its three variants, delimiting is disowned by its own authors because an attacker who learns your system prompt writes their own delimiters, encoding has the best number and wrecks weaker models, and datamarking drops attack success from roughly 50% to 3.1% at no measurable task cost. Assume the system prompt has leaked and randomise the marker, which is the paper's best engineering and generalises to any defence holding a secret. The authors name their own ceiling through fifty-year-old telephony: this is an in-band signal, so it asks the model to honour a convention and offers no guarantee. Deploy it, and do not let it occupy the slot a structural defence should have.