The paper in four sentences
"Prompt injection" was already known: a user types something like "ignore your instructions" to make a chatbot misbehave.
The authors point out a more dangerous version: once an assistant can search the web, read emails or open documents, an attacker can put instructions inside that content and wait for the assistant to read them. They call this indirect prompt injection.
They organise the possible attacks into a map (how the injected text gets in, and what it can then do, from stealing information to spreading itself) and demonstrate many of them on real and realistic systems.
They conclude that because a language model cannot reliably tell data from instructions, retrieving untrusted text is a bit like running untrusted code, and no simple defence fixes it.
Idea · §1–2 · Fig. 1
To the model, everything is one long text
A chatbot app works by building one long piece of text for the language model to continue. It starts with the developer's hidden instructions (the system prompt), then the user's request, and, if the app has tools, the contents of whatever web page or email it fetched. The model then writes what comes next.
Here is the problem. Nothing in that text reliably marks which parts are instructions to follow and which are material to read. Language models are trained to follow instructions wherever they appear. So if a web page contains a sentence addressed to the assistant, the assistant may treat it as an order.
Below is a harmless toy. A user asks an assistant to summarise a restaurant review. The page contains a line the user cannot see, styled to be invisible, that asks the assistant to promote a particular café. Switch between what the user sees and what the model receives.
TakeawayAn LLM app mixes trusted instructions and untrusted content into one text, and the model has no reliable way to tell them apart.
Check yourself
- What makes an injection "indirect"?
- The attacker never interacts with the assistant directly; they place text in content (a page, email, document, code) that the assistant will later retrieve on someone else's behalf.
- Why can't the user spot the problem by reading the page?
- The planted text can be hidden from people (invisible styling, comments, metadata) yet still be part of the text the app passes to the model.
Idea · §3 · Fig. 2
A map of the attack surface
The paper's main contribution is a way of thinking about the problem: a taxonomy, or organised map. On one side are the ways injected text can reach the model; on the other, what the attacker can achieve once it does. Click any item.
TakeawayAnything an assistant reads is a way in, and anything it can do (search, send, remember, write code) is something an attacker can try to use.
Idea · §4
Demonstrated on real systems
This was not only a thought experiment. The authors built test apps on OpenAI's GPT models with common tools (search, reading URLs, email, an address book, memory) and showed attacks working against them. They also tested Bing Chat, Microsoft's GPT-4-powered search assistant, which can read the page open in the Edge browser, using pages on their own computer so that no real users were exposed. And they showed that planted comments in code a programmer imports could steer GitHub Copilot's suggestions.
Among the effects they demonstrated, described here only in outline:
- An assistant was turned into a persuasive "social engineer" that tried to get the user to reveal personal details, which it could then leak through innocent-looking actions such as search queries or links.
- An assistant could be told to fetch further instructions from the attacker's server, giving remote control, or to save instructions in its long-term memory so the compromise survived into later conversations.
- An email assistant that both reads and sends mail could be made to pass the injected text on to other people's assistants: a self-spreading "worm" made of words.
- Answers could be quietly distorted: wrong summaries, hidden adverts, suppressed sources, or the assistant simply refusing to help.
TakeawayThe attacks worked on deployed products, not just lab models, and the more tools an assistant had, the more an injection could do.
Idea · §5.6
Why it is so hard to defend
The obvious fixes all have holes, and the authors explain why.
- Train the model to resist. Safety training can teach a model to ignore known tricks, but attackers rephrase, and patching one phrasing at a time is a game of whack-a-mole.
- Filter the input. Look for suspicious instructions in retrieved text. But the instructions can be disguised, encoded, split across steps, or simply written as ordinary prose.
- Mark the data clearly. Wrap retrieved text in labels saying "this is data, do not obey it". Helpful, but the model is still reading natural language, and a capable model can be talked out of its labels.
- A second model as a supervisor. Another model checks the first. It is also a language model, so it can also be fooled.
The authors' conclusion is sobering: they find it hard to imagine a foolproof defence while instructions and data share one channel. Try the toy defences below on two versions of the attack.
TakeawayFiltering and training reduce the risk; limiting what a model that reads untrusted text is allowed to do is the most reliable protection.
Check yourself
- Why doesn't "ask the user before acting" solve the problem completely?
- It stops unwanted actions such as sending emails, but the injected text can still change what the assistant tells the user (false summaries, adverts, persuasion), and users approving many prompts may stop reading them.
Abstract, restated
LLMs are increasingly integrated into applications with retrieval and tools, which blurs the line between data and instructions. The authors introduce indirect prompt injection (IPI): adversaries remotely exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved, without a direct interface. They derive a security taxonomy of impacts and vulnerabilities (data theft, worming, information-ecosystem contamination and others), demonstrate practical attacks against Bing Chat (GPT-4), code-completion engines and synthetic GPT-4-based applications, and argue that processing retrieved prompts can act as arbitrary code execution, manipulating functionality and controlling how and whether other APIs are called. Current mitigations are found insufficient. Greshake, Abdelnabi, Mishra, Endres, Holz & Fritz, AISec '23 (ACM CCS workshop); arXiv 2302.12173.
Section · §1–2
Threat model: from direct to indirect injection
Direct prompt injection (the prior threat model): the attacker is the user, crafting input to override the system prompt (goal hijacking, prompt leaking, jailbreaks). The victim is mostly the provider; the user harms only their own session.
Indirect prompt injection: the attacker controls some content that the application will retrieve during inference (a web page, an email, a document, a code repository, search results) on behalf of a different user. Because instruction-tuned models follow instructions anywhere in their context, the injected text can steer the model's outputs and, crucially, its tool calls. The attacker needs no access to the victim's session and can be anonymous and at scale.
The paper's security framing (§2) is the analogy with code execution: an LLM that processes untrusted retrieved data is comparable to a computer executing untrusted code, since in an LLM-integrated application natural-language "data" can change control flow (which APIs are called, with what arguments). The classic lesson from software security, keep code and data in separate channels, is exactly what a single-context LLM cannot do.
Check yourself
- Why is the analogy with SQL injection apt, and where does it break down?
- Both mix attacker-controlled data with a command stream. SQL injection is fixed by parameterised queries that separate the channels syntactically; an LLM has no such syntactic boundary, since both channels are natural language interpreted by the same model, so there is no equivalent of escaping.
Section · §3.1–3.2 · Fig. 2
Taxonomy: injection methods and threats
Injection methods (§3.1). Passive: content placed where retrieval will find it (public web pages, search-engine-indexed content, code and documentation). Active: content delivered to the target, e.g. emails processed by an email assistant. User-driven: users are tricked into pasting injected text themselves. Hidden: multi-stage injections where a small first payload makes the model fetch a larger one, encoded payloads, and (for multimodal models) instructions in images.
Threats (§3.2). Information gathering (exfiltrating user data or chat history), fraud (phishing, scams delivered by a trusted-seeming assistant), intrusion (persistent control: remote command fetching, memory persistence, API calls, spreading), malware (spreading links or payloads, worm-like propagation), manipulated content (wrong or biased summaries, disinformation, hidden ads, suppressed sources) and availability (denial of service: refusing, slowing, or corrupting answers). Targets range from untargeted mass injection on popular pages to specific individuals.
Section · §4 · Figs. 3–27
Experimental demonstrations
Setup (§4.1). (1) Synthetic applications: LangChain agents with text-davinci-003 and a chat-format GPT-4 app, with tools for search, viewing the current page, retrieving URLs, reading and sending email, an address book and a memory store. (2) Bing Chat, a black-box GPT-4-based system, tested via the Edge sidebar on local HTML files so no public content was poisoned. (3) GitHub Copilot, with injections placed in code and documentation that the completion engine includes in its context.
Results (§4.2–4.3), at the level of mechanism:
- Information gathering (§4.2.1): the injected assistant elicits personal data from the user and leaks it through side channels the app already has, such as search queries or URLs.
- Fraud and malware (§4.2.2–4.2.3): the assistant presents attacker links as trustworthy; an email-reading-and-sending app forwards the injection to contacts, a prompt-level worm.
- Intrusion (§4.2.4): remote control by having the model periodically retrieve new instructions; persistence by writing the injection into long-term memory; code-completion poisoning via imported files.
- Manipulated content and availability (§4.2.5–4.2.6): biased or false summaries, suppressed sources, disinformation and hidden ads; making the assistant unhelpful or slow.
- Hidden injections (§4.3): multi-stage payloads where a short visible instruction pulls in a longer hidden one, and encoded payloads the model decodes itself.
A notable property the authors highlight is that the model's own capabilities amplify the attack: stronger instruction following and better tool use make injections more effective, not less. The worm scenario can be reasoned about like an epidemic:
Section · §5 · Lineage
Mitigations, limitations and later work
Mitigations discussed (§5.6). RLHF and safety training (tends toward whack-a-mole patching of specific phrasings); input filtering of retrieved content (evaded by encoding, obfuscation and multi-stage payloads, and a dilemma: a weaker filter model may not understand encoded input that the stronger main model will decode); supervisor or moderation LLMs (themselves susceptible); interpretability-based detection of anomalous behaviour (speculative). The authors conclude a foolproof defence is hard to imagine with current architectures.
Limitations (§5.2). Demonstrations are qualitative proofs of concept on specific systems and dates; there are no large-scale success rates, and black-box products change frequently. Ethics (§5.1): findings were disclosed to OpenAI and Microsoft and no public content was poisoned.
Check yourself
- Why are system-level defences considered more robust than model-level ones?
- Model-level defences reduce the probability that the model obeys injected text but give no guarantee; system-level designs bound what can happen even if the model is fully compromised, because the compromised component lacks the privileges to act.
- Why might a more capable model be more vulnerable?
- Better instruction following, decoding and tool use mean injected instructions are understood and executed more competently, including encoded or multi-step ones a weaker model would garble.
Reference
Glossary
- Prompt injection
- Input that causes an LLM to follow instructions other than those intended by the application developer or user.
- Indirect
- Delivered through retrieved content rather than typed by the attacker into the app.
- System prompt
- Developer-written instructions placed at the start of the model's context.
- Exfiltration
- Leaking data out of a system via some outbound channel.
- Source / sink
- Where untrusted data enters; where it can cause an effect.
- Least privilege
- Give each component only the capabilities its task requires.
- Worm
- A payload that copies itself to new victims without further attacker action.