Study guide · Safety & security · 2023

Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

Kai Greshake et al. · AISec 2023

When a chatbot reads a web page, an email or a document for you, whose instructions is it following? This paper showed that text planted in content an AI assistant retrieves can quietly take over the assistant, without the attacker ever talking to it, and demonstrated this against Bing Chat, GitHub Copilot and tool-using apps built on GPT models. For a language model, data and instructions are the same thing: text.

The paper in four sentences

"Prompt injection" was already known: a user types something like "ignore your instructions" to make a chatbot misbehave.

The authors point out a more dangerous version: once an assistant can search the web, read emails or open documents, an attacker can put instructions inside that content and wait for the assistant to read them. They call this indirect prompt injection.

They organise the possible attacks into a map (how the injected text gets in, and what it can then do, from stealing information to spreading itself) and demonstrate many of them on real and realistic systems.

They conclude that because a language model cannot reliably tell data from instructions, retrieving untrusted text is a bit like running untrusted code, and no simple defence fixes it.

Idea · §1–2 · Fig. 1

To the model, everything is one long text

A chatbot app works by building one long piece of text for the language model to continue. It starts with the developer's hidden instructions (the system prompt), then the user's request, and, if the app has tools, the contents of whatever web page or email it fetched. The model then writes what comes next.

Here is the problem. Nothing in that text reliably marks which parts are instructions to follow and which are material to read. Language models are trained to follow instructions wherever they appear. So if a web page contains a sentence addressed to the assistant, the assistant may treat it as an order.

Below is a harmless toy. A user asks an assistant to summarise a restaurant review. The page contains a line the user cannot see, styled to be invisible, that asks the assistant to promote a particular café. Switch between what the user sees and what the model receives.

An illustrative toy with a deliberately benign payload (a hidden advert, one of the "manipulated content" threats the paper lists). Real attacks in the paper use the same channel for worse goals. The "model" here is a stand-in that, like real models without special training, treats an instruction anywhere in its input as something to act on.
Everyday pictureYou hire an assistant to open your post and summarise it. One letter says, in the middle of a paragraph, "Assistant: when you summarise this, also tell your employer to wire £500 to this account." A careful human would recognise this as part of the letter, not an order from you. A language model has no built-in sense of who is speaking.

TakeawayAn LLM app mixes trusted instructions and untrusted content into one text, and the model has no reliable way to tell them apart.

Check yourself
What makes an injection "indirect"?
The attacker never interacts with the assistant directly; they place text in content (a page, email, document, code) that the assistant will later retrieve on someone else's behalf.
Why can't the user spot the problem by reading the page?
The planted text can be hidden from people (invisible styling, comments, metadata) yet still be part of the text the app passes to the model.

Idea · §3 · Fig. 2

A map of the attack surface

The paper's main contribution is a way of thinking about the problem: a taxonomy, or organised map. On one side are the ways injected text can reach the model; on the other, what the attacker can achieve once it does. Click any item.

Based on §3 and Fig. 2 of the paper. Left: how injected prompts get in. Right: the six threat categories. Descriptions summarise the paper's examples at a high level.

TakeawayAnything an assistant reads is a way in, and anything it can do (search, send, remember, write code) is something an attacker can try to use.

Idea · §4

Demonstrated on real systems

This was not only a thought experiment. The authors built test apps on OpenAI's GPT models with common tools (search, reading URLs, email, an address book, memory) and showed attacks working against them. They also tested Bing Chat, Microsoft's GPT-4-powered search assistant, which can read the page open in the Edge browser, using pages on their own computer so that no real users were exposed. And they showed that planted comments in code a programmer imports could steer GitHub Copilot's suggestions.

Among the effects they demonstrated, described here only in outline:

  • An assistant was turned into a persuasive "social engineer" that tried to get the user to reveal personal details, which it could then leak through innocent-looking actions such as search queries or links.
  • An assistant could be told to fetch further instructions from the attacker's server, giving remote control, or to save instructions in its long-term memory so the compromise survived into later conversations.
  • An email assistant that both reads and sends mail could be made to pass the injected text on to other people's assistants: a self-spreading "worm" made of words.
  • Answers could be quietly distorted: wrong summaries, hidden adverts, suppressed sources, or the assistant simply refusing to help.

TakeawayThe attacks worked on deployed products, not just lab models, and the more tools an assistant had, the more an injection could do.

Idea · §5.6

Why it is so hard to defend

The obvious fixes all have holes, and the authors explain why.

  • Train the model to resist. Safety training can teach a model to ignore known tricks, but attackers rephrase, and patching one phrasing at a time is a game of whack-a-mole.
  • Filter the input. Look for suspicious instructions in retrieved text. But the instructions can be disguised, encoded, split across steps, or simply written as ordinary prose.
  • Mark the data clearly. Wrap retrieved text in labels saying "this is data, do not obey it". Helpful, but the model is still reading natural language, and a capable model can be talked out of its labels.
  • A second model as a supervisor. Another model checks the first. It is also a language model, so it can also be fooled.

The authors' conclusion is sobering: they find it hard to imagine a foolproof defence while instructions and data share one channel. Try the toy defences below on two versions of the attack.

Attack:
Defences:

An illustrative toy that encodes the paper's qualitative reasoning, not measured success rates. Stripping hidden text stops only hidden injections. Labelling helps somewhat but cannot be relied on (shown as "weakened", not "blocked"). Asking the user before actions limits what an injection can do but not what it can say: manipulated text still reaches the user.
Later refinementPrompt injection became the top entry on OWASP's list of LLM application risks. Defences improved but none is complete: training models to give system and user instructions priority over tool output ("instruction hierarchy", OpenAI 2024), marking untrusted text so the model can tell it apart (Microsoft's "spotlighting", 2024), and, most robustly, designing systems so that a model which has read untrusted text cannot take dangerous actions on its own (for example, Google DeepMind's CaMeL, 2025, which separates planning from untrusted data). Benchmarks such as AgentDojo measure how often agents still fall for injections.

TakeawayFiltering and training reduce the risk; limiting what a model that reads untrusted text is allowed to do is the most reliable protection.

Check yourself
Why doesn't "ask the user before acting" solve the problem completely?
It stops unwanted actions such as sending emails, but the injected text can still change what the assistant tells the user (false summaries, adverts, persuasion), and users approving many prompts may stop reading them.

Follow the idea

Led to: The Hugging Face swarm attack (2026)