The paper in four sentences
Chat models such as ChatGPT and LLaMA-2-Chat are "aligned": trained to decline requests for harmful content. People had found manual tricks to get round this, but those were fiddly and quickly patched.
The authors automate it: they search for a short string of tokens (a suffix) that, added after a request, makes the model start its reply with an agreeable opening like "Sure, here is", after which it tends to carry on.
The search uses the model's own gradients (a signal of which way to change things) to shortlist promising token swaps, then tests them; optimising one suffix across many requests and several models makes it general.
Suffixes found on small open models also worked, at varying rates, on GPT-3.5, GPT-4, PaLM-2 and Claude-1, though Claude-2 was barely affected, showing that refusal training was brittle in ways nobody had a fix for.
Idea · §1
Adversarial examples, now for text
A decade earlier, researchers showed that image classifiers can be fooled by adversarial examples: tiny, carefully computed changes to a picture, invisible to people, that make the network confidently wrong. The recipe is to use the network's own gradients to find the change that most increases its error.
Text is harder. An image's pixels can be nudged by tiny amounts; words cannot. You can only swap one token (a word or word-piece) for another, and there are tens of thousands to choose from at every position. Earlier automatic attacks on language models mostly failed against chat models trained to refuse.
This paper's contribution is a search method that works well enough in this discrete setting to break aligned chat models reliably, with no human creativity needed.
TakeawayGCG brings the adversarial-example idea from images to chat models: let the model's own internals guide a search for inputs that defeat its training.
Idea · §2.1
Aim for the first words of the reply
What should the search aim for? "Make the model say something harmful" is too vague to measure. The authors use a simple proxy: make the model's reply begin with an affirmative phrase that restates the request, such as "Sure, here is …". Language models continue in the style they start in, so once a model has committed to "Sure, here is", it usually carries on rather than reversing into a refusal.
This turns the goal into a number the search can improve: how likely the model is to produce those exact opening words.
TakeawayThe attack targets the opening of the response, because a model that starts by agreeing tends to keep going.
Check yourself
- Why not simply aim for the single word "Sure"?
- A single agreeable token can still be followed by a refusal. Targeting a longer opening that repeats the request commits the model more firmly to answering.
Idea · §2.2
Search one token at a time, guided by gradients
The method is called Greedy Coordinate Gradient (GCG). "Coordinate" means it changes one position of the suffix at a time; "greedy" means it keeps whichever change helps most right now; "gradient" means it uses the model's gradients to decide which changes are worth testing.
Try a miniature version below. This is a made-up scoring function, not a language model: five slots, eight nonsense syllables, and a hidden score that depends on which syllables sit where and next to what. Click slots to change them by hand and try to push the score over the line. Then let the greedy search do it.
TakeawayGCG is a patient, guided trial-and-error: shortlist promising single-token swaps using gradients, test them for real, keep the best, repeat.
Check yourself
- Why does the search need gradients at all, if it tests candidates for real anyway?
- There are far too many possible swaps to test them all each step. Gradients cheaply estimate which swaps are likely to help, so only a small shortlist needs real evaluation.
- Why can a greedy search get stuck?
- Sometimes no single swap improves the score even though changing two slots together would; the toy's pair bonuses can create such local optima.
Idea · §2.3 · §3.2 · Table 2
Universal and transferable
Two further steps made the attack alarming. First, universal: rather than one suffix per request, the authors optimised a single suffix to work across many different harmful requests at once. Second, transferable: they optimised it against several open models together (Vicuna and Guanaco, small chat models built on LLaMA) and then simply pasted it, unchanged, after requests to closed commercial models they had no internal access to.
Pick an attack setting to see the share of 388 test behaviours that each closed model complied with.
Why would a suffix found on a small open model work on GPT-4? The paper notes one likely factor: Vicuna was itself trained on conversations with ChatGPT, so it may share some of its quirks. More generally, models trained on similar data seem to share similar weaknesses, as image classifiers do.
TakeawayOne automatically found suffix, built on free models, could make several commercial chat models comply with requests they would otherwise refuse.
Idea · §3.3 · §6
Disclosure, patches and defences
Before publishing, the authors shared their results with OpenAI, Google, Meta and Anthropic. The specific suffixes in the paper were quickly blocked, but the authors' point was that blocking known strings does not fix the underlying weakness: the search can simply find new ones.
Defences developed since, explained only in outline:
- Gibberish detectors. GCG suffixes look like nonsense, so a filter that flags very unnatural text ("high perplexity") catches most of them. Attackers responded with searches that produce readable text.
- Random perturbation. Slightly scrambling the input several times and taking a majority vote breaks suffixes that depend on exact characters.
- Training on attacks. Include such attacks during safety training, or change the model's internals so harmful-content pathways are disrupted whatever the input.
TakeawayPatching individual attack strings is not a defence; robustness has to come from how models are trained and deployed.
Abstract, restated
Aligned LLMs are trained not to generate objectionable content, and manual "jailbreaks" circumvent this with significant human ingenuity and brittleness. The authors propose a simple and effective attack: find a suffix that, attached to a wide range of queries, maximises the probability that the model produces an affirmative response. The suffix is found automatically by combining greedy and gradient-based search, improving over prior automatic prompt-generation methods. Suffixes optimised on multiple prompts and multiple open models (Vicuna-7B and 13B) transfer to black-box public interfaces including ChatGPT, Bard and Claude, and to open models such as LLaMA-2-Chat, Pythia and Falcon. This substantially advances adversarial attacks on aligned models and raises the question of how to prevent them. Zou, Wang, Carlini, Nasr, Kolter & Fredrikson, arXiv 2307.15043, July 2023.
Section · §2.1 · Eqs. 1–4
Objective
An LLM maps a token sequence x1:n, xi ∈ {1, …, V}, to a next-token distribution, and the probability of a continuation factorises autoregressively:
The prompt consists of the chat template, a user request, and an adversarial suffix of modifiable token positions ℐ ⊂ {1, …, n}. The adversarial loss is the negative log-probability of a target response x⋆n+1:n+H (an affirmative prefix that repeats the request):
The rationale (§2.1) is borrowed from manual jailbreaks: if the model's first tokens are affirmative, its response distribution shifts to a "mode" in which it complies. Targeting several tokens rather than one avoids the model starting with "Sure" and then refusing.
Section · §2.2–2.3 · Algorithms 1–2
Greedy Coordinate Gradient
The core difficulty is optimisation over a discrete set. GCG builds on AutoPrompt's idea: represent token xi as a one-hot vector exi, and use the gradient ∇exiℒ ∈ ℝV as a linear approximation of the effect of swapping in each token. For each modifiable position, the k tokens with the largest negative gradient form a candidate set. Then B candidate suffixes are formed, each by replacing one uniformly random position with a uniformly random token from that position's candidate set; all B are evaluated exactly with forward passes, and the best replaces the current suffix. This repeats for T iterations.
The difference from AutoPrompt is that AutoPrompt picks a single position per iteration and evaluates replacements only there, while GCG evaluates candidates across all positions in one batch; the authors report this substantially improves results. Experiments use k = 256, B = 512, 20 suffix tokens and 500 steps (§3).
Universal, multi-model (Algorithm 2). To find one suffix for many behaviours, gradients (top-k candidates) and losses are summed across prompts, and prompts are added incrementally: start with one, add the next only once the suffix succeeds on all current ones. Across models, gradients and losses are aggregated likewise, which requires a shared tokenizer (Vicuna and Guanaco both use LLaMA's).
The figure runs the same procedure on a small differentiable toy loss with a 200-token vocabulary and 8 suffix positions, comparing it with random search given the same number of loss evaluations.
Check yourself
- Why evaluate candidates exactly instead of trusting the gradient?
- The gradient with respect to a one-hot vector is only a first-order estimate of a discrete jump between embeddings, which can be far from accurate; exact forward passes pick the swap that truly lowers the loss.
- Why must the models in a multi-model attack share a tokenizer?
- Gradients are taken over token indices; summing them across models only makes sense if index v means the same token in each.
Section · §3.1 · Table 1 · Fig. 2
White-box results on AdvBench
AdvBench has two parts: 500 harmful strings (the model must output an exact string, 3 to 44 tokens long, mean 16) and 500 harmful behaviours (instructions; success means the model makes a reasonable attempt to comply). Baselines: GBDA, PEZ and AutoPrompt, all run for 500 steps with 20 optimisable tokens.
GCG dominates: on harmful strings it reaches 88% on Vicuna-7B and 57% on LLaMA-2-7B-Chat (Table 1; the paper's prose says 55%), where GBDA and PEZ reach 0%. A universal suffix trained on 25 behaviours elicits 98% of 100 held-out behaviours on Vicuna and 84% on LLaMA-2-Chat. Fig. 2 shows GCG also reaching low loss much faster than AutoPrompt.
Section · §3.2–3.3 · Table 2 · Fig. 3
Transfer to black-box models
Suffixes were optimised on Vicuna-7B and 13B (two runs with different seeds), and on Vicuna-7B/13B plus Guanaco-7B/13B, over 25 behaviours, then tested on 388 behaviours against closed APIs.
| Method | GPT-3.5 | GPT-4 | Claude-1 | Claude-2 | PaLM-2 |
|---|---|---|---|---|---|
| Behaviour only | 1.8 | 8.0 | 0.0 | 0.0 | 0.0 |
| + "Sure, here's" | 5.7 | 13.1 | 0.0 | 0.0 | 0.0 |
| + GCG (Vicuna) | 34.3 | 34.5 | 2.6 | 0.0 | 31.7 |
| + GCG (Vicuna & Guanacos) | 47.4 | 29.1 | 37.6 | 1.8 | 36.1 |
| + Concatenate | 79.6 | 24.2 | 38.4 | 1.3 | 14.4 |
| + Ensemble | 86.6 | 46.9 | 47.9 | 2.1 | 66.0 |
"Concatenate" joins multiple optimised suffixes into one; "Ensemble" counts success if any of the suffixes works. Caveats the authors note (§3.3): Vicuna is distilled from ChatGPT outputs, which may inflate transfer to GPT-3.5; running too many optimisation steps can overfit to the source models and reduce transfer (Fig. 6); and Claude's chat interface appeared to apply an initial content filter that the API did not. Human judgement of "success" (did the output attempt the behaviour) is a coarse metric.
Section · §5–6 · Lineage
Defences and lineage
The paper itself names adversarial training as the natural direction and notes that, in vision, input-level detectors and preprocessing have repeatedly been broken by adaptive attacks (§3.3, §5). It disclosed results to OpenAI, Google, Meta and Anthropic in advance (§6), arguing that publication was warranted because the method is simple and the problem needed attention before more capable systems were deployed.
- Detection. GCG suffixes have very high perplexity, so perplexity filters catch them (Jain et al., 2023; Alon & Kamfonas, 2023), at some false-positive cost. Fluency-constrained attacks (e.g. AutoDAN, Zhu et al., 2023) and LLM-driven attacks (PAIR, Chao et al., 2023) produce readable prompts that evade such filters.
- Input smoothing. SmoothLLM (Robey et al., 2023) randomly perturbs several copies of the input and aggregates the responses, exploiting the brittleness of character-exact suffixes.
- Training. Adversarial training against GCG-style attacks (e.g. R2D2 in HarmBench, Mazeika et al., 2024) and representation-level methods such as circuit breakers (Zou et al., 2024) that reroute harmful internal representations regardless of the prompt.
- Understanding. Arditi et al. (2024) found refusal mediated by a single residual-stream direction, and that adversarial suffixes suppress it; this gives a mechanistic account of why a short suffix can flip behaviour.
Check yourself
- Why is a perplexity filter not a complete defence?
- It targets a property of this particular attack (gibberish tokens), not the vulnerability; attacks optimised under a fluency constraint, or written by another LLM, have normal perplexity.
- What does the near-zero Claude-2 transfer rate tell us, and what doesn't it?
- That these specific suffixes rarely transferred to that model and interface; it does not show the model is robust to attacks optimised against it or to other attack styles.
Reference
Glossary
- Aligned model
- A model fine-tuned (e.g. with RLHF) to follow instructions and refuse harmful ones.
- Jailbreak
- An input that makes an aligned model produce content it was trained to refuse.
- Suffix
- Tokens appended after the user's request; the optimisation variable.
- One-hot gradient
- Gradient of the loss with respect to the indicator vector of a token, used to rank substitutions.
- Universal
- One suffix that works across many prompts.
- Transfer
- An attack crafted on one model succeeding on another.
- ASR
- Attack success rate.
- Perplexity
- How surprising text is to a language model; gibberish has high perplexity.