Study guide · Safety & security · 2023

Universal and Transferable Adversarial Attacks on Aligned Language Models

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson · arXiv

Chat models are trained to refuse harmful requests, but is that refusal robust? This paper showed that an automated search can find a string of odd-looking tokens which, appended to a request, makes open models comply, and that a string found on open models often works on closed commercial ones too. It turned "jailbreaking" from hand-crafted tricks into an optimisation problem, and showed that safety training gives no guarantee against adversarial inputs.

The paper in four sentences

Chat models such as ChatGPT and LLaMA-2-Chat are "aligned": trained to decline requests for harmful content. People had found manual tricks to get round this, but those were fiddly and quickly patched.

The authors automate it: they search for a short string of tokens (a suffix) that, added after a request, makes the model start its reply with an agreeable opening like "Sure, here is", after which it tends to carry on.

The search uses the model's own gradients (a signal of which way to change things) to shortlist promising token swaps, then tests them; optimising one suffix across many requests and several models makes it general.

Suffixes found on small open models also worked, at varying rates, on GPT-3.5, GPT-4, PaLM-2 and Claude-1, though Claude-2 was barely affected, showing that refusal training was brittle in ways nobody had a fix for.

Idea · §1

Adversarial examples, now for text

A decade earlier, researchers showed that image classifiers can be fooled by adversarial examples: tiny, carefully computed changes to a picture, invisible to people, that make the network confidently wrong. The recipe is to use the network's own gradients to find the change that most increases its error.

Text is harder. An image's pixels can be nudged by tiny amounts; words cannot. You can only swap one token (a word or word-piece) for another, and there are tens of thousands to choose from at every position. Earlier automatic attacks on language models mostly failed against chat models trained to refuse.

This paper's contribution is a search method that works well enough in this discrete setting to break aligned chat models reliably, with no human creativity needed.

Everyday pictureA combination lock with 20 dials and 32,000 symbols on each. Trying every combination is hopeless. But if the lock gives a faint click when a dial gets "warmer", you can turn the dials one at a time towards the clicks. The gradient is that faint click.

TakeawayGCG brings the adversarial-example idea from images to chat models: let the model's own internals guide a search for inputs that defeat its training.

Idea · §2.1

Aim for the first words of the reply

What should the search aim for? "Make the model say something harmful" is too vague to measure. The authors use a simple proxy: make the model's reply begin with an affirmative phrase that restates the request, such as "Sure, here is …". Language models continue in the style they start in, so once a model has committed to "Sure, here is", it usually carries on rather than reversing into a refusal.

This turns the goal into a number the search can improve: how likely the model is to produce those exact opening words.

TakeawayThe attack targets the opening of the response, because a model that starts by agreeing tends to keep going.

Check yourself
Why not simply aim for the single word "Sure"?
A single agreeable token can still be followed by a refusal. Targeting a longer opening that repeats the request commits the model more firmly to answering.

Idea · §2.2

Search one token at a time, guided by gradients

The method is called Greedy Coordinate Gradient (GCG). "Coordinate" means it changes one position of the suffix at a time; "greedy" means it keeps whichever change helps most right now; "gradient" means it uses the model's gradients to decide which changes are worth testing.

Try a miniature version below. This is a made-up scoring function, not a language model: five slots, eight nonsense syllables, and a hidden score that depends on which syllables sit where and next to what. Click slots to change them by hand and try to push the score over the line. Then let the greedy search do it.

An abstract toy. Click a slot to cycle its syllable. The score is a fixed hidden function (each syllable has a value in each slot, plus bonuses and penalties for neighbouring pairs), standing in for "how likely the model is to begin with the target words". The greedy step tries every single-slot change (5 × 7 = 35 options) and keeps the best. A real suffix has about 20 slots and a vocabulary of about 32,000 tokens, roughly 640,000 options per step, which is why GCG uses gradients to shortlist a few hundred to test.

TakeawayGCG is a patient, guided trial-and-error: shortlist promising single-token swaps using gradients, test them for real, keep the best, repeat.

Check yourself
Why does the search need gradients at all, if it tests candidates for real anyway?
There are far too many possible swaps to test them all each step. Gradients cheaply estimate which swaps are likely to help, so only a small shortlist needs real evaluation.
Why can a greedy search get stuck?
Sometimes no single swap improves the score even though changing two slots together would; the toy's pair bonuses can create such local optima.

Idea · §2.3 · §3.2 · Table 2

Universal and transferable

Two further steps made the attack alarming. First, universal: rather than one suffix per request, the authors optimised a single suffix to work across many different harmful requests at once. Second, transferable: they optimised it against several open models together (Vicuna and Guanaco, small chat models built on LLaMA) and then simply pasted it, unchanged, after requests to closed commercial models they had no internal access to.

Pick an attack setting to see the share of 388 test behaviours that each closed model complied with.

Real numbers from Table 2 (attack success rate, %, averaged over 388 behaviours). "Request only": the harmful request with nothing added. "+ Sure, here's": asking the model to start that way. "GCG (Vicuna)": a suffix optimised on Vicuna models. "Ensemble": counting a success if any of several optimised suffixes works. Claude-2 was barely affected.

Why would a suffix found on a small open model work on GPT-4? The paper notes one likely factor: Vicuna was itself trained on conversations with ChatGPT, so it may share some of its quirks. More generally, models trained on similar data seem to share similar weaknesses, as image classifiers do.

TakeawayOne automatically found suffix, built on free models, could make several commercial chat models comply with requests they would otherwise refuse.

Idea · §3.3 · §6

Disclosure, patches and defences

Before publishing, the authors shared their results with OpenAI, Google, Meta and Anthropic. The specific suffixes in the paper were quickly blocked, but the authors' point was that blocking known strings does not fix the underlying weakness: the search can simply find new ones.

Defences developed since, explained only in outline:

  • Gibberish detectors. GCG suffixes look like nonsense, so a filter that flags very unnatural text ("high perplexity") catches most of them. Attackers responded with searches that produce readable text.
  • Random perturbation. Slightly scrambling the input several times and taking a majority vote breaks suffixes that depend on exact characters.
  • Training on attacks. Include such attacks during safety training, or change the model's internals so harmful-content pathways are disrupted whatever the input.
Later refinementJailbreak research grew into a field with shared benchmarks (HarmBench) and many attack styles, including ones where one language model attacks another in plain language. Later work found that refusal in many chat models is controlled by a single internal direction, which explains why it can be bypassed so cleanly (Arditi et al., 2024). Defences that act on the model's internal representations rather than on its inputs became a leading approach, though no method yet offers a guarantee.

TakeawayPatching individual attack strings is not a defence; robustness has to come from how models are trained and deployed.

Follow the idea

Built on: Adversarial examples (2013)

Led to: Abliteration (2024)