The paper in four sentences
Inside a language model, every word is represented by a long list of numbers, and "directions" in that space of numbers can stand for ideas.
Compare the model's internal numbers on harmful requests with those on harmless ones, take the average difference, and you get one arrow, the refusal direction.
Erase that arrow from the model's thinking and it answers requests it used to refuse; add it and it refuses even "write a poem about spring".
The erasing can be written permanently into the model's weights at almost no cost to its other skills, which shows that the safety training of these open models is shallow and easy to undo.
Idea · §1–2.1
Ideas live in directions
A language model reads text as tokens (word pieces). For each token it keeps a long list of numbers, a few thousand of them, which is updated layer by layer as the text flows through the network. Researchers call this list the residual stream: think of it as a shared notepad that every layer reads from and writes onto.
A list of numbers can be drawn as an arrow (a vector) in a space with thousands of dimensions. A long-standing observation in interpretability research is that many concepts correspond to particular directions in that space. Move the arrow further along the "French" direction and the text gets more French; along another direction, more positive in tone. This is called the linear representation idea.
Chat models are trained, after their main training, to refuse harmful requests. Nobody wrote a "refusal" rule into them; the behaviour was learned. The paper asks: where did that behaviour end up inside the network?
TakeawayA model's internal state is a long list of numbers, and concepts often correspond to directions in that space.
Idea · §2.3
Finding the refusal arrow: subtract two averages
The method is strikingly simple. Feed the model 128 harmful instructions and 128 harmless ones. At a chosen layer, record the internal arrow for each. Average the harmful arrows, average the harmless arrows, and subtract. The result, called the difference in means, points from "typical harmless request" towards "typical harmful request".
No training is involved, and no examples of harmful answers are needed: only two sets of questions. The figure shows the idea in two dimensions instead of thousands.
TakeawayThe refusal direction is just the average internal state on harmful prompts minus the average on harmless prompts.
Check yourself
- Why does the method not need any examples of harmful answers?
- It only looks at the model's internal state while reading the request, before it answers. Two lists of prompts are enough.
- In the toy, what happens to sorting accuracy when you shrink the separation to 0?
- It falls to about chance (50%): if both groups have the same average, the difference arrow is just noise.
Idea · §3 · Figs. 1, 3
One dial, both ways
Finding an arrow that correlates with harmfulness is not the same as showing it causes refusal. So the authors intervene. They test two edits while the model runs:
- Remove the direction: at every layer and every token, subtract whatever part of the internal arrow points along the refusal direction, leaving everything else untouched. The model now answers harmful requests it used to refuse.
- Add the direction: push the internal arrow further along it on a harmless request. The model now refuses perfectly innocent requests.
This held for all 13 chat models tested, from five families (Qwen, Yi, Gemma, Llama-2 and Llama-3) and from 1.8 to 72 billion parameters. Try both edits on a toy model below.
TakeawayRemoving the direction stops refusals and adding it causes them: in these models refusal is controlled by one direction.
Check yourself
- Why do the authors run both experiments, removing and adding?
- Removal alone shows the direction is necessary for refusal; addition shows it is sufficient to trigger refusal. Together they make the causal case.
- When the direction is removed, why can the model still write fluent, knowledgeable text?
- Only one direction out of thousands is erased. Everything the arrow says along the other directions is left as it was.
Idea · §4 · Tables 2–3
Writing it into the weights
Removing the direction while the model runs requires changing its code. The authors show something more consequential: the same effect can be achieved by a one-time edit of the model's weights, so that no layer can ever write anything along the refusal direction. The edited model is an ordinary file that behaves as if the direction were removed on every run. The community soon named this abliteration (ablation plus obliteration).
How much does it cost the model's other abilities? Very little on the benchmarks they checked. On general-knowledge tests (MMLU, ARC) and grade-school maths (GSM8K), the edited models scored within about a point or two of the originals. The one consistent drop was on TruthfulQA, a test of avoiding common misconceptions, which fell by 1 to 3.5 points.
The authors are frank about the ethics. Their method, they write, "marginally lowers the bar" for jailbreaking open-weight models, though they argue it does not substantially change the risk of releasing them, since fine-tuning could already remove safety training. They note it can be done on a 70-billion-parameter model for under $5 of compute.
TakeawayA tiny, one-off weight edit can strip refusal from an open-weight model while leaving most other skills intact.
Idea · §5 · Figs. 5–6
Why jailbreak gibberish works
Earlier work (GCG) found strange strings of characters that, added to the end of a harmful request, make models comply. Why do they work? The authors look inside a small model (Qwen 1.8B) and find that these "adversarial suffixes" turn the refusal direction down: with the suffix, the model's internal state on a harmful request looks like its state on a harmless one. Random strings of the same length do not do this.
They also find how: a handful of attention heads (parts of the network that decide which earlier words to focus on) normally carry the "this is harmful" signal from the request into the refusal direction. The suffix pulls their attention away from the request and onto itself.
TakeawayAdversarial suffixes jailbreak models by distracting the parts of the network that feed the refusal direction.
Check yourself
- What does the single-direction finding explain about suffix attacks?
- An attack does not need to change everything the model knows; it only needs to stop the refusal direction from being switched on, which can be done by redirecting a few attention heads.
Idea · §7 · Later work
What it means, and what came after
The headline lesson is that safety fine-tuning on these models was brittle: the behaviour hung on one direction that anyone with the weights could find in minutes. It is also a success story for interpretability, since understanding a mechanism immediately predicted how to switch it on and off.
The authors say plainly that their result is "an existence proof" that such a direction exists, not a careful study of the best way to find it, and that it may not hold for untested or larger models.
TakeawayRefusal in 2024's open chat models was a thin layer on top, easy to find and easy to peel off; later work showed the picture has more dimensions and can be made sturdier.
Abstract, restated
Conversational LLMs are fine-tuned to obey benign instructions and refuse harmful ones, but the mechanism is poorly understood. Across 13 popular open-source chat models up to 72B parameters, refusal is mediated by a one-dimensional subspace: for each model there is a direction whose erasure from the residual-stream activations prevents refusal of harmful instructions, and whose addition elicits refusal on harmless ones. This yields a white-box jailbreak, a rank-one weight edit that disables refusal with minimal effect on other capabilities. Mechanistically, adversarial suffixes suppress propagation of the refusal-mediating direction. The results underline the brittleness of current safety fine-tuning and show how internals can be used to control behaviour. Arditi, Obeso, Syed, Paleka, Panickssery, Gurnee & Nanda, NeurIPS 2024 (arXiv:2406.11717).
Section · §2.1–2.2 · Eq. 1 · Table 1
Setting: residual stream, data and models
Decoder-only transformer with residual stream activation xi(l) ∈ ℝdmodel at token i and layer l. Each block adds attention and MLP outputs to the stream:
Data. 𝒟harmful is drawn from AdvBench, MaliciousInstruct, TDC2023 and HarmBench; 𝒟harmless from Alpaca. Each has a train split of 128 and a validation split of 32, filtered to avoid overlap with evaluation sets. Evaluation uses JailbreakBench (100 harmful instructions) for bypassing, 100 held-out Alpaca instructions for inducing, and the 159 HarmBench "standard behaviours" for the jailbreak comparison.
Models (Table 1). Qwen Chat 1.8B/7B/14B/72B and Yi Chat 6B/34B (aligned by fine-tuning, AFT); Gemma IT 2B/7B, Llama-2 Chat 7B/13B/70B and Llama-3 Instruct 8B/70B (aligned by preference optimisation, APO).
Metrics (§2.5). Greedy decoding to 512 tokens. Refusal score: 1 if the completion contains any of a list of refusal substrings ("I'm sorry", "As an AI", ...), else 0. Safety score: 1 if Llama Guard 2 judges the completion safe. The two are complementary: string matching catches memorised refusals but says nothing about whether a non-refusal is coherent or actually harmful.
Section · §2.3 · Eq. 2 · App. C
Difference-in-means and choosing one vector
For each layer l and each post-instruction token position i ∈ I (the chat-template tokens after the user's message), compute the mean activation over harmful and harmless training prompts:
This yields |I| × L candidates. Each is scored on the validation splits for how well ablating it bypasses refusal on harmful prompts, how well adding it induces refusal on harmless prompts, and how little ablating it changes the model's behaviour on harmless prompts (a KL-divergence check); candidates too close to the final layers, where directions start to align with the unembedding, are excluded (App. C). The winning vector is normalised to r̂. The selected layers sit in the middle of the network for most models (for example layer 14 of 32 for Llama-2 7B, layer 23 of 40 for Qwen 14B; App. C, Table 5).
Difference-in-means is the simplest linear probe imaginable; the paper notes it is chosen for simplicity and works well, consistent with earlier representation-engineering and steering work (e.g. Marks & Tegmark 2023; Zou et al. 2023; Turner et al. 2023).
Check yourself
- Why restrict the search to post-instruction token positions?
- Those template tokens are where the model has read the whole request and is about to start answering, so a decision such as "refuse" should be represented there; they are also identical across prompts, making averages comparable.
- Why include a KL check on harmless prompts when selecting a direction?
- A direction can bypass refusal simply by breaking the model. Requiring little change on harmless prompts selects a direction that is specific to refusal rather than generally destructive.
Section · §2.4, §4.1 · Eqs. 3–5 · App. E
Three interventions, one projection
Activation addition (at the source layer l* only, all positions) and directional ablation (every layer, every position) are:
Ablation zeroes the stream's component along r̂ while leaving its orthogonal complement untouched. Because the residual stream is a sum of writes (embedding, then every attention and MLP output), the same effect is achieved by projecting r̂ out of every matrix that writes to the stream:
applied to the embedding matrix, positional embeddings, attention output projections, MLP output projections and any output biases. Appendix E proves equivalence to inference-time ablation: if no write has any component along r̂, then no residual activation does. The edit is rank-one per matrix. The figure shows a 2-D slice.
Check yourself
- Why is the weight edit equivalent to ablation at every layer, rather than at one layer?
- Every component that adds to the residual stream is edited, so the stream's projection onto r̂ starts at zero (embeddings) and every subsequent addition contributes zero; it stays zero at every layer and position.
- Why is addition applied only at one layer while ablation is applied everywhere?
- Adding at one layer is enough to push the state into "refuse" territory, which later layers then propagate. Ablation must stop any layer re-introducing the direction, so it has to act everywhere.
Section · §3–4 · Figs. 1, 3 · Tables 2–3
Results: bypass, induce, compare, coherence
Bypass (§3.1, Fig. 1). On JailbreakBench, directional ablation drives refusal scores down and safety scores down across all 13 models, i.e. the models comply with harmful requests rather than emitting incoherent text. Induce (§3.2, Fig. 3). Without intervention the models rarely refuse the 100 harmless Alpaca prompts; activation addition of r at l* sharply raises the refusal rate on these same prompts.
Jailbreak comparison (Table 2). HarmBench attack success rate (ASR, %) of the orthogonalised model ("Ortho") against baselines. GCG-M and GCG-T are universal (multi-prompt, transfer) suffixes; GCG, AutoPrompt (AP) and PAIR are optimised per prompt; DR is direct request. Ortho is reported with the model's default system prompt and, in parentheses, without one. For Qwen, Ortho is on par with per-prompt GCG; Llama-2 is strongly system-prompt dependent (22.6% with, 79.9% without for 7B).
Coherence (Table 3, §4.3). MMLU, ARC, GSM8K and TruthfulQA for orthogonalised vs original models. Apart from Qwen 7B and Yi 34B, all orthogonalised scores lie within 99% confidence intervals of the originals; TruthfulQA drops consistently. Cross-entropy on held-out text (App. G.3) suggests directional ablation is "more surgical" than activation-addition methods.
Section · §5 · Figs. 5–6 · App. H
Mechanism of adversarial suffixes
On Qwen 1.8B Chat the authors take 128 harmful instructions (from JailbreakBench and HarmBench) and run each with no suffix, with a GCG-optimised adversarial suffix, and with a random suffix of equal length. They measure the cosine similarity between the last-token activation and r̂.
- Suppression (§5.1, Fig. 5). With the adversarial suffix, the refusal-direction expression is heavily suppressed and closely resembles that of harmless instructions; random suffixes do not have this effect.
- Head hijacking (§5.2, Fig. 6). Using direct feature attribution, the eight attention heads that write most strongly onto r̂ have their contribution sharply reduced by the suffix, and their attention shifts from the instruction tokens to the suffix tokens.
This contrasts with Zou et al. (2023a), who had reported that adversarial suffixes do not significantly alter a "harmfulness" representation; here, what they alter is the refusal representation downstream of it.
Check yourself
- Why is the random-suffix control essential?
- Any extra text dilutes attention somewhat. The control shows that suppression of r̂ is specific to the optimised suffix, not a generic effect of a longer prompt.
Section · §7 · Lineage
Limitations, ethics and what came next
- Existence proof. The authors stress that difference-in-means is a simple choice and the work demonstrates that a refusal-mediating direction exists, not how best to extract it; results may not generalise to untested or larger models.
- Coarse metrics. Refusal is detected by substring matching and harm by Llama Guard 2; neither measures how useful a non-refused answer actually is.
- Ethics. The method needs no gradient optimisation and no harmful completions, and costs under $5 of compute for a 70B model. The authors judge that it marginally lowers the bar for jailbreaking open weights without substantially changing the risk profile, since fine-tuning attacks already existed.
- Implication for alignment. If preference optimisation and fine-tuning leave refusal concentrated in one direction, then robust safety for open-weight models cannot rest on that training alone.
Reference
Glossary
- Residual stream
- The per-token activation vector that every block reads from and adds to.
- Difference-in-means
- μ − ν: mean harmful activation minus mean harmless activation.
- Directional ablation
- Removing the component along r̂: x − r̂r̂⊤x.
- Activation addition
- Adding a steering vector to the stream at one layer.
- Weight orthogonalisation
- Projecting r̂ out of every stream-writing matrix (Eq. 5); "abliteration".
- AFT / APO
- Aligned by fine-tuning / by preference optimisation.
- ASR
- Attack success rate, here judged by the HarmBench classifier.
- DFA
- Direct feature attribution: a head's output projected onto a direction.