Study guide · Safety & security · 2024

Refusal in Language Models Is Mediated by a Single Direction

Andy Arditi et al. · NeurIPS 2024

When a chat model says "I'm sorry, I can't help with that", is the refusal spread across the whole network or does it hang on one thread? Arditi and colleagues show that, in 13 open chat models, it hangs on one thread: a single direction in the model's internal activations. Remove that direction and the model stops refusing; add it and the model refuses harmless requests. Editing the weights to remove it permanently became known as "abliteration".

The paper in four sentences

Inside a language model, every word is represented by a long list of numbers, and "directions" in that space of numbers can stand for ideas.

Compare the model's internal numbers on harmful requests with those on harmless ones, take the average difference, and you get one arrow, the refusal direction.

Erase that arrow from the model's thinking and it answers requests it used to refuse; add it and it refuses even "write a poem about spring".

The erasing can be written permanently into the model's weights at almost no cost to its other skills, which shows that the safety training of these open models is shallow and easy to undo.

Idea · §1–2.1

Ideas live in directions

A language model reads text as tokens (word pieces). For each token it keeps a long list of numbers, a few thousand of them, which is updated layer by layer as the text flows through the network. Researchers call this list the residual stream: think of it as a shared notepad that every layer reads from and writes onto.

A list of numbers can be drawn as an arrow (a vector) in a space with thousands of dimensions. A long-standing observation in interpretability research is that many concepts correspond to particular directions in that space. Move the arrow further along the "French" direction and the text gets more French; along another direction, more positive in tone. This is called the linear representation idea.

Chat models are trained, after their main training, to refuse harmful requests. Nobody wrote a "refusal" rule into them; the behaviour was learned. The paper asks: where did that behaviour end up inside the network?

Everyday pictureA mixing desk with thousands of sliders. Most ideas are not one slider but a particular combination of them, a "direction" you can push the whole desk in. The question is whether "refuse" is one such combination or something smeared across everything.

TakeawayA model's internal state is a long list of numbers, and concepts often correspond to directions in that space.

Idea · §2.3

Finding the refusal arrow: subtract two averages

The method is strikingly simple. Feed the model 128 harmful instructions and 128 harmless ones. At a chosen layer, record the internal arrow for each. Average the harmful arrows, average the harmless arrows, and subtract. The result, called the difference in means, points from "typical harmless request" towards "typical harmful request".

No training is involved, and no examples of harmful answers are needed: only two sets of questions. The figure shows the idea in two dimensions instead of thousands.

Illustrative toy, not the paper's data. Amber dots: harmful requests. Blue dots: harmless requests. Crosses: the two averages. The arrow between them is the difference-in-means direction. The readout says how often simply measuring "how far along the arrow" a point sits sorts it into the right group.
Everyday pictureAverage the height of basketball players and of jockeys and subtract: you get one number that captures the main way the two groups differ, without having to understand anything else about them.

TakeawayThe refusal direction is just the average internal state on harmful prompts minus the average on harmless prompts.

Check yourself
Why does the method not need any examples of harmful answers?
It only looks at the model's internal state while reading the request, before it answers. Two lists of prompts are enough.
In the toy, what happens to sorting accuracy when you shrink the separation to 0?
It falls to about chance (50%): if both groups have the same average, the difference arrow is just noise.

Idea · §3 · Figs. 1, 3

One dial, both ways

Finding an arrow that correlates with harmfulness is not the same as showing it causes refusal. So the authors intervene. They test two edits while the model runs:

  • Remove the direction: at every layer and every token, subtract whatever part of the internal arrow points along the refusal direction, leaving everything else untouched. The model now answers harmful requests it used to refuse.
  • Add the direction: push the internal arrow further along it on a harmless request. The model now refuses perfectly innocent requests.

This held for all 13 chat models tested, from five families (Qwen, Yi, Gemma, Llama-2 and Llama-3) and from 1.8 to 72 billion parameters. Try both edits on a toy model below.

Illustrative toy. The horizontal axis is the refusal direction; the vertical axis stands for every other direction. The toy model refuses whenever a prompt's arrow reaches the shaded zone. "Remove" sets each arrow's horizontal part to zero but keeps its vertical part, which is why the model's other knowledge survives. "Add" shifts both arrows to the right.

TakeawayRemoving the direction stops refusals and adding it causes them: in these models refusal is controlled by one direction.

Check yourself
Why do the authors run both experiments, removing and adding?
Removal alone shows the direction is necessary for refusal; addition shows it is sufficient to trigger refusal. Together they make the causal case.
When the direction is removed, why can the model still write fluent, knowledgeable text?
Only one direction out of thousands is erased. Everything the arrow says along the other directions is left as it was.

Idea · §4 · Tables 2–3

Writing it into the weights

Removing the direction while the model runs requires changing its code. The authors show something more consequential: the same effect can be achieved by a one-time edit of the model's weights, so that no layer can ever write anything along the refusal direction. The edited model is an ordinary file that behaves as if the direction were removed on every run. The community soon named this abliteration (ablation plus obliteration).

How much does it cost the model's other abilities? Very little on the benchmarks they checked. On general-knowledge tests (MMLU, ARC) and grade-school maths (GSM8K), the edited models scored within about a point or two of the originals. The one consistent drop was on TruthfulQA, a test of avoiding common misconceptions, which fell by 1 to 3.5 points.

The authors are frank about the ethics. Their method, they write, "marginally lowers the bar" for jailbreaking open-weight models, though they argue it does not substantially change the risk of releasing them, since fine-tuning could already remove safety training. They note it can be done on a 70-billion-parameter model for under $5 of compute.

Everyday pictureRather than blocking one slider every time the desk is used, you saw off the wire that connects it. The desk works as before in every other way.

TakeawayA tiny, one-off weight edit can strip refusal from an open-weight model while leaving most other skills intact.

Idea · §5 · Figs. 5–6

Why jailbreak gibberish works

Earlier work (GCG) found strange strings of characters that, added to the end of a harmful request, make models comply. Why do they work? The authors look inside a small model (Qwen 1.8B) and find that these "adversarial suffixes" turn the refusal direction down: with the suffix, the model's internal state on a harmful request looks like its state on a harmless one. Random strings of the same length do not do this.

They also find how: a handful of attention heads (parts of the network that decide which earlier words to focus on) normally carry the "this is harmful" signal from the request into the refusal direction. The suffix pulls their attention away from the request and onto itself.

TakeawayAdversarial suffixes jailbreak models by distracting the parts of the network that feed the refusal direction.

Check yourself
What does the single-direction finding explain about suffix attacks?
An attack does not need to change everything the model knows; it only needs to stop the refusal direction from being switched on, which can be done by redirecting a few attention heads.

Idea · §7 · Later work

What it means, and what came after

The headline lesson is that safety fine-tuning on these models was brittle: the behaviour hung on one direction that anyone with the weights could find in minutes. It is also a success story for interpretability, since understanding a mechanism immediately predicted how to switch it on and off.

The authors say plainly that their result is "an existence proof" that such a direction exists, not a careful study of the best way to find it, and that it may not hold for untested or larger models.

Later refinement"One direction" turned out to be too simple. Wollschläger et al. (2025) found several independent refusal directions forming multi-dimensional "concept cones". Abu Shairah et al. (2025) showed that training models to explain their refusals makes abliteration much less effective. Young (2025) compared the open-source tools that implement the idea and found that the damage they do to other skills varies widely between models.

TakeawayRefusal in 2024's open chat models was a thin layer on top, easy to find and easy to peel off; later work showed the picture has more dimensions and can be made sturdier.

Follow the idea

Built on: Universal jailbreaks (GCG) (2023)

Led to: Refusal concept cones (2025), Defending against abliteration (2025), Abliteration tools compared (2025)