Study guide · Language models & AI · 2021

RoFormer: Enhanced Transformer with Rotary Position Embedding

Jianlin Su et al. · arXiv

A Transformer has to be told where each word sits, and the original recipe simply added a position "barcode" to each word. Su and colleagues instead rotate each word's query and key by an angle proportional to its position, so that when two words are compared only the difference in their angles, how far apart they are, survives. This rotary scheme, RoPE, is now the default in most open large language models.

The paper in four sentences

Attention compares words by multiplying lists of numbers together, and on its own it has no idea of word order, so position information must be mixed in somehow.

RoPE does it by rotation: chop each word's query and key into pairs of numbers, treat each pair as an arrow, and turn the arrow by an angle that grows with the word's position.

When two rotated arrows are compared, the result depends only on the angle between them, so the attention score depends on how far apart the words are, not on where they sit in absolute terms.

Different pairs spin at different speeds, like the hands of a clock, which lets the model tell both near and far distances apart, and the influence between words tends to fade with distance.

Idea · §1–2

Attention is blind to order

In a Transformer, each word decides how much to pay attention to each other word by comparing two lists of numbers, its query and the other word's key. The comparison is a dot product: multiply matching entries and add them up. Nothing in that calculation knows which word came first. "Dog bites man" and "man bites dog" would get identical scores.

The original Transformer fixed this by adding a unique position pattern to each word before anything else happens, like stamping a page number onto every page. That works, but it describes absolute position ("I am word 37"). For language, what usually matters is relative position: "the adjective right before me", "the subject a few words back". Other researchers had added relative-position tricks inside attention, but they were fiddly, and many didn't fit with faster variants of attention.

The question this paper asks: is there a way to inject position so that the comparison between two words automatically depends only on their distance?

Everyday pictureTwo people in a queue. "Person 37 and person 40" is absolute; "three places apart" is relative. If the whole queue shuffles forward ten places, the absolute numbers all change but the relationship between them doesn't.

TakeawayAttention needs position information added, and what matters most for language is the distance between words rather than their absolute positions.

Check yourself
Why can't plain attention tell "dog bites man" from "man bites dog"?
Its scores come from comparing word contents only. Reordering the words reorders the scores but doesn't change any of them, so nothing marks who did what to whom.

Idea · §3.2.1 · Eqs. 12–13

Turn the arrows by position

Take two numbers from a query and treat them as an arrow on a flat page: the first number is how far right it points, the second how far up. Now rotate that arrow by an angle that depends on the word's position: word 1 turns by a little, word 2 by twice as much, word 10 by ten times as much. Do the same to the keys.

Here is the key fact: the dot product of two arrows depends on their lengths and the angle between them, nothing else. If the query is turned by "position × step" and the key by "its position × step", the angle between them changes by "difference in positions × step". So the score depends on the distance between the words. Shift both words along by the same amount, and both arrows turn by the same extra angle, leaving the angle between them, and the score, unchanged.

Try to break it below.

Faint arrows: the query and key before rotation (their content). Solid arrows: after rotating by m·θ and n·θ (in radians; 1 radian ≈ 57°). The table on the right logs every score you produce, grouped by distance m − n: every row with the same distance has the same score. Illustrative two-number vectors; real heads have 64 or 128 numbers, handled as 32 or 64 such pairs.
Everyday pictureTwo clock hands. If both are moved forward by an hour, the angle between them stays the same. RoPE only ever lets attention "see" that angle.

TakeawayRotating query and key by angles proportional to their positions makes their dot product depend only on the difference in positions.

Check yourself
A query at position 7 and a key at position 4 give some score. What other pair of positions gives the same score (same content)?
Any pair three apart in the same order, such as 103 and 100. Only the offset m − n = 3 matters.
Does rotating an arrow change its length?
No. Rotation only changes direction, so the word's content "strength" is untouched; only the angle carries position.

Idea · §3.2.2 · Eq. 15

Many hands, many speeds

One rotating arrow has a problem: angles wrap around. If the step is a quarter turn, then positions 0, 4, 8 … all point the same way and look identical. A real query has many numbers, so RoPE splits it into many pairs, and each pair spins at its own speed. The first pair spins fast (one radian per word), and each later pair spins slower than the one before, down to pairs that take tens of thousands of words to complete a turn.

That is exactly how a clock tells time: the second hand distinguishes nearby moments, the hour hand distant ones, and together they are unambiguous. The fast pairs let attention resolve "one word back" versus "two words back"; the slow pairs keep far-apart positions distinct.

A 16-number head has 8 pairs. Pair i rotates by θi = 10000−2(i−1)/16 radians per position, the paper's formula (Eq. 15) with d = 16. The number under each dial is how many positions one full turn takes. Watch the left dials whirl while the right ones barely move.

TakeawayDifferent pairs rotate at geometrically spaced speeds, so together they encode both short and long distances without ambiguity.

Check yourself
Why not make every pair rotate fast?
Fast rotations wrap around after a few positions, so distant positions would be confused with near ones. Slow pairs are needed to separate long distances.

Idea · §3.3 · §3.4.3 · Fig. 2

Side benefits: fading with distance, no extra parts

The paper points out three bonuses.

  • Fading with distance. Because the many pairs spin at different speeds, for far-apart words their contributions point in scattered directions and partly cancel. The authors derive an upper limit on the attention score and show numerically that it tends to shrink as distance grows, matching the intuition that far-away words usually matter less.
  • Nothing to learn. The rotations are fixed formulas, so RoPE adds no new trainable numbers to the model.
  • Works with fast attention. Some faster forms of attention ("linear attention") never compute the full table of scores, which ruled out many earlier relative-position methods. Rotation is applied to each query and key separately, so it fits.

TakeawayRoPE gives relative positions for free: no new parameters, a natural fade with distance, and compatibility with efficient attention variants.

Idea · §4 · What came next

Modest results, enormous adoption

The paper's own experiments are modest. In translation it scored slightly above the standard Transformer. When pre-training a BERT-style model, the loss went down faster. On a Chinese legal-document task, the RoPE model handled longer inputs (1,024 tokens) better than models limited to 512. The authors openly admit they cannot fully explain why it trains faster.

What made RoPE famous was what happened next. Open-source groups tried it in large language models in 2021, found it worked well, and it spread: PaLM, LLaMA and most open models since use it.

Later refinementModels trained with RoPE on, say, 4,000 words often break when given much longer text, because the slow dials reach angles they never saw in training. Fixes from 2023 onward stretch the rotation: "position interpolation" squeezes long inputs into the trained range, and methods such as YaRN or a larger base number slow the dials down. These let RoPE models be extended to contexts of 100,000 words or more with little extra training.

TakeawayRoPE's paper showed small gains, but its simplicity and relative-distance property made it the standard position method of modern open LLMs.

Follow the idea

Built on: Transformer (2017)

Led to: LLaMA (2023)