The paper in four sentences
Attention compares words by multiplying lists of numbers together, and on its own it has no idea of word order, so position information must be mixed in somehow.
RoPE does it by rotation: chop each word's query and key into pairs of numbers, treat each pair as an arrow, and turn the arrow by an angle that grows with the word's position.
When two rotated arrows are compared, the result depends only on the angle between them, so the attention score depends on how far apart the words are, not on where they sit in absolute terms.
Different pairs spin at different speeds, like the hands of a clock, which lets the model tell both near and far distances apart, and the influence between words tends to fade with distance.
Idea · §1–2
Attention is blind to order
In a Transformer, each word decides how much to pay attention to each other word by comparing two lists of numbers, its query and the other word's key. The comparison is a dot product: multiply matching entries and add them up. Nothing in that calculation knows which word came first. "Dog bites man" and "man bites dog" would get identical scores.
The original Transformer fixed this by adding a unique position pattern to each word before anything else happens, like stamping a page number onto every page. That works, but it describes absolute position ("I am word 37"). For language, what usually matters is relative position: "the adjective right before me", "the subject a few words back". Other researchers had added relative-position tricks inside attention, but they were fiddly, and many didn't fit with faster variants of attention.
The question this paper asks: is there a way to inject position so that the comparison between two words automatically depends only on their distance?
TakeawayAttention needs position information added, and what matters most for language is the distance between words rather than their absolute positions.
Check yourself
- Why can't plain attention tell "dog bites man" from "man bites dog"?
- Its scores come from comparing word contents only. Reordering the words reorders the scores but doesn't change any of them, so nothing marks who did what to whom.
Idea · §3.2.1 · Eqs. 12–13
Turn the arrows by position
Take two numbers from a query and treat them as an arrow on a flat page: the first number is how far right it points, the second how far up. Now rotate that arrow by an angle that depends on the word's position: word 1 turns by a little, word 2 by twice as much, word 10 by ten times as much. Do the same to the keys.
Here is the key fact: the dot product of two arrows depends on their lengths and the angle between them, nothing else. If the query is turned by "position × step" and the key by "its position × step", the angle between them changes by "difference in positions × step". So the score depends on the distance between the words. Shift both words along by the same amount, and both arrows turn by the same extra angle, leaving the angle between them, and the score, unchanged.
Try to break it below.
TakeawayRotating query and key by angles proportional to their positions makes their dot product depend only on the difference in positions.
Check yourself
- A query at position 7 and a key at position 4 give some score. What other pair of positions gives the same score (same content)?
- Any pair three apart in the same order, such as 103 and 100. Only the offset m − n = 3 matters.
- Does rotating an arrow change its length?
- No. Rotation only changes direction, so the word's content "strength" is untouched; only the angle carries position.
Idea · §3.2.2 · Eq. 15
Many hands, many speeds
One rotating arrow has a problem: angles wrap around. If the step is a quarter turn, then positions 0, 4, 8 … all point the same way and look identical. A real query has many numbers, so RoPE splits it into many pairs, and each pair spins at its own speed. The first pair spins fast (one radian per word), and each later pair spins slower than the one before, down to pairs that take tens of thousands of words to complete a turn.
That is exactly how a clock tells time: the second hand distinguishes nearby moments, the hour hand distant ones, and together they are unambiguous. The fast pairs let attention resolve "one word back" versus "two words back"; the slow pairs keep far-apart positions distinct.
TakeawayDifferent pairs rotate at geometrically spaced speeds, so together they encode both short and long distances without ambiguity.
Check yourself
- Why not make every pair rotate fast?
- Fast rotations wrap around after a few positions, so distant positions would be confused with near ones. Slow pairs are needed to separate long distances.
Idea · §3.3 · §3.4.3 · Fig. 2
Side benefits: fading with distance, no extra parts
The paper points out three bonuses.
- Fading with distance. Because the many pairs spin at different speeds, for far-apart words their contributions point in scattered directions and partly cancel. The authors derive an upper limit on the attention score and show numerically that it tends to shrink as distance grows, matching the intuition that far-away words usually matter less.
- Nothing to learn. The rotations are fixed formulas, so RoPE adds no new trainable numbers to the model.
- Works with fast attention. Some faster forms of attention ("linear attention") never compute the full table of scores, which ruled out many earlier relative-position methods. Rotation is applied to each query and key separately, so it fits.
TakeawayRoPE gives relative positions for free: no new parameters, a natural fade with distance, and compatibility with efficient attention variants.
Idea · §4 · What came next
Modest results, enormous adoption
The paper's own experiments are modest. In translation it scored slightly above the standard Transformer. When pre-training a BERT-style model, the loss went down faster. On a Chinese legal-document task, the RoPE model handled longer inputs (1,024 tokens) better than models limited to 512. The authors openly admit they cannot fully explain why it trains faster.
What made RoPE famous was what happened next. Open-source groups tried it in large language models in 2021, found it worked well, and it spread: PaLM, LLaMA and most open models since use it.
TakeawayRoPE's paper showed small gains, but its simplicity and relative-distance property made it the standard position method of modern open LLMs.
Abstract, restated
Position encoding is essential in Transformers. After surveying ways of integrating positional information, the paper proposes Rotary Position Embedding (RoPE), which encodes absolute position with a rotation matrix while incorporating explicit relative-position dependency in the self-attention formulation. RoPE offers flexibility of sequence length, decaying inter-token dependency with increasing relative distance, and the ability to equip linear self-attention with relative position encoding. The resulting RoFormer is evaluated on long-text classification benchmarks and consistently beats alternatives; it is integrated into Hugging Face Transformers. Su, Lu, Pan, Murtadha, Wen, Liu (Zhuiyi Technology), arXiv 2104.09864, 2021; later published in Neurocomputing (2024).
Section · §2 · Eqs. 1–11
Absolute, relative, and the goal
With qm = fq(xm, m), kn = fk(xn, n), self-attention weights are softmax(qm⊤kn/√d). Absolute schemes (§2.2) set f(xi, i) = W(xi + pi) with pi learned or sinusoidal. Expanding qm⊤kn then gives four terms (content–content, content–position, position–content, position–position), and relative schemes (§2.3) edit these: Shaw et al. (2018) add learned, clipped relative embeddings to keys and values; Transformer-XL (Dai et al., 2019) replaces pn by sinusoidal p̃m−n with trainable vectors u, v; T5 (Raffel et al., 2020) reduces position to a learned scalar bias bi,j; DeBERTa (He et al., 2020) keeps the two cross terms with relative embeddings.
All of these act on the expanded score, so they do not compose with kernelised linear attention, which never forms qm⊤kn. RoPE instead asks for functions such that
Section · §3.2 · Eqs. 12–16, 34
The rotary solution
For d = 2, identify ℝ2 with ℂ. Then
§3.4.1 derives this as the solution of Eq. 11 under natural initial conditions (Eq. 33). In real form it is a 2×2 rotation by mθ applied to Wqxm (Eq. 13). For general even d, split the vector into d/2 pairs and rotate each with its own frequency:
since RΘ,m⊤RΘ,n = RΘ,n−m for commuting orthogonal blocks. Orthogonality also keeps norms, which the paper credits with stability. The frequencies reuse the sinusoidal schedule of the original Transformer. (The paper writes θi = 10000−2i/d in §3.3 and 10000−2(i−1)/d in Eq. 15; the latter, starting at θ1 = 1, is the usual convention.) Because R is block-sparse, Eq. 34 computes it elementwise:
Note RoPE is applied to queries and keys only (not values), in every layer, rather than once at the input.
Section · §3.3 · §3.4.3 · Eqs. 18–19, 35–37 · Fig. 2
Long-term decay and linear attention
Group entries in pairs and write hi = q[2i:2i+1]k*[2i:2i+1] and Sj = Σi=0j−1 ei(m−n)θi. Abel summation gives
The paper plots the "relative upper bound" (1/(d/2)) Σi=1d/2 |Si| against relative distance and observes that it decays (Fig. 2). Recompute it here:
Linear attention (Eqs. 17–19). For kernelised attention Attention(Q,K,V)m = Σn φ(qm)⊤φ(kn)vn / Σn φ(qm)⊤φ(kn), RoPE multiplies RΘ,mφ(qm) and RΘ,nφ(kn) in the numerator only, leaving the denominator unchanged to avoid division by zero. The numerator can then contain negative terms, so the weights are no longer a strict probability distribution; the authors argue this still models value importance.
Check yourself
- Why does a larger base stretch the decay?
- All θj except θ0 shrink, so the phases (m − n)θj stay aligned for longer and the partial sums |Sj| stay large out to larger distances.
- Why can't T5-style relative biases be used with linear attention?
- They add a term bm−n to the explicit score, which linear attention never materialises; RoPE acts on each q and k before the kernel product.
Section · §4 · Tables 1–5 · Fig. 3
Experiments
- Machine translation (§4.1, Table 1). WMT 2014 En–De: Transformer-base 27.3 BLEU, RoFormer 27.5.
- Pre-training (§4.2, Fig. 3 left). Replacing BERT's sinusoidal/learned absolute positions with RoPE in masked-LM pre-training gives faster loss convergence; no final numbers are tabulated.
- GLUE (§4.3, Table 2). Mixed: RoFormer is higher than BERT on MRPC (89.5 vs 88.9), STS-B (87.0 vs 85.8) and QQP (86.4 vs 71.2), lower on SST-2 (90.7 vs 93.5), QNLI (88.0 vs 90.5) and MNLI (80.2/79.8 vs 84.6/83.4). The BERT column matches BERT-base's GLUE test-server results from the BERT paper, so the comparison may not be like for like (the large QQP gap in particular).
- Performer + RoPE (§4.4, Fig. 3 right). On Enwik8 language modelling, adding RoPE to linear-attention Performer converges faster to lower loss.
- Chinese long text (§4.5, Table 5). CAIL2019-SCM (legal case matching) test accuracy: BERT-512 67.77%, WoBERT-512 68.10%, RoFormer-512 68.29%, RoFormer-1024 69.79%. The gain comes mostly from the longer context RoPE was used with.
Stated limitations (§4.5.5). No thorough explanation of why RoPE converges faster than other position encodings, despite the relative-position and decay properties; no faithful explanation of its better long-text performance; and pre-training cost limits the experiments.
Section · Lineage
From a Chinese BERT variant to the LLM default
- Origin and adoption. Su first described the method on his blog in early 2021. EleutherAI tested it and used it in GPT-J and GPT-NeoX (2021–22); PaLM (2022), LLaMA (2023) and most later open models followed. GPT-NeoX applied it to only a fraction of each head's dimensions ("partial rotary").
- Context extension. Models extrapolate poorly beyond their training length because low-frequency pairs reach unseen angles. Position Interpolation (Chen et al., 2023) rescales m → m·L/L′; "NTK-aware" scaling and YaRN (Peng et al., 2023) rescale frequencies non-uniformly; later models simply train with larger bases (Code Llama used 106, Llama 3 used 5·105).
- Alternatives. ALiBi (Press et al., 2021) adds a linear distance penalty to scores and extrapolates better zero-shot; some recent models interleave layers with no positional encoding at all.
- Interpretation. Later analyses (e.g. Barbero et al., 2024) suggest the high-frequency pairs carry positional "patterns" while low-frequency pairs act as near-position-free semantic channels, one reason partial-rotary and frequency-truncated variants work.
Reference
Glossary
- Absolute PE
- A position vector added to each token embedding at the input.
- Relative PE
- Position information that enters attention only through offsets m − n.
- RoPE
- Rotary position embedding: rotate each 2-D pair of q and k by mθi.
- θi
- Per-pair angular frequency, 10000−2(i−1)/d.
- Base
- The 10000 in θi; larger values slow all but the first pair.
- Toeplitz
- A matrix constant along each diagonal; the RoPE position effect has this form.
- Linear attention
- Attention computed through feature maps φ in O(n) without forming the n × n score matrix.
- RoFormer
- The paper's name for a Transformer that uses RoPE in place of absolute position embeddings.
- Position interpolation
- Scaling positions down so a longer context maps into the trained range.