RoPE — A First-Principles Field Guide

01 · The problem

Attention doesn't know where anything is

Self-attention decides how much one token should attend to another using a single number — a dot product. That number has no idea how far apart the two tokens actually sit.

Inside every attention layer, each token is turned into two vectors: a query — the question a token is silently asking ("what am I looking for?") — and a key — the advertisement a token holds up ("here's what I've got"). How well they match is just a dot product: multiply matching numbers together, add them up.

Attention score between token i and token j $$\text{score}(i,j) = q_i \cdot k_j = \sum_{t} q_i^{(t)}\, k_j^{(t)}$$

swap which token sits where, and if q_i and k_j don't change, score(i, j) doesn't move at all

Try it — push "mat" further from "cat"

Each cell is still the same two content vectors. Only the distance between them changes.

distance, cat → mat

4 tokens

raw score, no position

0.560

⚠

What actually breaks: "The cat sat on the mat" and "The cat, having wandered the whole garden and back, eventually sat on the old mat" would score the "cat"–"mat" pair identically — one pair is close, the other isn't, and the raw formula can't tell. Position has to be injected from somewhere outside q_i · k_j.

What m and n actually mean

Before the notation gets any thicker: number a sentence's tokens starting from zero — that's each token's position, just its slot in the line. Recall score(i,j) = q_i · k_j: token i is the query, asking; token j is the key, being looked at. Position-embedding papers rename these two positions with single letters, and it's worth memorising which is which:

query position

the position of the token doing the asking — the query

key position

the position of the token being looked at — the key

If "cat" (position 1) is asking about "mat" (position 5), then m = 1 and n = 5, and the gap between them is n − m = 4. Click any two words below — the first click sets m, the second sets n.

m — query position

1 · "cat"

n — key position

5 · "mat"

gap = n − m

💡

One letter to keep separate: further down you'll meet a third letter, i. It has nothing to do with sentence position — it names which slice of a single vector's numbers is being discussed, not which word. m and n answer "which word." i answers "which piece of that word's vector."

The 2017 fix — stamp a position onto the meaning

Every token starts life as an embedding — a vector encoding its meaning, computed purely from the word itself, with no idea where it landed. The original transformer's fix: compute a second vector that depends only on position — built from sine and cosine waves at different frequencies — and add it straight onto the embedding.

Sinusoidal absolute position embedding, added once at the input $$PE_{(pos,\,2t)} = \sin\!\Big(\frac{pos}{10000^{2t/d}}\Big) \qquad PE_{(pos,\,2t+1)} = \cos\!\Big(\frac{pos}{10000^{2t/d}}\Big)$$

the resulting vector now differs by position — but meaning and position can never be cleanly separated again

⚠

Two real costs: the position pattern is mixed permanently into the same numbers that carry meaning, so the model must carry that mixture through every layer. And what gets encoded is absolute position — the model has to indirectly infer that word 1 and word 5 are four apart by comparing two absolute stamps, and a learned table that only saw positions up to 2,047 during training has no stamp at all for position 3,000.

02 · The idea

Instead of stamping, spin

RoPE leaves the meaning vector completely untouched. It changes the direction the query and key point in — like turning a clock hand — by an angle that depends on position.

Picture a query or key vector as a hand on a clock face — sitting at some angle, some fixed distance from the center. RoPE's entire idea: before comparing a query against a key, spin each hand by an angle proportional to its token's position. Position 0 doesn't spin at all — zero times anything is zero. Position 5 spins five times as far as position 1. The spin only ever depends on the position number, times a fixed rate.

position 0 → angle 0.00 rad

position

drag the slider — the hand spins, but never grows or shrinks

Three things worth sitting with. It only touches q and k — the value vector, and the original embedding, are never rotated; position lives entirely inside the attention step, recomputed fresh in every layer. It never changes the vector's length — spinning a clock hand moves where it points, not how long it is, so a word's underlying content strength survives untouched. There's nothing to learn — the spin rate is a fixed formula decided before training starts, not a table the model adjusts.

The one formula you need — rotating a 2D pair

Draw a circle of radius r. Any point on it is (r·cos θ, r·sin θ) for some angle θ. Rotating a vector (x₀, x₁) by θ means sliding it to a new angle, θ further around that same circle — the radius never changes. The compact way to write "compute the new coordinates" is the rotation matrix:

2D rotation matrix $$R(\theta) = \begin{bmatrix}\cos\theta & -\sin\theta \\[2px] \sin\theta & \cos\theta\end{bmatrix} \qquad \begin{bmatrix}x_0' \\ x_1'\end{bmatrix} = R(\theta)\begin{bmatrix}x_0 \\ x_1\end{bmatrix}$$

every RoPE step, at every scale, is exactly this one box — applied to many pairs in parallel

03 · Real vectors have hundreds of dimensions

One clock isn't enough — build a whole wall of them

Query and key vectors aren't 2 numbers, they're often hundreds. RoPE handles that by chopping the vector into pairs, and giving each pair its own personal spin rate.

You can't spin a 768-number vector on a single circle. So RoPE slices it into consecutive pairs of dimensions — (0,1), (2,3), (4,5) … — and treats each pair as its own tiny 2D vector, sitting on its own circle. For a vector with d dimensions, that's d/2 independent little clocks. Each one, indexed by i = 0, 1, 2, … (the "which slice" letter, unrelated to m and n), spins at its own fixed rate:

Spin rate for pair i, out of d/2 pairs $$\theta_i = \text{base}^{-2i/d}, \qquad i = 0, 1, \dots, \tfrac{d}{2}-1$$

base is a constant, almost always 10000. d is the vector's total dimension count. Plug in d = 8 (four pairs) and base = 10000:

pair i	θᵢ = 10000⁻²ⁱ/⁸	full turn every…
0	1.000	~6.3 tokens — tracks nearby words
1	0.100	~62.8 tokens
2	0.010	~628 tokens
3	0.001	~6,283 tokens — tracks long-range structure

position m 0

drag the position — watch the fast pairs whirl while the slow ones barely move

Block diagram — splitting, rotating, reassembling

four pairs shown for clarity — a real 768-dim vector runs 384 of these boxes in parallel

Pair 0 spins fastest — about 1 radian per step. By word 6 or 7 it's already spun most of the way around: exquisitely sensitive to nearby words, nearly useless for telling word 200 apart from word 400 — both look like blur. Pair 3 crawls — 0.001 radian per step, needing roughly 6,283 words for one full turn. It can't tell word 1 from word 2 apart at all, but it patiently tracks very long-range position. Read together, fast and slow hands behave like a mechanical wristwatch — a second hand, a minute hand, an hour hand — pinning down a moment precisely whether it happened a second ago or a year ago.

04 · At real scale

Where this actually sits in a real model

Every example so far used a toy 8-number vector. Here's the identical formula, run at the size real models use, and exactly where it slots into the attention pipeline.

Block diagram — RoPE's place in the pipeline

RoPE is inserted at exactly one place: right after Q and K are projected, right before they meet in the similarity score. V is never touched — it carries content, not position, all the way through.

two boxes, inserted once, per layer — that's the entirety of RoPE's footprint on the pipeline

Take a hidden size of 768 — a common round number in practice. Split into pairs, that's 384 independent little clocks — not 4. Same formula, just 96 times more of them:

Spin rate for pair i, out of 384 pairs $$\theta_i = 10000^{-2i/768}, \qquad i = 0, 1, \dots, 383$$

pair i	θᵢ	completes a full turn every…
0	1.000000	6.3 tokens
96	0.100000	62.8 tokens
192	0.010000	628.3 tokens
288	0.001000	6,283.2 tokens
383	0.000102	61,342.7 tokens

all 384 spin rates at once, log scale — pair 0 is fast, pair 383 is nearly frozen

Walk through one real token. Say "transformer" sits at position m = 100 in a long document, and its 768-number query vector has just been computed. RoPE slices it into 384 pairs and spins each one by 100 × θᵢ: pair 0 spins by 100 × 1.000 = 100 radians — almost 16 full rotations, so only the leftover fraction of a turn matters against a nearby token. Pair 383 spins by 100 × 0.000102 ≈ 0.0102 radians — a barely perceptible nudge, but every other position in the document gets its own slightly different nudge, so across a 50,000-token document this slowest pair alone still separates the beginning from the end.

Per-head, not per-vector

worked example

768 ÷ 12 heads = 64

a 768-dim hidden size split across 12 heads gives head dimension 64 → 32 pairs per head, not 384. θᵢ = 10000⁻²ⁱ/⁶⁴.

LLaMA-7B

4096 ÷ 32 heads = 128

head dimension 128 → 64 pairs per head. θᵢ = 10000⁻²ⁱ/¹²⁸.

general rule

d_head = d_model / h

RoPE always rotates the per-head dimension — plug in whatever head size your model actually uses.

🛠️

For the engineer reading the code: query and key vectors get reshaped into multiple attention heads first, and RoPE is applied per head, on the much smaller head dimension — never the full 768/4096-number hidden vector directly. One more gap between this derivation and most production code: everything above rotates adjacent coordinate pairs — (0,1), (2,3), (4,5) … Most real implementations (Hugging Face's and Meta's included) instead pair dimension i with dimension i + d/2 — the first half of the vector against the second half. It's the exact same rotation, just a different, equally valid pairing of coordinates — if you diff this guide's math against actual source and the indices look shuffled, that's why.

05 · The payoff

Rotate both by their own position — only the gap survives

This is the fact that makes the whole scheme worth building — and it costs zero learned parameters to get.

Put the pieces together. Instead of the position-blind score(i,j) = q_i · k_j, RoPE computes:

Score with RoPE applied $$\text{score}(m,n) = \big[R(m\theta)\,q\big]\cdot\big[R(n\theta)\,k\big]$$

Rotating two vectors by the same shared amount doesn't change the angle between them — only rotating them by different amounts does. Spin a protractor under two pencils resting on it, and the angle between the pencils never changes; only the difference in how far you turned each pencil individually would change it. The algebra says exactly this — a rotation matrix has one convenient property, reversing it is the same as rotating backward:

The identity that makes it work $$R(\theta)^\top = R(-\theta)$$

Chased through the dot product $$\big[R(m\theta)q\big]\cdot\big[R(n\theta)k\big] \;=\; q^\top R(m\theta)^\top R(n\theta)\,k \;=\; q^\top R(-m\theta)R(n\theta)\,k \;=\; q^\top R\big((n-m)\theta\big)\,k$$

The m and n on the left disappear on the right — only their difference, n − m, the gap between the two tokens, is left standing. Move both tokens ten slots to the right, keep the same 4-word gap, and the score doesn't move by a single decimal. Try it below: drag m and n freely, then lock the gap and slide both together.

query pos m 1

key pos n 4

q · "cat", rotated by m·θ

k · "mat", rotated by n·θ

gap = n − m

score q · k

−0.138

lock the gap, drag anywhere — the dials spin to new absolute angles but the score stays frozen

06 · A full numeric example

Same gap, four different sentences — one identical score

Not abstractions — real numbers, checked with actual matrix multiplication, not rounded to look tidy.

Say token "cat" produces the query q = [0.90, 0.25], and token "mat" produces the key k = [0.40, 0.80] — two ordinary content vectors, with no position baked in yet. Say the spin rate for this pair of dimensions is θ = 0.3 radians per position step. Four different placements, all with "mat" sitting exactly 3 slots after "cat":

m	n	gap	rotated q	rotated k	score
1	4	3	[0.786, 0.505]	[−0.601, 0.663]	−0.1376
5	8	3	[−0.186, 0.915]	[−0.835, −0.320]	−0.1376
0	3	3	[0.900, 0.250]	[−0.378, 0.811]	−0.1376
10	13	3	[−0.926, −0.120]	[0.260, −0.856]	−0.1376

Every rotated vector looks completely different — the arithmetic genuinely spins them to new coordinates each time — yet the dot product lands on exactly the same number, to four decimal places, every single time. Compare that to a score with no position information at all — the flat, 2017-style dot product, q · k = 0.5600, no matter the gap:

RoPE score, varies with gap no position info — always 0.560

Seeing it in code

rotary_position_embedding.py

def rope(x, pos, base=10000): d = x.shape[-1] i = torch.arange(d // 2) # which pair: 0, 1, 2, ... theta = base ** (-2 * i / d) # the d/2 fixed spin rates ang = pos * theta # angle = position * theta_i, per pair x1, x2 = x[..., 0::2], x[..., 1::2] # the (even, odd) coordinates of each pair xr1 = x1 * ang.cos() - x2 * ang.sin() # rotation matrix, first row xr2 = x1 * ang.sin() + x2 * ang.cos() # rotation matrix, second row return interleave(xr1, xr2) q = rope(q, pos); k = rope(k, pos) # applied to q and k — never to v, never to the embedding

theta is computed once from arithmetic, with no nn.Parameter in sight — nothing for gradient descent to touch. x1, x2 are literally the coordinates of each 2D pair; xr1, xr2 are exactly R(θ) applied to that pair, using each pair's own θᵢ. The last line is worth re-reading: q and k get rotated — v never appears in this function at all.

✓

Interview soundbite: RoPE rotates every query and key by a fixed angle set by its position, so the attention score becomes a function of the distance between two tokens — with zero learned parameters, and without ever touching the value vector or the original embedding.

07 · Consequences

What this buys you in practice

Every design choice above has a real, practical payoff — and one honest limitation.

Content strength survives untouched. Because rotation preserves length, a word's key can stay just as "confident" a hundred tokens in as it was at the start — position only ever changes direction, never magnitude. Zero extra parameters. There's no position table to store, and nothing extra for the model to overfit to. Relative, not absolute. The model only ever has to learn "how to react to a gap of 4," not "how to react to word #301 specifically" — which generalises far better across sentence lengths. Gentler with longer sequences — the angle m·θ is a formula, not a lookup, so it's still well-defined for a position the model never trained on.

⚠

The honest limit: in practice, raw RoPE still degrades somewhat past its training length — which is exactly why 2023–2024 long-context models layer extra scaling on top (NTK-aware scaling, YaRN, position interpolation) rather than replacing RoPE outright. RoPE is the substrate; those are patches for the extrapolation gap it doesn't fully close alone.

	2017 · absolute position embedding	now · rotary position embedding
Where applied	added to the token embedding, once, at the input	applied to q and k, freshly, inside every layer
Learned?	sometimes (a trainable table); sinusoidal itself is fixed	always fixed — zero parameters
Embedding magnitude	altered by the addition	exactly preserved — rotation only
What's encoded	absolute position	relative distance, directly
Long-context behaviour	poor — no representation past the trained length	better by construction, still commonly extended with scaling tricks

Who actually ships it

The full timeline

Every later idea responds to a limitation the previous one exposed — a straight line from "the model needs some notion of order" to "don't burn a parameter on it."

2017

Sinusoidal position embedding

"Attention Is All You Need" adds fixed sine/cosine patterns to the input embedding — the first fix for a permutation-invariant architecture.

2018–19

Learned absolute position tables

BERT and GPT-2 replace the fixed sinusoid with a trainable lookup table — flexible, but with a hard ceiling at the trained sequence length.

2018–19

Relative position representations

Shaw et al. (2018), then Transformer-XL (2019), fold relative offsets directly into attention — the direct conceptual ancestor of RoPE's "only the gap matters."

2021

RoFormer — Rotary Position Embedding

Su et al. rotate q and k by an angle proportional to position — relative distance falls out of the dot product algebraically, with zero parameters.

2021

ALiBi

Press et al.'s alternative: bias attention scores by a fixed penalty proportional to distance, instead of rotating vectors — simpler, also extrapolates well.

2022–23

GPT-NeoX, PaLM, LLaMA adopt RoPE

RoPE moves from one paper's proposal to the default choice for new open and closed-weight language models.

2023–24

NTK-aware scaling · YaRN

Extending trained models to far longer contexts by reinterpreting RoPE's frequency schedule after the fact, rather than retraining from scratch.

Decision guide

Do you actually need to think about this?

Answer these in order — by the second question you'll know what your architecture needs.

Are you training a new transformer from scratch?

→Yes → use RoPE by default. It's the safe, proven, zero-parameter choice every major open model now ships with.

→No, fine-tuning or serving an existing checkpoint → the position scheme is already baked in; you inherit whatever the base model uses.

Do you need much longer context than the model trained on?

→Yes → look at NTK-aware scaling or YaRN to stretch an existing RoPE model's effective range without retraining from scratch.

→No, typical lengths are fine → plain RoPE, no extra scaling needed.

Are you debugging a RoPE implementation against this guide's math?

→Check whether the code pairs (i, i+d/2) — the common "rotate-half" convention — rather than adjacent pairs (2i, 2i+1). Same rotation, different index layout.

→Confirm RoPE is applied per attention head, on d_model / h dimensions — not on the full hidden size.

🎯

The one-paragraph version: attention's raw dot product has no idea where anything sits. RoPE fixes that not by stamping a position onto meaning, but by rotating query and key vectors by an angle proportional to position — a rotation that costs zero learned parameters, preserves every vector's length exactly, and, thanks to one property of rotation matrices, makes the resulting attention score a function of relative distance alone. Everything since — ALiBi, NTK-aware scaling, YaRN — is a variation on that same two-line idea: fold position into the score directly, instead of into the embedding.