Rotary position embeddings

Module 2 said attention ignores order, so positions must be injected. The older method added a position vector to each token's embedding. RoPE instead rotates the query and key vectors by an angle that grows with the token's position. It rotates the features in pairs, and each pair at its own speed.

def rope(x, positions, base=10000.0):
    freqs = base ** (-np.arange(0, d, 2) / d)
    ang = positions[:, None] * freqs[None, :]
    # rotate each (even, odd) pair of features by its angle

A tiny example: take the vector [1, 0, 0, 0]. The first pair rotates by 1 radian per position. At position 1 it becomes [0.54, 0.841, 0, 0], and at position 2, [-0.416, 0.909, 0, 0]. The length stays 1 each time; only the direction turns. Slower pairs (the later ones) turn by smaller angles, so the model gets both fine and coarse position signals.

The payoff is a clean property, and the tests prove it with real numbers:

• Rotation does not change a vector's length.
• The score between a query at position m and a key at position n depends only on the distance m - n. In the demo, the score is 5.331525 for positions (3, 1), (10, 8) and (100, 98).

So the model sees relative distance directly, which tends to generalise better. Extending a model to longer inputs than it was trained on becomes a matter of adjusting the rotation speeds, which several long-context techniques do. The base number sets the speeds: a larger base makes the slow pairs turn more slowly (Medium 3 explores this).