How Models Know Word Order

Attention is order-blind by construction

Self-attention compares every token with every other token through dot products, and a set of dot products has no idea which token came first. Feed it these two sentences with plain word embeddings and it sees the exact same multiset of vectors:

Same three embeddings, radically different meaning. Every transformer therefore injects position into the computation. Two families dominate: add a positional vector to each embedding (sinusoidal, learned), or rotate the query and key vectors by a position-dependent angle (RoPE, used by Llama, Qwen, and most current open models). You will build both below.

Lab 1: the sinusoidal encoding, computed from the formula

PE(pos, 2i)   = sin( pos / base^(2i / d_model) )
PE(pos, 2i+1) = cos( pos / base^(2i / d_model) )    base = 10000 in the paper

Every cell below is that formula evaluated live for the sliders you set: nothing is precomputed. Left columns cycle fast (wavelength of a few positions), right columns cycle slowly, so each row becomes a unique fingerprint the model can add to its token embedding. Shrink the base and watch the slow columns speed up; grow d and watch new slow columns appear on the right.

Lab 2: RoPE, position as rotation

Adding a position vector changes what a token IS. Rotary embeddings instead change how tokens MEET: split the query and key vectors into 2D pairs, rotate each pair of q by m times its frequency and each pair of k by n times the same frequency, then take the ordinary dot product. The rotations cancel except for their difference, so the score depends only on the offset m minus n. This toy head has 4 dimensions (two pairs); every number is computed from the vectors and angles on screen.