Attention is a dictionary made differentiable: a query scores every key, softmax turns the scores into weights, and the weights blend the stored values. Drag the vectors to feel it, then train a real attention layer in your browser and watch the lookup sharpen.
A dictionary you can differentiate
A normal dictionary lookup is all or nothing: the key matches or it does not, and nothing about that choice can be nudged by a gradient. Attention replaces the exact match with a similarity score and the single winner with a weighted blend. Same idea, soft edges:
# hard lookup
memory = {"A": 1.3, "B": -1.9}
memory["B"] # exactly -1.9
memory["b"] # KeyError
# all or nothing, no gradient# soft lookup (attention) scores = [q . k1, q . k2] weights = softmax(scores) output = w1*v1 + w2*v2 # every value chips in, smoothly
Query
What you are looking for, as a vector. In the playground below it is the gold arrow you drag around.
Keys
What each stored slot advertises about itself. The dot product q·k measures how well a key matches the query: aligned directions score high, opposite directions score negative.
Values
What each slot actually contains. Values never enter the scoring; they are only blended by the softmax weights at the end.
In the RNNs & LSTMs guide a sequence had to squeeze through one hidden state, step by step, and old information faded on the way. Attention takes the opposite bet: keep every position around and let the model look back directly at whichever one it needs.
Step 1: drive the lookup by hand
Four keys store the values 10, 20, 30 and 40. Drag the gold query toward a key and watch its dot-product score rise, its softmax weight take over, and the output slide toward that stored value. Then drag a key itself: the same lookup changes because the memory changed. Every number on the right is computed live from the vectors you set.
Drag the gold query q or any key tip, or Tab to one and steer it with the arrow keys. Similar directions score high; opposite directions score negative. The dashed circle marks length 1.
Scores are multiplied by this before the softmax. Crank it up and the soft lookup approaches a hard argmax lookup; the weight entropy below shows how peaked the blend is.
Live: score, softmax, blend
Attention output = weighted blend of values
0.447×10 + 0.209×20 + 0.101×30 + 0.243×40
21.40
Step 2: let gradients learn the lookup
You just aimed the query by hand. The point of attention is that a network can learn to aim it. Below, a tiny attention layer plays a recall game 3,000 times with plain SGD from fixed seeds: real forward passes, real gradients through the softmax, right here in your browser. Watch the heatmap start as a uniform smear at chance and sharpen onto the matching slots.
The recall game
Each episode stores 4 slots. Every slot carries a type tag (A, B, C or D, each appearing once, in random order) and a random number in [-2, 2]. The model is then given a cue type and must output the number stored at the slot with the matching tag. One held-out example:
slot 1
D: 1.30
slot 2
B: -1.89
slot 3
A: -1.16
slot 4
C: -0.51
Cue C must recall -0.51. The gold border marks the matching slot the model must find on its own.
What is learnable
Two tiny embedding tables: Wq maps the cue type to a query vector and Wk maps each slot type to a key vector (4x4 numbers each, 32 parameters total). Scores are scaled dot products, q·k/√4, softmaxed into weights that blend the stored values. The loss is the squared recall error, and every gradient step below backpropagates through the softmax for real. Stored values pass through unweighted by any value projection, a simplification so the lookup itself is the only thing being learned. The √d score scaling is the convention from Vaswani et al. 2017, "Attention Is All You Need".
Training loss (EMA, log scale)
Squared recall error per episode; sampled every 25 episodes.
Run training to draw the loss curve.
Attention heatmap before training
Diagonal mean 0.247 (0.25 = uniform chance, 1.0 = perfect lookup)
| cue | slot 1 (A) | slot 2 (B) | slot 3 (C) | slot 4 (D) |
|---|---|---|---|---|
| A | 0.24 | 0.25 | 0.26 | 0.25 |
| B | 0.25 | 0.22 | 0.24 | 0.29 |
| C | 0.24 | 0.25 | 0.26 | 0.25 |
| D | 0.26 | 0.25 | 0.23 | 0.26 |
A perfect lookup is a bright diagonal: cue A attends to the slot typed A, and so on. Weights in each row sum to 1.
| Held-out evaluation (500 fresh episodes) | Untrained model | After training |
|---|---|---|
| Mean squared recall error | 0.9514 | run training |
| Mean weight on the matching slot | 0.247 | run training |
| Argmax finds the matching slot | 25.2% | run training |
Both columns are measured on the same 500 episodes from a seed the training never used. A model with uniform weights blends all four values equally, which is why the untrained row sits near chance.
Step 3: why does this train at all?
A Python dict solves the recall game perfectly, yet no gradient descent could ever have produced it. What makes the attention version trainable?
Where this goes next
This guide stopped, on purpose, at one query attending over one stored sequence: the bare mechanism. The famous variants are the same three moves arranged differently. When every position in a sequence issues its own query and attends over all the others, that is self-attention; run several lookups in parallel and you have multi-head attention; stack those layers and you are building transformers and language models. All of that lives in the LLMs category, starting with the Self-Attention guide.