Generation is fast because attention keys and values for past tokens are computed once and stored. Watch a real cache fill cell by cell, size it with a live bytes calculator, and measure the attention work it saves.
Sign in to save progress
Why the past can be cached at all
attention(Q, K, V) = softmax( Q K^T / sqrt(d) ) V per layer, each token i produces: q_i (used once) k_i, v_i (reused forever)
K and V never change
Causal masking means past tokens never see the future, so their keys and values are identical no matter how many tokens follow. Compute them once, write them to GPU memory, and every later step just reads them back.
Queries are not cached
A query is used exactly once, when its own token attends over the cache. Only K and V are stored, which is why it is a KV cache and not a QKV cache.
The bill is memory
The cache grows linearly with context length and with every sequence in the batch. At serving scale it, not the weights, is often what caps how many requests fit on a GPU.
Watch the cache fill during decoding
Each row is one layer of a toy transformer, each column one position, and each cell holds that position's keys and values for all heads. Prefill computes the whole prompt in one parallel pass, then decoding writes exactly one new column per generated token. Press Generate, or Step through it.
Ready0 of 18 positions cached, 0 B of 216.0 KB total.
One cell (layer x position)
2.0 KB
2 tensors x 8 heads x 64 dims x 2 B
One token (all layers)
12.0 KB
2.0 KB x 6 layers, written once, never recomputed
Cache now
0 B
12.0 KB x 0 cached positions
Token strings are illustrative; the grid mechanics and every byte count are computed from the toy configuration above.
Size a real cache: the bytes calculator
Now scale the same formula to production shapes. Every number below is computed live from the sliders. Try a preset, then push the sequence length up and watch the cache walk past whole GPUs. Then switch the attention variant: grouped-query attention (GQA) keeps all query heads but shares K/V across groups, shrinking only the cache.
Cache dtype
Attention variant (KV heads: 32)
KV cache size (batch 1)
2.00 GB
2 (K+V) x 32 layers x 32 kv heads x 128 dims x 4k tokens x 2 B = 2,147,483,648 bytes
Cache alone as a share of GPU memory (weights, activations, and framework overhead still need their own room):
GPU capacities from NVIDIA published product specifications.
What the memory buys: attention FLOPs, with and without the cache
Without a cache, every generated token re-runs attention for the entire sequence, so the per-step cost grows quadratically. With the cache, each step computes K/V for one token and reads the rest, so it grows linearly. The chart uses the layers, heads, and head dimension you set in the calculator above.
Total, with cache
154.82 GFLOPs
Total, without cache
28.01 TFLOPs
Attention work saved
180.9x less
Counts cover attention scores and weighted sums only (QK^T and AV from the attention formula), per layer per query head. Projections, MLP blocks, and softmax are excluded on both sides, so the ratio isolates what caching saves.