Click every block in the decoder stack to see what it does. GPT-style transformers stack these blocks N times to process and generate text.
Sign in to save progress
Click any block to expand its explanation. Blocks 3–6 repeat N times (96 layers in GPT-3; GPT-4's architecture is unpublished).
× N LAYERS (e.g. 96 for GPT-3)
0 / 8 blocks explored
Click a block on the left to see what it does
Watch a token travel through the full transformer pipeline.
Input text
"Hello"
Token ID
[9906]
Embedding
[0.12, −0.34, ...]
After Attention
context vector
After FFN
transformed vector
Logits
{world:4.2, there:3.1}
Probs
{world:32%, there:18%}
| Model | Layers | Heads | Dim | Params |
|---|---|---|---|---|
| GPT-2 | 12 | 12 | 768 | 117M |
| GPT-3 | 96 | 96 | 12,288 | 175B |
| LLaMA 3 | 32 | 32 | 4,096 | 8B |
Key Insight
The “Attention Is All You Need” paper (2017) by Vaswani et al. replaced RNNs entirely with attention. It has over 100,000 citations, making it one of the most influential papers in AI history.
Explore 6 more blocks and play the animation to complete this guide.