NeuroNomixer
  • Home
  • Blog
  • Visual Guides
  • Authors
  • Contact
Sign InSign Up
HomeBlogAuthorsContactPrivacy Policy

© 2026 NeuroNomixer — Built with Next.js & Tailwind CSS

Visual Guides/Self-Attention: How Transformers Focus
LLMs

Self-Attention: How Transformers Focus

Select a sentence and watch the attention heatmap light up. Step through Query, Key, and Value vectors to see how transformers resolve word ambiguity through context.

Exploration Progress1/3 sentences · 1/3 QKV steps

Sign in to save progress

1. Choose a Sentence

2. Attention Heatmap

Click a row to highlight that token's attention pattern. Hover a cell to inspect the weight.

The
cat
sat
on
the
mat
The
50%
10%
10%
10%
10%
10%
cat
10%
40%
30%
10%
0%
10%
sat
10%
30%
30%
10%
10%
10%
on
20%
10%
10%
40%
10%
10%
the
50%
10%
0%
10%
20%
10%
mat
10%
10%
20%
10%
10%
40%
← attended-to tokens (Keys)

Y-axis: attending token (Query) · X-axis: attended-to token (Key) · Color intensity = attention weight

3. Q, K, V Explained

Q

Query (Q)

What am I looking for?

Q = X · W_Q

Each token projects itself into "query space", essentially asking: what context do I need to understand myself?

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V

Scaling by √dₖ prevents vanishing gradients in high dimensions.

4. Multi-Head Attention

Multiple heads run in parallel, each learning different relationship types simultaneously.

Head 1: Syntactic

The
bank
near
the
river
The
50
10
20
10
10
bank
10
40
30
10
10
near
10
20
50
10
10
the
20
10
10
40
20
river
10
10
20
10
50

Head 2: Semantic

The
bank
near
the
river
The
30
20
10
20
20
bank
10
10
20
10
50
near
10
30
20
20
20
the
30
10
10
30
20
river
10
50
10
10
20

Head 3: Positional

The
bank
near
the
river
The
40
30
10
10
10
bank
30
30
30
10
0
near
10
20
40
20
10
the
10
10
30
30
20
river
10
0
20
30
40

Multiple heads capture different relationships simultaneously.

Key Insight

The attention mechanism runs in O(n²) time: processing a 10K token sequence requires 100M attention computations. This is why context window scaling is challenging.

← All GuidesTransformer Architecture →