Multimodal models do not grow a separate vision brain. They slice the image into patches, linearly project each patch into the same embedding space as text tokens, and hand the transformer one mixed sequence. Draw below and watch it happen with real matrix math.
Step 1: The image is just numbers
This 24 x 24 grayscale image is a grid of brightness values between 0 and 1, nothing more. Draw on it with the pointer or pick a preset (the keyboard route). Everything runs in your browser; no pixel ever leaves this page.
Brush
Presets
The gold lines are the patch boundaries at the current patch size. Each cell inside a boundary becomes one entry of that patch's input vector.
Step 2: Patches become tokens in one sequence
The image is cut into non-overlapping patches. Each patch is flattened to a vector and multiplied by a projection matrix W to produce an 8-dim embedding, the exact operation a vision transformer performs. Here W is a fixed seeded random matrix, not learned weights; the flattening and the matrix multiplication are computed for real from your pixels. The patch tokens then join the text prompt tokens in a single sequence.
Image patches
9
Values per patch
64
Projection W
64 x 8
Sequence length
13 tokens
Purple strips: text token embeddings from a fixed seeded lookup table (standing in for a learned embedding table, with this simplified word-level tokenization). Teal and purple strip cells encode each embedding value: teal positive, purple negative, opacity by magnitude. Click any patch token for the exact numbers.
Step 3: Inspect one patch, number by number
Flatten, multiply, done. There is no convolution stack and no feature detector in the projection step: a single linear map takes raw pixels into the token embedding space.
No patch selected yet. Click any image patch token in the sequence above to see its pixels flattened, multiplied through the projection matrix, and turned into an embedding vector.
What patch size costs
Patch size is the resolution-versus-cost dial. Smaller patches keep more spatial detail per token but multiply the sequence length that attention must process. These bars are computed for the 24 x 24 image above plus its 4-token prompt.
Halving the patch side length quadruples the image token count: cost scales with (side / patch)². Production vision transformers work the same way at scale: ViT-Base/16 slices a 224x224 input into 16x16 pixel patches, which is (224 / 16)² = 196 image tokens per image (Dosovitskiy et al., 2020, "An Image Is Worth 16x16 Words").
One shared space
After projection, a patch token and a text token are the same kind of object: a vector of 8 numbers. Attention layers never check where a token came from; cross-modal understanding is just attention over one mixed sequence.
Axes are the first two of the 8 embedding dimensions, plotted directly (no dimensionality reduction). Redraw the image or move the patch-size slider and the teal points move, because they are recomputed from your pixels. The purple text tokens stay put: their embeddings come from the fixed lookup table.
Why linear projection is enough
The projection only needs to lift pixels into the model's vector space. All the heavy lifting, edges, shapes, objects, happens later inside the transformer's attention and MLP layers, trained end to end.
Position must be added back
Flattening patches destroys their layout. Real models add a positional embedding to every patch token so the transformer knows which patch came from where. This demo omits that step to keep the projection visible.
The same trick everywhere
Audio spectrograms, video frames, even protein structures ride the same rails: chop the input into pieces, project each piece into the token embedding space, concatenate with text, run the transformer.