Transformer 3D
About this app
The GPT-2 architecture to walk through: residual stream through 12 blocks, masked attention as bridges, MLP d → 4d → d, unembedding and softmax, one token at a time; parameters per component and GPT-3 for comparison.
How GPT-2 is built
GPT-2 small splits text into tokens from a vocabulary of 50,257 entries. An embedding table turns each token into a vector of d = 768 numbers, and a learned position embedding is added. This residual stream passes through 12 identical blocks: each sublayer reads it through a LayerNorm and adds its result, x ← x + Attn(LN(x)), then x ← x + MLP(LN(x)). Only the masked attention lets positions exchange information; the MLP expands each position on its own from d to 4d and back.
What the app shows
Every token has its own lane, the residual stream. The 12 blocks lie across it like bridges; the attention spans all lanes, and the grid on top of it is the causal mask. At the end only the last position becomes next-token probabilities through unembedding and softmax. Clicking a part explains in the component panel what it does and how many parameters it has. Without a model the app shows example values; on a laptop or desktop you can load the real GPT-2 (124M) in the browser.
Things to try
Start a run, in slow motion if you like, and follow a token from the embedding to the softmax. “Next token” appends the result and shows the autoregressive loop. Switch off the causal mask: every position then sees later tokens too, as in an encoder. In the model panel you count parameters, roughly 12·L·d² for the blocks plus embeddings, and switch to GPT-3 with 96 blocks and d = 12,288.
Frequently asked questions
- How is a transformer like GPT built?
- A GPT model consists of embedding tables for tokens and positions, a stack of identical blocks and an output layer. Each block contains masked self-attention and an MLP, each with a LayerNorm before it and a residual addition after it. Finally, the values of the last position are turned into next-token probabilities by a softmax.
- What is the residual stream?
- The residual stream is the vector of each position that runs from the embedding to the output. Every sublayer reads it and adds its result; nothing is replaced. The stream thus collects the contributions of all blocks, and during training the gradient flows freely through the whole network.
- What is the causal mask in attention for?
- The causal mask makes each position attend only to itself and earlier positions. Weights for later positions are set to −∞ before the softmax and thus become zero. This lets a model practise predicting the next token at every position at once during training without peeking.
- What can I try in this app?
- You send tokens through the 3D model of GPT-2, click on parts and read their role and parameter count. You can switch off the causal mask, compare GPT-2 small with GPT-3 and load the real GPT-2 on a computer.
Subject: Neural networks
More from Neural networks
- Agent loop
- Attention Visualiser
- Backpropagation — blame in reverse
- Basin of Attraction
- Your job under AI
- The temperature dial
- Diffusion model — images from noise (3D)
- Embedding Space Explorer (3D)
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps