Transformer inside
About this app
A small GPT runs in the browser, every number visible: tokens, embedding, Q·Kᵀ and the attention matrix with mask, residual stream, MLP and probabilities with temperature and top-k/top-p; zooms for every computation step.
With explain mode: an AI tutor walks you through the app, points out what matters in the scene and answers your questions.
How a transformer computes the next word
A language model like GPT only ever predicts the next token. Each token first becomes a vector: token embedding plus position embedding. Then several transformer blocks follow. In self-attention every token forms a query and compares it by dot product with the keys of the tokens before it; divided by the square root of the head dimension and normalised with softmax, this gives weights that mix the values. The causal mask forbids looking into the future. An MLP then processes each token on its own, and both steps add their result to the residual stream. Finally the model compares the last vector with all word embeddings; softmax gives the probabilities.
What the app shows
A small GPT built like GPT-2 runs here: 4 blocks, 4 attention heads, 64 dimensions, 436 words. Every intermediate value is visible: vector strips per token, query and keyᵀ, the attention matrix with hatched mask, bands from the values to the selected token, the residual stream, the MLP with 256 neurons and, on the right, the probabilities of the next words. Four zooms show each computation step with numbers: the embedding matrix, Q·Kᵀ ÷ √16 and softmax, GELU in the MLP and the logits of all words.
What you can try
Pick an example or type your own text and keep drawing word by word. In "The cat drinks milk. It" you can find the head that looks from the pronoun to the cat. Change the temperature between 0.1 and 2 and watch the distribution become peaked or flat. Switch between top-k and top-p and see which words fall below the cut line. Clicking a word on the right appends it.
Frequently asked questions
- What does attention do in a transformer?
- It lets every token gather information from the tokens before it. Query and key give one value per pair by dot product, softmax turns these into weights, and the weighted sum of the values is the token's new vector.
- What does temperature do in language models?
- The logits are divided by the temperature before the softmax. Small values make the distribution peaked and the text predictable; large values flatten it and allow rare words. The model's knowledge does not change.
- What is the difference between top-k and top-p?
- Top-k keeps the k most likely words. Top-p keeps the smallest group of most likely words whose probabilities add up to at least p. The next word is then drawn from the words that remain.
- Does the app run a real language model?
- Yes, a small GPT with 229,504 parameters runs directly in the browser. It is trained on short German and English sample texts and is only reliable on those topics; GPT-2 small has 124 million parameters.
Subject: Neural networks
More from Neural networks
- Agent loop
- Attention Visualiser
- Backpropagation — blame in reverse
- Basin of Attraction
- Your job under AI
- The temperature dial
- Diffusion model — images from noise (3D)
- Embedding Space Explorer (3D)
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps