Attention Visualiser
About this app
Self-attention made visible: query, keys, softmax(QKᵀ/√d)·V, five specialised heads, a matrix heat map with causal mask, a second layer and a by-hand calculation with three tokens.
How attention computes
Self-attention is the core of the transformer. Each token carries three vectors: a query (“what am I looking for?”), a key (“what do I offer?”) and a value (“my content”). The query is compared with the keys of all allowed tokens by dot product, divided by √d_k and turned into weights by a softmax, so they add up to 1. The output is the weighted average of the values: Attention(Q, K, V) = softmax(QKᵀ/√d_k)·V. In GPT-style models a causal mask blocks any look at later tokens.
What you see
In the 3D view an example sentence stands as a row of tokens; arcs show the attention row of the selected query token, with thickness as weight. Five idealised heads can be toggled one by one: previous token, sentence start as a sink, syntax, reference (such as “it” → “animal”) and content words. A heatmap shows the full matrix, and with the causal mask the upper triangle is blocked. A second layer mixes vectors that are already mixed. The by-hand mode recomputes the “the cat sat” example with d_k = 2 exactly in six steps.
Things to try
Pick one of five example sentences and move the query position through it. Toggle heads on and off and watch matrix and arcs change. Remove the causal mask: every token then also sees to the right, as in an encoder like BERT, and predicting the next word would be cheating. In by-hand mode, drag the tip of the query vector and switch off the division by √d_k; without it the weights become sharper.
Frequently asked questions
- What is self-attention?
- Self-attention is a mechanism that lets each token in a sequence take in information from the other tokens of the same sequence. Its query is compared with the keys of all tokens, and the values are averaged with the resulting weights. The result is a context-enriched vector for every token.
- Why divide by √d_k in attention?
- Dot products of random vectors spread more widely as the dimension d_k grows. Without scaling, scores would get very large for big d_k, and the softmax would put almost all weight on a single token. Dividing by √d_k keeps the variance around 1 and the weights softer.
- What does the causal mask do in GPT?
- The causal mask sets all scores for tokens to the right of the current position to −∞, so they receive weight 0 after the softmax. This stops the model from seeing the next word in advance during training. Encoder models such as BERT skip the mask and see the whole sentence.
- What can I try in this app?
- You choose sentence, query token and active heads and see attention as 3D arcs and as a matrix. You toggle the mask and a second layer, and in by-hand mode you recompute every step from the dot products to the weighted sum.
Subject: Neural networks
More from Neural networks
- Agent loop
- Backpropagation — blame in reverse
- Backpropagation: learning from one mistake
- Basin of Attraction
- Your job under AI
- The temperature dial
- Diffusion model — images from noise (3D)
- Embedding Space Explorer (3D)
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps