Transformer 3D

About this app

The GPT-2 architecture to walk through: residual stream through 12 blocks, masked attention as bridges, MLP d → 4d → d, unembedding and softmax, one token at a time; parameters per component and GPT-3 for comparison.

Subject: Neural networks

More from Neural networks

• Agent loop
• Attention Visualiser
• Backpropagation — blame in reverse
• Basin of Attraction
• Your job under AI
• The temperature dial
• Diffusion model — images from noise (3D)
• Embedding Space Explorer (3D)

A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps