Backpropagation: learning from one mistake
About this app
One whole learning round on a real network (64 → 8 → 8 → 10, 682 parameters): it takes a sloppy 3 for an 8. Error, loss as one square, slope per weight, blame layer by layer backwards, a ball rolls downhill over a real loss surface, and afterwards it recognises the 3. Draw yourself, change the learning rate, practise a thousand times.
With explain mode: an AI tutor walks you through the app, points out what matters in the scene and answers your questions.
What backpropagation computes
A neural network is wrong, and the error is condensed into a single number, the loss L = Σ(ŷᵢ − yᵢ)². Backpropagation then answers, for every weight, how steeply L rises if exactly this weight grows a little: the partial derivative ∂L/∂w. To do so the chain rule runs back from the output: δˡ = (Wˡ⁺¹)ᵀ δˡ⁺¹ ⊙ σ′(zˡ), and every weight receives ∂L/∂w = δ (at the target) times a (at the source). Backpropagation itself changes nothing. Only gradient descent w ← w − η·∇L moves all weights a step downhill at the same time.
What the app shows
The app computes with a real small network: 64 inputs for an 8×8 pixel image, two hidden layers of 8 neurons each, 10 outputs, 682 parameters in total. It recognises all ten clean digits but takes a sloppily written 3 for an 8 (0.92 against 0.14). In seven stations you see the error per digit, the error squares merging into one loss square with L = 1.59, real slices L(w) for single weights with their tangent, the blame colouring the network layer by layer backwards, and a real loss surface over which a ball rolls downhill. Afterwards the network recognises the 3.
Things to try
Step through the seven stations with next and start the next learning step with one more round. Draw your own digits straight into the pixel image, choose the correct answer and change the learning rate η between 0.05 and 3. In the blame backwards station, notice how many weights get no blame at all. With 1,000 learning steps the network practises on 60 noisy digits, the way real networks practise millions of times. The magnifier explains every part at three levels.
Frequently asked questions
- What is backpropagation in simple terms?
- Backpropagation is the calculation a neural network uses to find out which of its weights contributed how much to an error. It applies the chain rule backwards from the output, layer by layer, and yields a slope ∂L/∂w for every weight. Gradient descent then uses these slopes to move the weights so that the error gets smaller.
- Does backpropagation spread the error evenly?
- No. Each weight receives the share δ · a, the blame at its target neuron times the signal at its source. Weights whose source delivers no signal, such as a dark pixel with the value 0, get exactly zero blame, however large the total error is. The signs differ too: some weights have to grow, others to shrink.
- What is the difference between backpropagation and gradient descent?
- Backpropagation computes the gradient, the direction and strength in which the loss rises for every weight. Gradient descent uses this gradient and takes the actual learning step w ← w − η·∇L. The learning rate η sets how large this step is. If it is too large, the loss can rise again.
- What can I try in this app?
- You follow a whole learning round on a real network with 682 parameters, draw your own digits into the pixel image, choose the correct answer, set the learning rate and let the network practise a thousand steps. All numbers in the values panel are computed live from the network.
Subject: Neural networks
More from Neural networks
- Agent loop
- Attention Visualiser
- Backpropagation — blame in reverse
- Basin of Attraction
- Your job under AI
- The temperature dial
- Diffusion model — images from noise (3D)
- Embedding Space Explorer (3D)
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps