Skip to content
All apps

Backpropagation — blame in reverse

About this app

Backpropagation on a 2-2-2 network with real numbers in six steps: forward pass, loss, δ at the output and in the hidden layer, gradient per weight, only the update lowers the loss — checked numerically.

What backpropagation does

Backpropagation computes how strongly the error L of a neural network depends on each individual weight. First the input runs forward: each neuron forms z = Σ w·a + b and passes on a = σ(z). Then it runs backwards. At the output, δ = ∂L/∂z is the deviation times the slope of the sigmoid. A hidden neuron receives blame from every output, each time through the weight it contributed with (chain rule). Each weight gets the gradient ∂L/∂w = δ · a. Backpropagation itself changes nothing; only the update w ← w − η · ∂L/∂w makes the network learn.

What the app shows

The app computes a small network with two inputs, two hidden neurons and two outputs, giving 8 weights and 4 biases, with sigmoid everywhere and error L = ½ Σ (y − t)². The process has six stages: forward, error at the output, δ at the output, δ in the hidden layer, gradient per weight, and update. Each sphere’s fill level is its activation, and line thickness shows a weight’s magnitude. The values table checks every gradient against a numerical derivative (ε = 10⁻⁶).

Things to try

Pick one of the three training examples A, B or C and step through the six stages with next. Click a weight to see its calculation. Notice that no weight changes during the backward steps. Set the learning rate between 0.05 and 3 and check whether the error falls after the update or rises when η is too large. Train ×10 runs several learning steps. In example C the signals from both outputs partly cancel in the hidden layer.

Frequently asked questions

What is backpropagation in simple terms?
Backpropagation is a method that works out, for every weight of a neural network, how the error changes if that weight is nudged slightly. It does this by applying the chain rule from the output back to the input. The resulting gradients are then used by gradient descent to adjust the weights.
What does δ mean in backpropagation?
δ is the derivative of the error with respect to a neuron’s weighted sum z, that is ∂L/∂z. At the output it is the deviation times the slope of the activation function. In hidden layers it is computed back from the δ values of the next layer, weighted by the connections.
Does backpropagation spread the error evenly across all weights?
No. Each weight gets its own gradient ∂L/∂w = δ · a, which depends on the activation at its source and the δ at its target. A weight with a small input activation receives a small gradient, however large the overall error is.
What can I try in this app?
You step through the forward and backward pass, see every number and click weights to see their calculation. You set the learning rate, train repeatedly and compare each gradient with a numerical check.

Subject: Neural networks

More from Neural networks

A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps