Training Sandbox
About this app
Reading training dynamics from the loss curves: learning rate, batch size, epochs, training and validation loss, early stopping at the validation minimum, L2, dropout, augmentation and more data against overfitting.
Reading learning curves
When a neural network trains, the training loss measures how well it fits the points it learns from. The validation loss measures the same on points it has never seen; only this one tells you anything about new data. If both fall together, the network is learning the structure. If the training loss keeps falling while the validation loss rises again, it is memorising noise: overfitting. If both stay high, the network is underfitting, for example because it is too small or the learning rate too low.
What the app shows
A small network with tanh layers and a sigmoid output learns live to separate two classes in one of three datasets: moons, circles or spiral. It trains with cross-entropy and SGD with momentum 0.9. A chart shows training loss, validation loss and the noisy loss per mini-batch, next to the decision boundary with training and validation points. The app names the current state, such as overfitting, underfitting or diverging, and marks the validation minimum. Earlier runs remain as comparison curves.
Things to try
Pick a large network and few training points: after a while the validation loss climbs again. Then switch on early stopping with a patience of 25 epochs or raise the number of points between 20 and 600. Try L2 regularisation, dropout and augmentation and watch the gap between validation and training loss close. A learning rate that is too large makes the loss jump, one that is too small barely moves it. A mini-batch of 1 turns every update into a noisy random step.
Frequently asked questions
- What is overfitting in neural networks?
- Overfitting means that a model memorises the training data including its noise. The error on the training data keeps falling, while the error on new data rises again. Typical causes are a model that is too large for the amount of data and training for too long.
- What is early stopping?
- Early stopping monitors the error on a validation set during training. If it has not improved for a set number of epochs, training stops. The weights from the point of lowest validation error are kept.
- How do you choose the right learning rate?
- If the learning rate is too small, the loss barely falls. If it is too large, steps overshoot the minimum and the loss jumps or diverges. People usually try values across several orders of magnitude and read the behaviour off the learning curve.
- What can I try in this app?
- You set learning rate, mini-batch size, epochs, amount of data, network size, L2, dropout and augmentation and watch training and validation loss live. You can jump back to the validation minimum and compare several runs side by side.
Subject: Neural networks
More from Neural networks
- Agent loop
- Attention Visualiser
- Backpropagation — blame in reverse
- Backpropagation: learning from one mistake
- Basin of Attraction
- Your job under AI
- The temperature dial
- Diffusion model — images from noise (3D)
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps