Bias-variance / overfitting
About this app
Many training sets from the same world as a family of models: average model against the truth (bias), spread of the models (variance) and the exact decomposition test error = bias² + variance + noise over polynomial degree or ridge λ.
The statistics
A model’s expected test error splits exactly into test error = bias² + variance + noise σ². Bias is the systematic gap between the average model and the true function; variance is how much the models spread when fitted to different training sets from the same world. Noise is a floor no model can go below. A model that is too simple has high bias (underfitting), one that is too flexible has high variance (overfitting). Training error keeps falling as complexity grows, because it cannot see the variance.
What you see
At the top, new training sets are drawn again and again from the same world, a true function plus noise, and a degree-d polynomial is fitted to each. Every thin line is one model, the thick line their average and a band their spread. At a movable probe x₀ all predictions are stacked as dots. Below, a chart splits the test error into bias², variance and noise across polynomial degree or ridge regularisation λ. The total is re-measured on fresh test data, and 5-fold cross-validation estimates the error from a single data set.
Things to try
Raise the polynomial degree d step by step: first the average model approaches the true curve, then the individual models start to swing wildly. Find the minimum of the U-shaped test-error curve. Set a high degree and increase ridge λ; variance drops, bias rises and the model becomes usable again. Change the noise, the points per training set and the number of training sets. Show your own data set to see that a single set reveals no variance.
Frequently asked questions
- What is the bias-variance tradeoff?
- The bias-variance tradeoff describes the tension between simple and flexible models. Simple models systematically miss the true structure (high bias); flexible ones react strongly to random fluctuations in the training data (high variance). The lowest test error lies at a compromise in between.
- What is overfitting?
- Overfitting happens when a model reproduces the training data, noise included, very closely but performs poorly on new data. Training error is then low and test error high. Remedies include simpler models, more data, regularisation such as ridge and choosing the model by cross-validation.
- How does ridge regularisation help against overfitting?
- Ridge regression penalises large coefficients with an extra term λ·Σβⱼ². This makes the models smoother and less variable between data sets; variance falls and bias rises a little. With a well-chosen λ even a model with many parameters becomes usable again.
- What can I set in the bias-variance app?
- You set polynomial degree d, ridge λ, noise σ, points per training set and the number of training sets, and choose whether the error curve runs over d or λ. You can also draw new training sets, show your own data set and compare with 5-fold cross-validation.
Subject: Statistics
More from Statistics
- Anscombe's quartet
- Box plot workshop
- Geometric mean
- Law of large numbers
- Histogram workshop
- Hypothesis test & p-value
- Confidence interval
- Correlation: why standardise?
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps