Model validation: train, validation, test
About this app
Split, validation curve, k-fold cross-validation and a one-time test on a polynomial regression — plus data leakage and tuning on the test set as counterexamples.
Three datasets, three roles
Training data fit a model’s parameters. Validation data or k-fold cross-validation choose hyperparameters such as model complexity. The test set is used exactly once at the very end and then gives an unbiased estimate of the error on new data. Training error is not a quality measure, because it keeps falling as complexity grows. Validation error instead forms a U shape: underfitting on the left, overfitting on the right, the best model in between. Using the test set repeatedly for selection, or letting information leak across splits, produces numbers that are too optimistic.
What you see
Polynomial regression in seven steps: hold-out split, fitting on training data, validation curve, k-fold cross-validation, final test, data leakage and tuning on the test set. Three cards show the data and split, the model fit together with the true relationship, and the error (MSE) against polynomial degree with the noise floor σ². A badge shows whether the test set is still sealed and how often it has been opened.
Things to try
In step 2, raise the degree to 15 and watch the polynomial pass through almost every training point. In step 3, study the validation curve and set the degree to d*. In step 4, change the number of folds K. In step 6, turn on leakage and compare test error with fresh data. In step 7, test 50 configurations directly on the test set and see performance drop on fresh data.
Frequently asked questions
- What is the difference between validation and test data?
- Validation data are used to choose between model variants or hyperparameters. Test data are used once, after all decisions have been made, to estimate the final model’s performance without bias. If the test set is used for selection, its estimate is no longer unbiased.
- How does k-fold cross-validation work?
- The data are split into K equally sized parts (folds). Each fold serves once as validation data while the other K − 1 are used for training. The mean and spread of the K errors give a more stable basis for model selection than a single split.
- What is data leakage in machine learning?
- Data leakage means that information from validation or test data gets into training. Typical causes are duplicates across splits, normalising or selecting features on all data before splitting, and future data in the training set for time series. The result is error estimates that are too good.
- What can I do in this app?
- You go through seven steps, set the number of data points, validation and test shares, the number of folds and the polynomial degree, and draw new data or reshuffle the split. The app shows training, validation and cross-validation error, the final test, and the effect of leakage and tuning on the test set.
Subject: Machine Learning
More from Machine Learning
- AdaBoost
- Anatomy of a prompt
- Data privacy traffic lights
- Decision Tree
- Ensemble: Bagging vs. Boosting
- Finding faces: masks versus learning from examples
- k-Means
- k-Nearest Neighbors (kNN)
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps