Random Forest
About this app
Many random decision trees vote: the majority smooths the jagged boundaries of single trees and lowers the test error — bootstrap samples, feature randomness, out-of-bag error and test error over the number of trees.
How a random forest decides
A random forest trains many decision trees and lets them vote. Each tree sees a bootstrap sample, n points drawn with replacement; about a third of the points are left “out of bag” and can be used to estimate the error. In addition, each tree only considers a random subset of features at every split. A deep single tree also learns the noise and has a jagged boundary. Because the trees in a forest go wrong in different places, many errors average out: the boundary becomes smoother and test error drops.
What you see
A vote map where colour strength shows the forest’s share of votes, next to a single tree with its in-bag and out-of-bag points. A chart plots the forest’s test error, the out-of-bag estimate and the average single-tree error against the number of trees on a logarithmic axis. Testing uses 1000 fresh points from the same distribution. An overview shows all trees as small maps.
Things to try
Grow the forest on the “moons”, “circles” or “XOR” datasets and tap individual trees. Notice that the error curve ends on a plateau as trees are added and does not rise again. Turn off bootstrap and allow all features per split: then every tree is identical. Add the noise feature x₃ and compare different numbers of features per split.
Frequently asked questions
- How does a random forest work?
- A random forest consists of many decision trees, each trained on a bootstrap sample with a random choice of features at every split. For classification all trees vote and the majority decides. The randomness makes the trees different so that their errors partly cancel out.
- Can a random forest overfit by adding more trees?
- Adding trees does not cause overfitting; test error approaches a plateau as the number of trees grows. Overfitting can still come from other settings or very noisy data. More trees mainly cost computing time.
- What is the out-of-bag error?
- With bootstrap sampling, about a third of the training points go unused by each tree. Predicting each point only with the trees that did not see it gives the out-of-bag error. It estimates test error without needing a separate test set.
- What can I do in this app?
- You set the number of trees (up to 200), maximum depth, features per split and bootstrap, choose dataset, noise and number of points, and grow the forest. The app shows the vote, individual trees, the out-of-bag error and test error across the number of trees.
Subject: Machine Learning
More from Machine Learning
- AdaBoost
- Anatomy of a prompt
- Data privacy traffic lights
- Decision Tree
- Ensemble: Bagging vs. Boosting
- Finding faces: masks versus learning from examples
- k-Means
- k-Nearest Neighbors (kNN)
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps