Ensemble: Bagging vs. Boosting
About this app
Many shallow decision trees combined into one strong model: single tree (jagged staircase), bagging (average out variance → smooth boundary) and boosting (sequentially on hard cases). Interlocking "two moons", accuracy over the number of trees — Random Forest made tangible.
Many weak learners, one strong model
An ensemble combines many weak learners, here shallow decision trees, into a stronger model. Bagging trains the trees independently on random bootstrap samples and lets them vote by majority. Because independent errors partly average out, this mainly reduces variance. Boosting, here AdaBoost, trains sequentially: after each tree, misclassified points get more weight and the next tree focuses on them. This mainly reduces bias. A random forest is bagging plus a random choice of features at every split.
What you see
Two interlocking half-moons (“two moons”) as training data and the decision boundary of the selected mode: a single shallow tree with a jagged staircase boundary, bagging with an averaged vote, or boosting with sequentially weighted trees. A chart shows training and test accuracy against the number of trees. Test accuracy is estimated on a dense grid using the true class pattern.
Things to try
Start with “single tree” at a depth of 1 to 4. Switch to bagging and slide the number of trees T from 1 to 50: the boundary becomes smoother. Then compare boosting at the same depth. Notice which method gains a lot from just a few trees and where test accuracy levels off. “New data” draws a fresh sample.
Frequently asked questions
- What is the difference between bagging and boosting?
- Bagging trains many models in parallel on bootstrap samples and averages their predictions, which mainly reduces variance. Boosting trains models one after another, each giving more weight to the previous models’ errors, which mainly reduces bias. Bagging usually uses deep trees, boosting usually shallow ones.
- What is a bootstrap sample?
- A bootstrap sample draws n points with replacement from n training points. Some points appear several times, and on average about a third are left out. Each tree therefore sees a slightly different dataset and the trees make different errors.
- Why do ensembles often beat a single tree?
- A single decision tree reacts strongly to small changes in the data and therefore has high variance. When many different trees average their predictions, random errors partly cancel out. Boosting can additionally correct systematic errors step by step.
- What can I do in this app?
- You choose single tree, bagging or boosting, set the number of trees T (1 to 50) and the maximum tree depth (1 to 4), and draw new data. The app shows the decision boundary and training and test accuracy across the number of trees.
Subject: Machine Learning
More from Machine Learning
- AdaBoost
- Anatomy of a prompt
- Data privacy traffic lights
- Decision Tree
- Finding faces: masks versus learning from examples
- k-Means
- k-Nearest Neighbors (kNN)
- AI theatre
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps