Skip to content
All apps

AdaBoost

About this app

Decision stumps trained one after another on weighted data: mistakes grow, the next stump targets them, vote weight α per round, the ensemble's exact staircase boundary, train and test error over up to 60 rounds.

How boosting works

AdaBoost trains very weak learners in sequence, here decision stumps that each draw a single axis-parallel split. Every training point carries a weight, initially 1/N. The stump is trained on the weighted data and has weighted error ε. Misclassified points then gain weight and correctly classified ones lose weight, so the next stump focuses on the hard cases. Each stump gets the vote α = ½·ln((1−ε)/ε); at ε = 0.5 (guessing) α = 0. The weighted vote of all stumps is the ensemble’s decision.

What you see

Four cards: the current weak learner with point size showing weight, the ensemble boundary as a staircase of straight cuts, the vote α per round as bars, and the error history. The error chart shows training error, test error, ε per stump and the theoretical bound Π 2√(ε(1−ε)) on the training error. A table lists split, error and α for each round; clicking a row or a bar shows the ensemble only up to that round.

Things to try

Load “circle”, “diagonal” or “moons”, or place your own class A and class B points. Train round by round with “+1 weak learner” and watch the previous stump’s mistakes grow. Later stumps usually have a larger ε and therefore a smaller α. Notice that training error keeps falling while test error eventually levels off, at which point extra rounds no longer help.

Frequently asked questions

How does AdaBoost work?
AdaBoost builds an ensemble of weak learners trained one after another. After each round it increases the weights of misclassified points so the next learner concentrates on them. In the end all learners vote with weights, and more accurate learners get a larger vote α.
How is the learner weight α computed in AdaBoost?
For a learner with weighted error ε, α = ½·ln((1−ε)/ε). A nearly perfect learner gets a large α, while a learner with ε = 0.5 gets α = 0 because it is no better than guessing.
What is the difference between boosting and bagging?
Bagging trains many models independently on bootstrap samples and lets them vote equally, which mainly reduces variance. Boosting trains sequentially, each learner depends on the previous one’s errors, and votes are weighted unequally, which mainly reduces bias.
What can I do in this app?
You pick a dataset or place points, train round by round or automatically, and show test data. The app displays the current stump with point weights, the ensemble boundary, α per round, and training and test error over up to 60 rounds.

Subject: Machine Learning

More from Machine Learning

A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps