How does gradient boosting build its ensemble?
Options
- It adds small trees one at a time, each trained to correct the errors of the current ensemble (Correct option)
- It trains one tree and prunes it by cross-validation
- It trains deep trees independently on bootstrap samples and averages them
- It picks the best of many random trees
Answer
It adds small trees one at a time, each trained to correct the errors of the current ensemble
Why
Boosting is sequential. Each new shallow tree fits the gradient of the loss (roughly, the errors left) of what has been built so far, and its contribution is scaled by a learning rate.
That mostly lowers bias, while a random forest mostly lowers variance.
Too many trees or too high a learning rate can overfit, so use early stopping on a validation set.
Remember
Boosting: sequential, each tree fixes the last errors.