Why does a single, very deep decision tree tend to overfit?
Options
- It cannot split on numeric features
- It can keep splitting until it fits even the noise of the training set, so small changes in the data change the tree a lot (Correct option)
- Trees need scaled features, and unscaled data confuses them
- Trees are always too shallow to learn real patterns
Answer
It can keep splitting until it fits even the noise of the training set, so small changes in the data change the tree a lot
Why
Without limits (maximum depth, minimum samples per leaf, pruning), a tree keeps splitting until each leaf is nearly pure, even if it only memorizes noise.
That gives low bias and high variance: a slightly different training set produces a very different tree.
Limiting the depth or averaging many trees (a forest) reduces the variance.
Remember
Deep trees memorize: low bias, high variance.