A random forest ranks a customer ID column among its most important features when using the default impurity-based importance. What is a better check?
Options
- Trust the ranking: the forest saw the data
- Drop the columns with the lowest importance and retrain
- Permutation importance computed on held-out data (Correct option)
- Add more trees until the ranking stabilizes
Answer
Permutation importance computed on held-out data
Why
Impurity-based importance is computed on the training data and favours features with many distinct values, like IDs, because they let trees split into tiny pure groups. That is memorization, not signal.
Permutation importance shuffles one feature at a time on held-out data and measures how much the score drops, so useless features get a value near zero.
It can still mislead when features are strongly correlated.
Remember
Prefer permutation importance on held-out data.