You pick the 20 features most correlated with the label on the full dataset, then run cross-validation on those 20. Why is the score too optimistic?
Options
- Twenty features are too few
- The selection used the labels of the validation folds, so it leaks information; it must be done inside each training fold (Correct option)
- Cross-validation cannot be used after feature selection
- Correlation is not a valid way to rank features
Answer
The selection used the labels of the validation folds, so it leaks information; it must be done inside each training fold
Why
Choosing features with the labels of all rows lets the validation rows influence which features are kept. With many candidate features, some correlate with the label by chance, and the CV score then looks good.
Put feature selection inside the pipeline so it is fitted again on each training fold.
Correlation is a fair first filter; the mistake is where in the process it runs.
Remember
Selection is part of training: do it inside CV.