You standardize every feature using the mean and standard deviation of the whole dataset, and only then split it into train and test. What is wrong?
Options
- Test-set statistics leaked into the preprocessing, so the test score can look better than reality (Correct option)
- Scaling before splitting makes the model overfit
- Nothing: scaling does not learn anything from the labels
- Standardization must always be done after training
Answer
Test-set statistics leaked into the preprocessing, so the test score can look better than reality
Why
Scaling learns something from the data: the mean and the standard deviation. If those include the test rows, information from the test set reached the training process. This is data leakage.
The fix is to fit the scaler on the training data only and then apply it to the test data. A scikit-learn Pipeline does this for you, also inside cross-validation.
Labels are not needed for leakage: any statistic that peeks at the test set counts.
Remember
Fit preprocessing on training data only.