Why do you keep a test set that the model never sees during training?
Options
- To make training run faster on a smaller dataset
- To estimate how the model will perform on new, unseen data (Correct option)
- To choose the best hyperparameters
- To balance the classes before training
Answer
To estimate how the model will perform on new, unseen data
Why
A model can memorize its training data, so its score on that data says little about the future.
The test set stands in for new data: you measure it once, at the end, to get an honest estimate.
If you tune hyperparameters on it, it stops being honest. Use a separate validation set (or cross-validation) for tuning.
Remember
Test data gives an honest estimate; do not tune on it.