You tried 50 hyperparameter combinations and reported the best score on the test set. What is the problem?
Options
- The score is optimistic: the test set guided the choice, so it is no longer unseen (Correct option)
- 50 combinations is too few to be reliable
- The score is pessimistic, since the best model was picked
- None: this is how the test set is meant to be used
Answer
The score is optimistic: the test set guided the choice, so it is no longer unseen
Why
Choosing the best of 50 results on the same data selects partly on luck. The winning score is biased upward, and the test set no longer measures performance on unseen data.
Tune on a validation set or with cross-validation. Touch the test set once, for the final model.
If you must tune a lot and still want an honest number, use nested cross-validation.
Remember
Select on validation data, report on test data.