You must forecast next week's demand using three years of weekly data. Which validation scheme fits?
Options
- Shuffled 5-fold cross-validation
- A random 80/20 split
- Time-ordered splits: train on earlier weeks and validate on later ones (for example
TimeSeriesSplit) (Correct option) - Leave-one-out cross-validation
Answer
Time-ordered splits: train on earlier weeks and validate on later ones (for example TimeSeriesSplit)
Why
In production the model only sees the past and predicts the future. Validation must copy that. Shuffling lets the model train on weeks after the ones it is tested on, which leaks information about trends and seasonality, and the scores come out too good.
TimeSeriesSplit makes folds where the training window always comes before the validation window.
Keep a final period at the end as the untouched test set.
Remember
Time data: validate forward in time.