A model had a great cross-validation score but disappoints in production. What do you check?
Answer
- Leakage: features only known after the outcome, preprocessing or feature selection fitted on all the data, duplicated rows, or the same entity in both training and validation.
- Split design: random splits on time-ordered or grouped data instead of splits by time or by group.
- Training-serving skew: different feature code, missing values, units or time zones online.
- Drift: has the input distribution, or the behaviour behind the labels, changed since the training period?
- Population: does production traffic include cases that never appeared in the training data (selection bias)?
- Metric: is the offline metric a good proxy for the product goal, and was the threshold set for production costs?
- Compare offline and online feature values for the same cases to find the first difference.
Practice this question in Repaso
