The online metric of a model slowly gets worse over several weeks. How do you investigate?
Answer
- Confirm it is real: check the metric definition, the sample size and seasonality before blaming the model.
- Rule out the service and the data pipeline: errors, latency, missing or late data, changed schemas or units.
- Compare live inputs with the training reference (drift) and look at the prediction distribution, overall and by slice.
- Compare live features with the same features recomputed offline to look for training-serving skew.
- When labels are available, check whether the quality fell on all slices or only on some, and since when.
- Check what changed around that date: code, model version, upstream data, product or user behaviour.
- Then act: fix the data or the code, retrain with recent data through the normal gate, or roll back, and add a monitor so it is caught earlier next time.
Practice this question in Repaso
