The labels for your model's predictions arrive about 60 days later. How do you monitor the model in the meantime?
Options
- Track input data quality and the feature and prediction distributions against a reference as early warnings, and compute the real metrics when labels arrive (Correct option)
- Monitor only latency and error rates of the service
- Retrain every night and assume it fixes everything
- You cannot: wait for the labels
Answer
Track input data quality and the feature and prediction distributions against a reference as early warnings, and compute the real metrics when labels arrive
Why
Without labels you cannot measure accuracy, but you can see symptoms: missing values, changed ranges, shifts in feature distributions or in the share of predicted positives.
These are alarms, not proof: a shift does not always hurt accuracy. When labels arrive, compute the real metrics (grouped by prediction date) and compare.
Latency and errors are needed too, but they say nothing about quality.
Remember
No labels yet: watch the inputs and the predictions.