Overall model quality is stable, but support tickets show it works badly for customers in one region. Which monitoring practice would have shown this?
Options
- Looking only at the overall average
- Counting the number of requests
- Checking uptime
- Computing quality and data metrics by slice (region, device, customer type) and alerting on each slice (Correct option)
Answer
Computing quality and data metrics by slice (region, device, customer type) and alerting on each slice
Why
An average can be stable while a small group gets much worse results, because the large groups hide it.
Monitor important slices separately for data drift, prediction distribution and quality metrics, with enough volume to be meaningful. The same slices belong in the evaluation gate before a release.
Pick the slices from business risk and from where the data differs, since you cannot alert on all combinations.
Remember
Monitor by slice, not only overall.