With millions of rows per day, a statistical drift test flags almost every feature as "significantly different" every day, with tiny p-values. What is a better approach?
Options
- Use the size of the change (for example a distance or PSI) with thresholds tied to its impact, instead of p-values alone (Correct option)
- Ignore drift monitoring altogether
- Lower the p-value threshold to 0.5
- Retrain automatically each time a test fails
Answer
Use the size of the change (for example a distance or PSI) with thresholds tied to its impact, instead of p-values alone
Why
With huge samples, even a harmless difference becomes statistically significant, so the alert never stops and gets ignored.
A drift measure with a magnitude (PSI, Jensen-Shannon distance, Wasserstein distance) lets you set thresholds on how big a change matters, and you can prioritize the features the model relies on most.
Always confirm that an alert corresponds to a real effect on predictions or quality.
Remember
Alert on the size of the change, not only significance.