Which signal of data drift can you monitor in production even before any labels arrive?
Options
- A comparison of the distribution of live inputs with the training data, for example with a PSI or KS statistic (Correct option)
- The training loss of the model
- The size of the model file
- The number of hyperparameters
Answer
A comparison of the distribution of live inputs with the training data, for example with a PSI or KS statistic
Why
Drift means the data the model sees now differs from what it learned from. You can measure that on the inputs alone, because inputs need no labels.
Common checks compare each feature's live distribution with a reference window using statistics such as the Population Stability Index (PSI) or the Kolmogorov-Smirnov test.
It is an early warning: drift does not always lower quality, and quality can drop without input drift.
Remember
Compare live inputs with the training reference.