How can you check systematically for training-serving skew in a live system?
Options
- Compare the training loss with the validation loss
- Check the Docker image size
- Increase the number of replicas
- Log the exact features the model receives in production and compare them with the same features recomputed offline, or with the training distributions (Correct option)
Answer
Log the exact features the model receives in production and compare them with the same features recomputed offline, or with the training distributions
Why
Skew means the model gets different inputs in production than the pipeline gave it in training. The way to see it is to look at what it really receives.
Logging serving features (sampled) lets you compare them with an offline recomputation for the same requests, which finds code differences, or with training statistics, which finds missing values, changed units or encodings.
The best prevention is to run the same feature code in both places.
Remember
Log serving features and compare them with offline values.