A feature is computed with SQL for training but re-implemented in the serving code, and the online values differ slightly. What is this called and what is the best fix?
Options
- Training-serving skew; use one shared definition of the feature (or log the serving values and train on them) (Correct option)
- Overfitting; add regularization
- Concept drift; retrain more often
- Data leakage; remove the feature
Answer
Training-serving skew; use one shared definition of the feature (or log the serving values and train on them)
Why
Training-serving skew is a difference between the data the model was trained on and what it gets in production, here caused by two implementations of one feature.
The fixes: compute the feature with the same code (or a feature store) in both places, or log the exact values at serving time and train on those logs, and test that offline and online values match.
Retraining does not help if the two pipelines disagree.
Remember
Same feature code for training and serving.