To build a training set, each row is joined to the latest value of a feature customer_total_spend, taken from a table that is updated daily. The model is great offline and poor online. What is the likely problem?
Options
- The table is too small
- The join used values from after the event time, a leak; features must be joined as they were at that moment (a point-in-time join) (Correct option)
- The model is too simple
- The table needs more indexes
Answer
The join used values from after the event time, a leak; features must be joined as they were at that moment (a point-in-time join)
Why
For a row dated 1 March, the feature must be the value known on 1 March. Using the latest value includes purchases made after that date, so the model sees the future in training, but not in production.
A point-in-time (as-of) join picks, for each row, the latest feature value with a timestamp not after the row's event time.
Feature stores such as Feast do this correctly.
Remember
Join features as they were at event time.