A new ranking model improves the offline metric, but click-through falls in the online A/B test. What is the most plausible explanation?
Options
- The A/B testing tool must be buggy
- The model needs more regularization
- The offline data only reflects what the old system showed, and the offline metric is only a proxy for the goal, so the improvement did not carry over (Correct option)
- The test set was too large
Answer
The offline data only reflects what the old system showed, and the offline metric is only a proxy for the goal, so the improvement did not carry over
Why
Logged data only contains items the old system chose to show, so the model is evaluated on a biased sample (selection bias) and its choices outside that sample are never checked. The offline metric, such as NDCG on logged clicks, is also only a proxy for the goal.
The online experiment measures the real effect, so trust it, after checking the sample size and that users were assigned at random.
Use offline metrics to filter candidates and online tests to decide.
Remember
Offline metrics are proxies; the A/B test decides.