You need real evidence that a new recommender raises purchases. Which rollout gives it?
Options
- An A/B test with random assignment of users to the old and the new model (Correct option)
- A shadow deployment
- An offline test on last year's data
- Asking colleagues which one looks better
Answer
An A/B test with random assignment of users to the old and the new model
Why
Random assignment makes the two groups comparable, so a difference in purchases can be attributed to the model.
Shadow mode logs predictions but users never see them, so it says nothing about behaviour. Offline data reflects the old system's choices.
Decide the metric and the sample size before the test starts, and run it for full weekly cycles.
Remember
A/B test = causal evidence of user impact.