The training pipeline runs in CI with the same code and data as last month, but the metrics changed. What do you check first?
Options
- The color of the dashboard
- The number of CPU cores
- Unpinned library versions, random seeds and whether the data snapshot is really the same (Correct option)
- The name of the branch
Answer
Unpinned library versions, random seeds and whether the data snapshot is really the same
Why
Reproducibility needs all the inputs fixed: exact dependency versions (a lock file), random seeds, the data snapshot (not "the latest table") and the configuration.
A library update can change defaults, and a table that is still being updated is not the same data.
Once these are pinned, the same inputs should give the same metric, or at least one within a known noise range.
Remember
Pin versions, seeds and data to reproduce a run.