What should continuous integration for an ML project run, in addition to unit tests?
Options
- Nothing else: unit tests are enough
- Data and schema checks, a quick training run on a small sample, an evaluation gate against a baseline, and a test that the model can be packaged and served (Correct option)
- A full training run on all the data for every commit
- Only code formatting checks
Answer
Data and schema checks, a quick training run on a small sample, an evaluation gate against a baseline, and a test that the model can be packaged and served
Why
Code can pass its unit tests while the pipeline is broken. ML adds failure modes that CI can check cheaply.
A smoke training run on a small sample shows that the code runs end to end. A quality gate compares the model with a baseline or the current model. A serving test checks that the packaged model loads and answers.
Training on all the data for every commit is too slow and costly; save it for scheduled runs.
Remember
CI for ML tests code, data and model.