An automatic weekly retraining job trained on data corrupted by an upstream bug, and the worse model went live. Which safeguard was missing?
Options
- A larger training set
- A faster training cluster
- A promotion gate: evaluate the candidate against the current model on a fixed, trusted holdout and on data-quality checks, and only promote it if it passes, with a quick rollback (Correct option)
- More frequent retraining
Answer
A promotion gate: evaluate the candidate against the current model on a fixed, trusted holdout and on data-quality checks, and only promote it if it passes, with a quick rollback
Why
Automation without a gate turns any data problem into a production problem. A new model should have to earn its place.
The gate compares the candidate with the current champion on a fixed evaluation set that does not change with the new data, checks for regressions on important slices, and runs the data validation on the training set. A failed gate keeps the old model and alerts a person.
The old version stays deployable for a rollback.
Remember
Retraining needs a promotion gate.