An upstream team renames a column. The pipeline silently fills the feature with nulls, and model quality collapses weeks later. What would have caught it early?
Options
- Data validation checks at ingestion (schema, null rate, ranges, distributions) that fail the pipeline or raise an alert (Correct option)
- A bigger model
- More frequent retraining
- A faster GPU
Answer
Data validation checks at ingestion (schema, null rate, ranges, distributions) that fail the pipeline or raise an alert
Why
Many ML failures are data failures that nobody sees, because the code keeps running. Retraining on the broken data would just make the model learn the breakage.
Validate the data at the pipeline's entrance and before training: expected columns and types, maximum share of nulls, value ranges, and distribution checks against a reference. Tools such as TFDV or Great Expectations help.
Fail fast and tell the owner of the data.
Remember
Validate data at the door; fail fast.