A pipeline step failed halfway and was run again, and now the training table has duplicated rows. Which design principle prevents this?
Options
- Never re-run failed steps
- Add more workers
- Make steps idempotent: running a step twice gives the same result, for example by overwriting a date partition instead of appending blindly (Correct option)
- Store the table in a different format
Answer
Make steps idempotent: running a step twice gives the same result, for example by overwriting a date partition instead of appending blindly
Why
Failures and retries are normal in pipelines. If a step appends rows, a retry duplicates them.
An idempotent step writes its output so that repeating it replaces the same result: it overwrites a partition keyed by the run date, or it upserts by a unique key.
This also makes backfills (re-running past dates) safe.
Remember
Idempotent steps make retries and backfills safe.