How do you decide between batch, online and streaming inference for a new use case?
Answer
- Latency need: if nobody waits for the result, use batch; if a user or system waits for seconds or milliseconds, use online.
- Freshness of the inputs: if the decision depends on events from the last minutes, you need streaming features, whatever the serving mode.
- Volume and cost: batch uses hardware efficiently and is cheap; online needs always-on capacity, autoscaling and headroom for peaks.
- Reuse: precomputing predictions for a limited set of entities (users, products) in batch and serving them from a store is often the simplest option that is fast enough.
- Complexity and operations: each step towards real time adds monitoring, on-call and consistency work between training and serving.
- Failure plan: what happens if the model is unavailable? A stale batch result or a rule-based default can be a good fallback.
- Start with the simplest mode that meets the requirement, and move to a more complex one only when the numbers justify it.
Practice this question in Repaso