A model server groups several incoming requests into one batch before running the model on a GPU. What is the trade-off?
Options
- Higher throughput and better hardware use, at the cost of some extra waiting time (latency) for each request (Correct option)
- Higher accuracy for lower throughput
- Lower cost and lower latency at the same time, always
- No trade-off: it only changes the logs
Answer
Higher throughput and better hardware use, at the cost of some extra waiting time (latency) for each request
Why
GPUs are most efficient when they process many inputs at once. Waiting a few milliseconds to fill a batch raises the number of requests handled per second.
But the first request in a batch waits for the others, so its latency grows. Dynamic batching lets you cap that wait.
Tune the maximum batch size and wait time against your latency target.
Remember
Batching: more throughput, a bit more latency.