A GPU-based model service has spiky traffic. Scaling on CPU usage reacts badly. What is a better signal for autoscaling?
Options
- The time of day
- A signal that reflects the real load of this workload, such as requests in flight, queue length or latency (Correct option)
- The number of developers online
- The size of the model file
Answer
A signal that reflects the real load of this workload, such as requests in flight, queue length or latency
Why
For GPU inference the CPU is often not the bottleneck, so CPU usage can stay low while requests pile up.
Scaling on concurrency (requests in flight), queue length or latency follows the real pressure. Also account for the time a new replica needs to load the model, which can take minutes, and keep a margin or a minimum number of replicas.
Scaling to zero saves money but adds cold-start latency.
Remember
Scale on load signals that fit the workload.