The average latency of a model service is fine, but users complain about slow responses. What do you look at?
Options
- The average only: it is fine
- The size of the training data
- The accuracy of the model
- Percentile latencies (p95, p99) broken down by stage: queueing, feature lookup, preprocessing and model (Correct option)
Answer
Percentile latencies (p95, p99) broken down by stage: queueing, feature lookup, preprocessing and model
Why
The average hides the slow tail: a few percent of requests can be many times slower and still leave the mean looking good, while every user meets them sometimes.
Break the time down by stage. Typical culprits are cold starts, waiting in the batching queue, a slow feature lookup, garbage collection and noisy neighbours.
Set the service-level target on a percentile, such as p99 below 300 ms.
Remember
Averages hide the tail: use percentiles by stage.