The same inputs reach a prediction service again and again, and each call is slow. What is a cheap way to speed it up?
Options
- Train a bigger model
- Remove the monitoring
- Disable logging
- Cache predictions keyed by the input (and the model version), with an expiry time (Correct option)
Answer
Cache predictions keyed by the input (and the model version), with an expiry time
Why
If inputs repeat, a cache returns the stored result in microseconds instead of running the model.
Put the model version in the cache key, so a new model never serves old answers, and set an expiry for data that changes. Only cache predictions that are deterministic and not personal in a way that would leak between users.
Measure the hit rate to see if it is worth it.
Remember
Cache by input + model version, with an expiry.