What does quantizing a model do?
Options
- It trains the model on more data
- It encrypts the model file
- It adds layers to the model
- It stores weights (and sometimes activations) in lower-precision numbers, such as 8-bit integers, to cut size and speed up inference, with a possible small loss of accuracy (Correct option)
Answer
It stores weights (and sometimes activations) in lower-precision numbers, such as 8-bit integers, to cut size and speed up inference, with a possible small loss of accuracy
Why
Weights are normally 32-bit floating-point numbers. Converting them to 8-bit integers or 16-bit floats makes the model smaller, uses less memory bandwidth and can run faster on suitable hardware.
Accuracy can drop a little, so measure it with your own evaluation set, on the target hardware, before shipping.
Pruning and distillation are related ways to get a smaller model.
Remember
Quantization: smaller and faster, check the accuracy.