What is the main overhead of multiprocessing.Pool.map compared to threads?
Options
- The GIL is shared between processes, so they block each other
- Arguments and results must be pickled and sent between processes (Correct option)
- Processes cannot use more than one CPU core at the same time
- Pool.map cannot return results, so you must write to files
Answer
Arguments and results must be pickled and sent between processes
Why
Processes do not share memory. To send work to another process, Python must turn the arguments into bytes (pickling), send them, and convert the results back.
For tiny tasks or very large data, this copying can take longer than the work itself.
Send work in bigger chunks so the overhead is paid less often.
Remember
Processes pay a cost to copy data (pickle) between them.