In python2 it appears that multiprocessing uses pickle protocol 0 which > must cause a big slowdown (a factor of 100) relative to protocol 2, and > uses pickle instead of cPickle. > > Even on Python 2.x, multiprocessing uses protocol 2, not protocol 0. The default for the `pickle` module changed, but multiprocessing has always used a binary pickle protocol to communicate between processes. Have a look at multiprocessing's forking.py in Python 2.7.
As some context here for folks that may not be aware, Sturla is referring to his earlier shared memory implementation <https://github.com/sturlamolden/sharedmem-numpy> he wrote that avoids actually pickling the data, and instead essentially pickles a pointer to an array in shared memory. As Sturla very nicely summed up, it saves memory usage, but doesn't help the deeper issues. You're far better off just communicating between processes as opposed to using shared memory.
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
