Davin Potts added the comment: Though it's been discussed elsewhere, issue17560 is a good one where the matter of "really big" objects are being communicated between processes via multiprocessing. In it, Richard shares some detail about the implementation in multiprocessing, its constraints and motivation.
That discussion also highlights that the pickler has been made pluggable in multiprocessing since Python 3.3. That is, if you wish, you can use something other than Python's pickle to serialize objects and, in the extreme, potentially communicate them in a completely new way (perhaps even via out-of-band communication, though that was not the intention and would be arguably extreme). I do not think Python's pickle is necessarily what we should expect multiprocessing to use to help communicate objects between processes. Just because pickle is Python's serialization strategy does not also mean it must necessarily also be used in such communications. Thankfully, we have the convenience of using something other than pickle (or newer versions of pickle, since there have been versioned updates to pickle's format over time with some promising improvements). @Luis: To your specific question about the need for Queue, the benefits of a consistent behavior and methodology whether forking/spawning on one OS with its caveats versus another are just that: simplicity. The pluggable nature of the pickler opens up the opportunity for folks (perhaps such as yourself) to construct and plug-in optimized solutions for scenarios or edge cases of particular interest. The fact that we all start with a consistent behavior across platforms and process creation strategies is very appealing from a reduced complexity point of view. ---------- nosy: +davin _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue23979> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
