[issue23979] Multiprocessing Pool.map pickles arguments passed to workers

Change by Socob <206a8...@opayq.com>: -- nosy: +Socob ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue23979] Multiprocessing Pool.map pickles arguments passed to workers

Changes by Davin Potts : -- status: open -> closed ___ Python tracker ___ ___

[issue23979] Multiprocessing Pool.map pickles arguments passed to workers

Davin Potts added the comment: @Luis: Before closing this bug, could you give a quick description of the fix you put in place? Two reasons: 1) Someone else encountering this frustration with similar needs might well benefit from hearing what you did. 2) To provide evidence that there is

[issue23979] Multiprocessing Pool.map pickles arguments passed to workers

Luis added the comment: Thanks for information and explanations. The option of writing a tweaked serialization mechanism in Queue for Pool and implement a sharedmem sounds like fun, not sure if the pure-copy-on-write of forking can be achieved tho, it would be nice to know if it is actually

[issue23979] Multiprocessing Pool.map pickles arguments passed to workers

Davin Potts added the comment: Though it's been discussed elsewhere, issue17560 is a good one where the matter of really big objects are being communicated between processes via multiprocessing. In it, Richard shares some detail about the implementation in multiprocessing, its constraints

[issue23979] Multiprocessing Pool.map pickles arguments passed to workers

Luis added the comment: Thanks for answer, although I still think I haven't made myself fully understood here, allow me to paraphrase: ...You need some means of transferring objects between processes, and pickling is the Python standard serialization method Yes, but the question that stands

[issue23979] Multiprocessing Pool.map pickles arguments passed to workers

Josh Rosenberg added the comment: The Pool workers are created eagerly, not lazily. That is, the fork occurs before map is called, and Python can't know that the objects passed as arguments were inherited in the first place (since they could be created after the Pool was created). If you

[issue23979] Multiprocessing Pool.map pickles arguments passed to workers

New submission from Luis: Hi, I've seen an odd behavior for multiprocessing Pool in Linux/MacOS: - import multiprocessing as mp from sys import getsizeof import numpy as np def f_test(x): print('process has received argument %s' % x ) r = x[:100] # return

[issue23979] Multiprocessing Pool.map pickles arguments passed to workers

Josh Rosenberg added the comment: The nature of a Pool precludes assumptions about the availability of specific objects in a forked worker process (particularly now that there are alternate methods of forking processes). Since the workers are spun up when the pool is created, objects created