[issue32986] multiprocessing, default assumption of Pool size unhelpful

Change by Eryk Sun : -- nosy: +eryksun ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Matt Harvey added the comment: @njs your sketch in msg313406 looks good. Probably better to go with OMP_NUM_THREADS than NCPUS. M -- ___ Python tracker

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Nathaniel Smith added the comment: > You mean duplicating "nproc"'s logic in Python? Yeah. > If someone wants to do the grunt work of implementing/testing it... Well, that's true of any bug fix / improvement :-). The logic isn't terribly complicated though, something roughly

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Nathaniel Smith added the comment: I can't find any evidence that NPROCS is used by other batch schedulers (I looked at SLURM, Torque, and SGE). @M J Harvey, do you have any other examples of systems that use it? -- ___ Python

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Antoine Pitrou added the comment: > So this seems like a reasonable heuristic approach to me. You mean duplicating "nproc"'s logic in Python? If someone wants to do the grunt work of implementing/testing it... There's also the question of how that affects non-scientific

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Nathaniel Smith added the comment: That stackoverflow thread points to the GNU coreutils 'nproc', which is an interesting compendium of knowledge about this problem. It looks like their algorithm is roughly: 1. Determine how many CPUs *could* this program access, by going

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Antoine Pitrou added the comment: Note that to avoid any kind of environment variable-driven Denial of Service, we should probably ignore NCPUS when sys.flags.ignore_environment is set. -- ___ Python tracker

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Matt Harvey added the comment: Hi, No, using the affinity's not useful to us as, in the general case, the batch system (PBS Pro in our case) isn't using cgroups or cpusets (it does control ave cpu use by monitoring rusage of the process group). Several other batch

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Antoine Pitrou added the comment: I don't think we want to hardcode special cases for each resource-limiting framework out there: some people care about Docker, others about cgroups, CPU affinity settings, etc. NCPUS has the nice property that each of those frameworks can

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Matthew Rocklin added the comment: I agree that this is a common issue. We see it both when people use batch schedulers as well as when people use Docker containers. I don't have enough experience with batch schedulers to verify that NCPUS is commonly set. I would

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Change by Antoine Pitrou : -- type: behavior -> enhancement versions: -Python 3.6, Python 3.7 ___ Python tracker ___

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Nathaniel Smith added the comment: This is a duplicate of bpo-26692 and bpo-23530, and possibly others. My impression is that len(os.sched_getaffinity(os.getpid())) is the Right Guess. Currently sched_getaffinity isn't implemented on Windows, but bpo-23530 has some example

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Antoine Pitrou added the comment: Matt, what do you think about this proposal? Is NCPUS the right thing to look at? -- nosy: +Matthew Rocklin type: -> behavior versions: +Python 3.7, Python 3.8 ___ Python tracker

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Change by Ned Deily : -- nosy: +davin, pitrou ___ Python tracker ___ ___ Python-bugs-list

[issue32986] multiprocessing, default assumption of Pool size unhelpful

Change by Roundup Robot : -- keywords: +patch pull_requests: +5726 stage: -> patch review ___ Python tracker ___

[issue32986] multiprocessing, default assumption of Pool size unhelpful

New submission from M J Harvey : Hi, multiprocessing's default assumption about Pool size is os.cpu_count() ie all the cores visible to the OS. This is tremendously unhelpful when running multiprocessing code inside an HPC batch system (PBS Pro in my case), as there's