[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Changes by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Changes by Charles-François Natali : -- stage: needs patch -> resolved status: open -> closed ___ Python tracker ___

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Serhiy Storchaka added the comment: Argument Clinic code was not regenerated. Actually the commit breaks Argument Clinic. $ make clinic ./python -E ./Tools/clinic/clinic.py --make Error in file ./Modules/posixmodule.c on line 11211: Docstring for os.cpu_count does not have a summary line!

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Charles-François Natali added the comment: It's just a doc improvement, I'm not convinced it really needs backporting. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530 ___

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Berker Peksag added the comment: It would be nice to backport the patch to the 3.4 and 3.5 branches. -- nosy: +berker.peksag ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530 ___

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Roundup Robot added the comment: New changeset b9e3838e9664 by Charles-François Natali in branch 'default': Issue #23530: Improve os.cpu_count() description. https://hg.python.org/cpython/rev/b9e3838e9664 -- nosy: +python-dev ___ Python tracker

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Charles-François Natali added the comment: Committed. Julian, thanks for the patch! -- resolution: - fixed stage: - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Julian Taylor added the comment: any comments on the doc changes? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530 ___ ___ Python-bugs-list

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Julian Taylor added the comment: attached documentation update patch. -- keywords: +patch Added file: http://bugs.python.org/file38369/0001-Issue-23530-Update-documentation-clarify-relation-of.patch ___ Python tracker rep...@bugs.python.org

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

eryksun added the comment: Well, we already expose CPU affinity: import os os.sched_getaffinity(0) {0} os.sched_getaffinity only exists on some POSIX systems, such as Linux. For Windows, here's a ctypes version of sched_getaffinity and sched_setaffinity: import sys from ctypes

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Davin Potts added the comment: Adding an option does sound like a better possibility. Still, when I start looking through the examples that psutil provides, it reminds me how this is but one small piece of a much larger picture which psutil has done a nice, focused job of working to address.

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Julian Taylor added the comment: certainly for anything that needs good control over affinity psutils is the best choice, but I'm not arguing to implement full process control in python. I only want python to provide the number of cores one can work on to make best use of the available

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Charles-François Natali added the comment: Well, we already expose CPU affinity: import os os.sched_getaffinity(0) {0} IMO the current implementation is sufficient (and talking about overcommitting for CPU is a bit moot if you're using virtual machine anyways). The current documentation

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Julian Taylor added the comment: oh thats great so python already has what I want. Then just an small documentation update would be good, I'll have a go at a patch later. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530 ___ ___ Python-bugs-list

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

New submission from Julian Taylor: multiprocessing.cpu_count and os.cpu_count which are often used to determine how many processes one can run in parallel do not respect the cpuset which may limit the process to only a subset of online cpus leading to heavy oversubscription in e.g.

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Julian Taylor added the comment: I do agree that its probably safer to not change the default return value. But adding a option (or new function) would still be good, the number of available cpus is more often the number you actually want in practice. To the very least the documentation should

[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

Davin Potts added the comment: Detecting the number of processors available to a process is a distinct concept from reporting the number of processors present on a system. cpu_count is currently focused on the latter. Functionality to report the number of effectively-available processors is