The worker pool approach is probably the way to go, but there is a fair bit of overhead to creating a multiprocessing job. So fewer, larger jobs are faster than many small jobs.
So you do want to make the jobs as large as you can without wasting CPU time. -CHB On Wed, Aug 18, 2021 at 9:09 AM Barry <ba...@barrys-emacs.org> wrote: > > > > On 18 Aug 2021, at 16:03, Chris Angelico <ros...@gmail.com> wrote: > > > > On Thu, Aug 19, 2021 at 12:52 AM Marc-Andre Lemburg <m...@egenix.com> > wrote: > >> > >>> On 18.08.2021 15:58, Chris Angelico wrote: > >>> On Wed, Aug 18, 2021 at 10:37 PM Joao S. O. Bueno < > jsbu...@python.org.br> wrote: > >>>> > >>>> So, > >>>> It is out of scope of Pythonmultiprocessing, and, as I perceive it, > from > >>>> the stdlib as a whole to be able to allocate specific cores for each > subprocess - > >>>> that is automatically done by the O.S. (and of course, the O.S. > having an interface > >>>> for it, one can write a specific Python library which would allow > this granularity, > >>>> and it could even check core capabilities). > >>> > >>> Python does have a way to set processor affinity, so it's entirely > >>> possible that this would be possible. Might need external tools > >>> though. > >> > >> There's os.sched_setaffinity(pid, mask) you could use from within > >> a Python task scheduler, if this is managing child processes (you need > >> the right permissions to set the affinity). > > > > Right; I meant that it might require external tools to find out which > > processors you want to align with. > > > >> Or you could use the taskset command available on Linux to fire > >> up a process on a specific CPU core. lscpu gives you more insight > >> into the installed set of available cores. > > > > Yes, those sorts of external tools. > > > > It MAY be possible to learn about processors by reading /proc/cpuinfo, > > but that'd still be OS-specific (no idea which Unix-like operating > > systems have that, and certainly Windows doesn't). > > And next you find out that you have to understand the NUMA details > of your system because the memory attached to the CPUs is not the same > speed. > > > > > All in all, far easier to just divide the job into far more pieces > > than you have processors, and then run a pool. > > As other already stated using a worker pool solves this problem for you. > All you have to do it break your big job into suitable small pieces. > > Barry > > > > ChrisA > > _______________________________________________ > > Python-ideas mailing list -- python-ideas@python.org > > To unsubscribe send an email to python-ideas-le...@python.org > > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/UQNSUSHUONT4AO6NJEPEUENQG2AINAMO/ > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/62AXMS62J2H7TBHANIXZTTS2RJPUZZ5Z/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LIJV3DAK3I6J3QQFJ2HTGVJHHLQZIHCL/ Code of Conduct: http://python.org/psf/codeofconduct/
