Hi Edwin,
Thanks for providing some concrete numbers.
Is it expected that creating 100 processes takes 6.3ms per process, but
that creating 1000 process takes 40ms per process? That's over 6 times
as long in the latter case.
Cheers,
Mark.
On 12/06/2020 11:29 am, Edwin Zimmerman wrote:
On 6/12/2020 6:18 AM, Edwin Zimmerman wrote:
On 6/12/2020 5:08 AM, Paul Moore wrote:
On Fri, 12 Jun 2020 at 09:47, Mark Shannon <m...@hotpy.org> wrote:
Starting a new process is cheap. On my machine, starting a new Python
process takes under 1ms and uses a few Mbytes.
Is that on Windows or Unix? Traditionally, process creation has been
costly on Windows, which is why threads, and in-process solutions in
general, tend to be more common on that platform. I haven't done
experiments recently, but I do tend to avoid multiprocess-type
solutions on Windows "just in case". I know that evaluating a new
feature based on unsubstantiated assumptions informed by "it used to
be like this" is ill-advised, but so is assuming that everything will
be OK based on experience on a single platform :-)
Here's a test on Windows 10, 4 logical cpus, 8 GB of ram:
timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100,
globals=globals())
0.6297528999999997
timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000,
globals=globals())
40.281721199999964
Or this way:
timeit.timeit("""os.system('python.exe -c "exit()"')""",number=100,
globals=globals())
17.461259299999995
--Edwin
For comparison, on a single core linux cloud server with 512 mb of ram:
timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100,
globals=globals())
0.354354709998006
timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000,
globals=globals())
3.847851719998289
So yeah, process creation is still rather costly on Windows.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/BLBNMZXKYDKRDYRFNEHYPMNHNFMOU4WG/
Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/XKAMVXHG3FJDNAGFWYCLUGM3L7VFV52P/
Code of Conduct: http://python.org/psf/codeofconduct/
