Re: [Python-Dev] python process creation overhead

2014-05-13 Thread Gregory Szorc
On 5/12/2014 8:43 PM, Nick Coghlan wrote: On 13 May 2014 10:19, dw+python-...@hmmz.org wrote: On Mon, May 12, 2014 at 04:22:52PM -0700, Gregory Szorc wrote: Why can't Python start as quickly as Perl or Ruby? On my heavily abused Core 2 Macbook with 9 .pth files, 2.7 drops from 81ms

Re: [Python-Dev] python process creation overhead

2014-05-13 Thread Stephen J. Turnbull
Gregory Szorc writes: But the great many of us still on 2.7 likely won't see a benefit, correct? How insane would it be for people to do things like pass -S in the shebang and manually implement the parts of site.py that are actually needed? Well, since it probably won't work wink/

Re: [Python-Dev] python process creation overhead

2014-05-12 Thread Gregory Szorc
On 5/10/2014 2:46 PM, Victor Stinner wrote: Le 10 mai 2014 22:51, Gregory Szorc gregory.sz...@gmail.com mailto:gregory.sz...@gmail.com a écrit : Furthermore, Python 3 appears to be 50% slower than Python 2. Please mention the minor version. It looks like you compared 2.7 and 3.3. Please

Re: [Python-Dev] python process creation overhead

2014-05-12 Thread dw+python-dev
On Mon, May 12, 2014 at 04:22:52PM -0700, Gregory Szorc wrote: Why can't Python start as quickly as Perl or Ruby? On my heavily abused Core 2 Macbook with 9 .pth files, 2.7 drops from 81ms startup to 20ms by specifying -S, which disables site.py. Oblitering the .pth files immediately knocks

Re: [Python-Dev] python process creation overhead

2014-05-12 Thread Nick Coghlan
On 13 May 2014 10:19, dw+python-...@hmmz.org wrote: On Mon, May 12, 2014 at 04:22:52PM -0700, Gregory Szorc wrote: Why can't Python start as quickly as Perl or Ruby? On my heavily abused Core 2 Macbook with 9 .pth files, 2.7 drops from 81ms startup to 20ms by specifying -S, which disables

[Python-Dev] python process creation overhead

2014-05-10 Thread Gregory Szorc
I was investigating speeding up Mercurial's test suite (it runs ~13,000 Python processes) and I believe I've identified CPython process/interpreter creation and destruction as sources of significant overhead and thus a concern for any CPython user. Full details are at [1]. tl;dr 10-18% of CPU

Re: [Python-Dev] python process creation overhead

2014-05-10 Thread Antoine Pitrou
Hello, On Sat, 10 May 2014 13:05:54 -0700 Gregory Szorc gregory.sz...@gmail.com wrote: I was investigating speeding up Mercurial's test suite (it runs ~13,000 Python processes) and I believe I've identified CPython process/interpreter creation and destruction as sources of significant

Re: [Python-Dev] python process creation overhead

2014-05-10 Thread Victor Stinner
Le 10 mai 2014 22:51, Gregory Szorc gregory.sz...@gmail.com a écrit : Furthermore, Python 3 appears to be 50% slower than Python 2. Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time. There is still

Re: [Python-Dev] python process creation overhead

2014-05-10 Thread Donald Stufft
On May 10, 2014, at 5:46 PM, Victor Stinner victor.stin...@gmail.com wrote: Le 10 mai 2014 22:51, Gregory Szorc gregory.sz...@gmail.com a écrit : Furthermore, Python 3 appears to be 50% slower than Python 2. Please mention the minor version. It looks like you compared 2.7 and 3.3. Please

Re: [Python-Dev] python process creation overhead

2014-05-10 Thread Guido van Rossum
Yeah, but 200 test in 30 minutes is 9 *seconds* per test -- the Python startup time is only a tiny fraction of that (20-40 *milliseconds*). On Sat, May 10, 2014 at 3:33 PM, Donald Stufft don...@stufft.io wrote: On May 10, 2014, at 5:46 PM, Victor Stinner victor.stin...@gmail.com wrote: Le

Re: [Python-Dev] python process creation overhead

2014-05-10 Thread Donald Stufft
Yes right, sorry I didn’t mean to imply that all that time was spent in the Python start up time. I’ve personally never actually spent time to figure out which part of that was slow because getting visibility inside of a subprocess.Popen is a pain and I’m slowly trying to rewrite our tests to not