Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-23 Thread Brett Cannon
On Fri Apr 18 2014 at 5:03:33 PM, Ezio Melotti ezio.melo...@gmail.com wrote: Hi, On Thu, Apr 17, 2014 at 9:09 PM, Brett Cannon bcan...@gmail.com wrote: On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote: Hi. On 14.4.2014. 23:51, Brett Cannon

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-20 Thread M.-A. Lemburg
On 18.04.2014 23:03, Ezio Melotti wrote: Hi, On Thu, Apr 17, 2014 at 9:09 PM, Brett Cannon bcan...@gmail.com wrote: On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote: Hi. On 14.4.2014. 23:51, Brett Cannon wrote: Now the question is whether the

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-18 Thread Ezio Melotti
Hi, On Thu, Apr 17, 2014 at 9:09 PM, Brett Cannon bcan...@gmail.com wrote: On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote: Hi. On 14.4.2014. 23:51, Brett Cannon wrote: Now the question is whether the maintenance cost of having to rebuild Python

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Brett Cannon
On Wed Apr 16 2014 at 4:53:25 PM, Terry Reedy tjre...@udel.edu wrote: On Wednesday, April 16, 2014 2:57:35 PM, Terry Reedy tjre...@udel.edu mailto:tjre...@udel.edu wrote: PS. In the user process sys.modules, there are numerous null entries like these:

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Jurko Gospodnetić
Hi. On 14.4.2014. 23:51, Brett Cannon wrote: Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. I would really love to have better startup times in production, but

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Guido van Rossum
On Thu, Apr 17, 2014 at 10:33 AM, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote: I would really love to have better startup times in production, What's your use case? I understand why startup time is important for Hg, but I'd like to understand what other situations occur frequently

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Brett Cannon
On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote: Hi. On 14.4.2014. 23:51, Brett Cannon wrote: Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Mark Young
I think he meant modifying the source files themselves for debugging purposes (e.g. putting print statements in itertools.py). 2014-04-17 14:09 GMT-04:00 Brett Cannon bcan...@gmail.com: On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote: Hi. On

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Antoine Pitrou
On Thu, 17 Apr 2014 18:09:22 + Brett Cannon bcan...@gmail.com wrote: I would really love to have better startup times in production, but I would also really hate to lose the ability to hack around in stdlib sources during development just to get better startup performance. In

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Guido van Rossum
I still do that! On Thu, Apr 17, 2014 at 11:17 AM, Antoine Pitrou solip...@pitrou.netwrote: On Thu, 17 Apr 2014 18:09:22 + Brett Cannon bcan...@gmail.com wrote: I would really love to have better startup times in production, but I would also really hate to lose the ability to

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Jurko Gospodnetić
Hi. On 17.4.2014. 20:15, Mark Young wrote: I think he meant modifying the source files themselves for debugging purposes (e.g. putting print statements in itertools.py). Exactly! :-) Best regards, Jurko Gospodnetić ___ Python-Dev

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Ethan Furman
On 04/17/2014 10:33 AM, Jurko Gospodnetić wrote: In general, what I really like about using Python for software development is the ability to open any stdlib file and easily go poking around using stuff like 'import pdb;pdb.set_trace()' or simple print statements. +1 -- ~Ethan~

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Giampaolo Rodola'
On Thu, Apr 17, 2014 at 8:17 PM, Antoine Pitrou solip...@pitrou.net wrote: On Thu, 17 Apr 2014 18:09:22 + Brett Cannon bcan...@gmail.com wrote: I would really love to have better startup times in production, but I would also really hate to lose the ability to hack around in

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Jurko Gospodnetić
Hi. On 17.4.2014. 19:57, Guido van Rossum wrote: On Thu, Apr 17, 2014 at 10:33 AM, Jurko Gospodnetić jurko.gospodne...@pke.hr mailto:jurko.gospodne...@pke.hr wrote: I would really love to have better startup times in production, What's your use case? I understand why startup time is

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Guido van Rossum
I'm sorry to keep asking dumb questions, but your description didn't job my understanding of what you are comparing here. What is slower than what? On Thu, Apr 17, 2014 at 11:47 AM, Brett Cannon bcan...@gmail.com wrote: Because people keep bringing it up, below is the results of hacking up the

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Donald Stufft
On Apr 17, 2014, at 2:23 PM, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote: Hi. On 17.4.2014. 19:57, Guido van Rossum wrote: On Thu, Apr 17, 2014 at 10:33 AM, Jurko Gospodnetić jurko.gospodne...@pke.hr mailto:jurko.gospodne...@pke.hr wrote: I would really love to have better

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Brett Cannon
On Thu Apr 17 2014 at 3:21:49 PM, Guido van Rossum gu...@python.org wrote: I'm sorry to keep asking dumb questions, but your description didn't job my understanding of what you are comparing here. What is slower than what? Startup where the stdlib is entirely in a zip file is slower than the

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Guido van Rossum
On Thu, Apr 17, 2014 at 1:31 PM, Brett Cannon bcan...@gmail.com wrote: On Thu Apr 17 2014 at 3:21:49 PM, Guido van Rossum gu...@python.org wrote: I'm sorry to keep asking dumb questions, but your description didn't job my understanding of what you are comparing here. What is slower than

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Martin v. Löwis
Am 17.04.14 20:47, schrieb Brett Cannon: Because people keep bringing it up, below is the results of hacking up the interpreter to include a sys.path entry for ./python35.zip instead of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ recursively. TL;DR, zipimport performance no

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Martin v. Löwis
Am 17.04.14 20:47, schrieb Brett Cannon: Because people keep bringing it up, below is the results of hacking up the interpreter to include a sys.path entry for ./python35.zip instead of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ recursively. TL;DR, zipimport performance no

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Brett Cannon
On Thu Apr 17 2014 at 5:21:14 PM, Martin v. Löwis mar...@v.loewis.de wrote: Am 17.04.14 20:47, schrieb Brett Cannon: Because people keep bringing it up, below is the results of hacking up the interpreter to include a sys.path entry for ./python35.zip instead of hard-coding to

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-17 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/17/2014 06:06 PM, Brett Cannon wrote: On Thu Apr 17 2014 at 5:21:14 PM, Martin v. Löwis mar...@v.loewis.de wrote: Am 17.04.14 20:47, schrieb Brett Cannon: Because people keep bringing it up, below is the results of hacking up the

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-16 Thread Martin v. Löwis
Am 14.04.14 23:51, schrieb Brett Cannon: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. [...] Thoughts? They still get read from disk, except that it is the operating system

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-16 Thread Terry Reedy
On 4/16/2014 12:25 PM, Martin v. Löwis wrote: Am 14.04.14 23:51, schrieb Brett Cannon: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. [...] Thoughts? They still get read from

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-16 Thread Dr. Brett Cannon
Is this Python 2 or 3? In Python 2 it means an attempt to perform a relative import failed but an absolute in succeeded, e.g. from idlelib you imported os, so import tried idlelib.is and then os. You should definitely consider using a future import to guarantee absolute imports. On Wednesday,

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-16 Thread Nick Coghlan
On 16 April 2014 12:25, Martin v. Löwis mar...@v.loewis.de wrote: Am 14.04.14 23:51, schrieb Brett Cannon: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. [...] Thoughts? They

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-16 Thread Terry Reedy
On Wednesday, April 16, 2014 2:57:35 PM, Terry Reedy tjre...@udel.edu mailto:tjre...@udel.edu wrote: PS. In the user process sys.modules, there are numerous null entries like these: sys.modules['idlelib.os'] sys.modules['idlelib.tokenize'__]

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Chris Angelico
On Tue, Apr 15, 2014 at 8:21 AM, Brett Cannon bcan...@gmail.com wrote: In my work environment (Python 2.7.2, all the heavy lifting done in C++), startup costs are dominated by dynamic linking of all our C++ libraries and their Boost wrappers: Sure, but not everyone uses Boost or has long

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Nick Coghlan
On 14 Apr 2014 18:37, Glenn Linderman v+pyt...@g.nevcal.com wrote: On 4/14/2014 2:51 PM, Brett Cannon wrote: Freezing everything except encodings.__init__, os, and _sysconfigdata, I suppose these are omitted because they can vary in different environments? But isn't Python built for a

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread M.-A. Lemburg
On 15.04.2014 09:45, Chris Angelico wrote: On Tue, Apr 15, 2014 at 8:21 AM, Brett Cannon bcan...@gmail.com wrote: In my work environment (Python 2.7.2, all the heavy lifting done in C++), startup costs are dominated by dynamic linking of all our C++ libraries and their Boost wrappers: Sure,

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Eric Snow
On Mon, Apr 14, 2014 at 3:51 PM, Brett Cannon bcan...@gmail.com wrote: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Eric Snow
On Tue, Apr 15, 2014 at 1:45 AM, Chris Angelico ros...@gmail.com wrote: Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; migrating hg to Py3 will (if I understand the above figures correctly) widen that gap, so any

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Daniel Holth
IIRC it is no longer the case that ZIP imports (involving only one file for a lot of modules) are much faster than regular FS imports? On Tue, Apr 15, 2014 at 10:34 AM, Eric Snow ericsnowcurren...@gmail.com wrote: On Tue, Apr 15, 2014 at 1:45 AM, Chris Angelico ros...@gmail.com wrote: Specific

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Brett Cannon
On Tue, Apr 15, 2014 at 11:19 AM, Daniel Holth dho...@gmail.com wrote: IIRC it is no longer the case that ZIP imports (involving only one file for a lot of modules) are much faster than regular FS imports? It's definitely minimized since Python 3.3 and the caching of stat results at the

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Antoine Pitrou
Le 15/04/2014 09:45, Chris Angelico a écrit : Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; Keep in mind those shoot-outs usually rely on large repositories and/or non-trivial operations, so startup time is

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Antoine Pitrou
Le 14/04/2014 23:51, Brett Cannon a écrit : It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Chris Angelico
On Wed, Apr 16, 2014 at 2:40 AM, Antoine Pitrou solip...@pitrou.net wrote: Le 15/04/2014 09:45, Chris Angelico a écrit : Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; Keep in mind those shoot-outs usually

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Guido van Rossum
Can we please stop the argument about Hg vs. Git? On Tue, Apr 15, 2014 at 12:54 PM, Chris Angelico ros...@gmail.com wrote: On Wed, Apr 16, 2014 at 2:40 AM, Antoine Pitrou solip...@pitrou.net wrote: Le 15/04/2014 09:45, Chris Angelico a écrit : Specific use-case that I can see:

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Chris Angelico
On Wed, Apr 16, 2014 at 4:54 AM, Guido van Rossum gu...@python.org wrote: Can we please stop the argument about Hg vs. Git? My apologies. All I was saying was that hg is a use case where startup performance really does matter, as opposed to the ones presented earlier in the thread where a

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Stefan Behnel
Brett Cannon, 14.04.2014 23:51: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-14 Thread Glenn Linderman
On 4/14/2014 2:51 PM, Brett Cannon wrote: consider freezing all the modules ... Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules all versus select number. So I'm guessing the proposal is to freeze all the modules that Python

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-14 Thread Skip Montanaro
On Mon, Apr 14, 2014 at 4:51 PM, Brett Cannon bcan...@gmail.com wrote: Thoughts? Interesting idea, but YAGNI? In my work environment (Python 2.7.2, all the heavy lifting done in C++), startup costs are dominated by dynamic linking of all our C++ libraries and their Boost wrappers: % time

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-14 Thread Brett Cannon
On Mon, Apr 14, 2014 at 6:15 PM, Skip Montanaro s...@pobox.com wrote: On Mon, Apr 14, 2014 at 4:51 PM, Brett Cannon bcan...@gmail.com wrote: Thoughts? Interesting idea, but YAGNI? Not at all. Think of every script you execute that's written in Python. One of the things the Mercurial folks

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-14 Thread Brett Cannon
On Mon, Apr 14, 2014 at 6:07 PM, Glenn Linderman v+pyt...@g.nevcal.comwrote: On 4/14/2014 2:51 PM, Brett Cannon wrote: consider freezing all the modules ... Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules all versus

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-14 Thread Glenn Linderman
On 4/14/2014 2:51 PM, Brett Cannon wrote: Freezing everything except encodings.__init__, os, and _sysconfigdata, I suppose these are omitted because they can vary in different environments? But isn't Python built for a particular environment... seems like os could be included? Seems like