Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Fri Apr 18 2014 at 5:03:33 PM, Ezio Melotti wrote: > Hi, > > On Thu, Apr 17, 2014 at 9:09 PM, Brett Cannon wrote: > > > > > > On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić > > wrote: > >> > >>Hi. > >> > >> On 14.4.2014. 23:51, Brett Cannon wrote: > >> > Now the question is whether

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On 18.04.2014 23:03, Ezio Melotti wrote: > Hi, > > On Thu, Apr 17, 2014 at 9:09 PM, Brett Cannon wrote: >> >> >> On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić >> wrote: >>> >>>Hi. >>> >>> On 14.4.2014. 23:51, Brett Cannon wrote: Now the question is whether the maintenance cost of

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

Hi, On Thu, Apr 17, 2014 at 9:09 PM, Brett Cannon wrote: > > > On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić > wrote: >> >>Hi. >> >> On 14.4.2014. 23:51, Brett Cannon wrote: >> > Now the question is whether the maintenance cost of having to rebuild >> > Python for a select number of st

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/17/2014 06:06 PM, Brett Cannon wrote: > On Thu Apr 17 2014 at 5:21:14 PM, "Martin v. Löwis" > wrote: > >> Am 17.04.14 20:47, schrieb Brett Cannon: >>> Because people keep bringing it up, below is the results of >>> hacking up the interpreter to

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Thu Apr 17 2014 at 5:21:14 PM, "Martin v. Löwis" wrote: > Am 17.04.14 20:47, schrieb Brett Cannon: > > Because people keep bringing it up, below is the results of hacking up > > the interpreter to include a sys.path entry for ./python35.zip instead > > of hard-coding to /usr/lib/python35.zip a

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

Am 17.04.14 20:47, schrieb Brett Cannon: > Because people keep bringing it up, below is the results of hacking up > the interpreter to include a sys.path entry for ./python35.zip instead > of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ > recursively. TL;DR, zipimport performance

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

Am 17.04.14 20:47, schrieb Brett Cannon: > Because people keep bringing it up, below is the results of hacking up > the interpreter to include a sys.path entry for ./python35.zip instead > of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ > recursively. TL;DR, zipimport performance

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Thu, Apr 17, 2014 at 1:31 PM, Brett Cannon wrote: > > On Thu Apr 17 2014 at 3:21:49 PM, Guido van Rossum > wrote: > >> I'm sorry to keep asking dumb questions, but your description didn't job >> my understanding of what you are comparing here. What is slower than what? >> > > Startup where t

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Thu Apr 17 2014 at 3:21:49 PM, Guido van Rossum wrote: > I'm sorry to keep asking dumb questions, but your description didn't job > my understanding of what you are comparing here. What is slower than what? > Startup where the stdlib is entirely in a zip file is slower than the status quo of

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Apr 17, 2014, at 2:23 PM, Jurko Gospodnetić wrote: > Hi. > > On 17.4.2014. 19:57, Guido van Rossum wrote: >> On Thu, Apr 17, 2014 at 10:33 AM, Jurko Gospodnetić >> mailto:jurko.gospodne...@pke.hr>> wrote: >> >> I would really love to have better startup times in production, >> >> Wh

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

I'm sorry to keep asking dumb questions, but your description didn't job my understanding of what you are comparing here. What is slower than what? On Thu, Apr 17, 2014 at 11:47 AM, Brett Cannon wrote: > Because people keep bringing it up, below is the results of hacking up the > interpreter to

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

Hi. On 17.4.2014. 19:57, Guido van Rossum wrote: On Thu, Apr 17, 2014 at 10:33 AM, Jurko Gospodnetić mailto:jurko.gospodne...@pke.hr>> wrote: I would really love to have better startup times in production, What's your use case? I understand why startup time is important for Hg, but I'

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Thu, Apr 17, 2014 at 8:17 PM, Antoine Pitrou wrote: > On Thu, 17 Apr 2014 18:09:22 + > Brett Cannon wrote: > > > > > >I would really love to have better startup times in production, but > I > > > would also really hate to lose the ability to hack around in stdlib > > > sources during

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On 04/17/2014 10:33 AM, Jurko Gospodnetić wrote: In general, what I really like about using Python for software development is the ability to open any stdlib file and easily go poking around using stuff like 'import pdb;pdb.set_trace()' or simple print statements. +1 -- ~Ethan~ _

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

Hi. On 17.4.2014. 20:15, Mark Young wrote: I think he meant modifying the source files themselves for debugging purposes (e.g. putting print statements in itertools.py). Exactly! :-) Best regards, Jurko Gospodnetić ___ Python-Dev mailing

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

I still do that! On Thu, Apr 17, 2014 at 11:17 AM, Antoine Pitrou wrote: > On Thu, 17 Apr 2014 18:09:22 + > Brett Cannon wrote: > > > > > >I would really love to have better startup times in production, but > I > > > would also really hate to lose the ability to hack around in stdlib >

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Thu, 17 Apr 2014 18:09:22 + Brett Cannon wrote: > > > >I would really love to have better startup times in production, but I > > would also really hate to lose the ability to hack around in stdlib > > sources during development just to get better startup performance. > > > >In gener

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

I think he meant modifying the source files themselves for debugging purposes (e.g. putting print statements in itertools.py). 2014-04-17 14:09 GMT-04:00 Brett Cannon : > > > On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić < > jurko.gospodne...@pke.hr> wrote: > >>Hi. >> >> On 14.4.2014.

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić < jurko.gospodne...@pke.hr> wrote: >Hi. > > On 14.4.2014. 23:51, Brett Cannon wrote: > > Now the question is whether the maintenance cost of having to rebuild > > Python for a select number of stdlib modules is enough to warrant > > putting i

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Thu, Apr 17, 2014 at 10:33 AM, Jurko Gospodnetić < jurko.gospodne...@pke.hr> wrote: > I would really love to have better startup times in production, What's your use case? I understand why startup time is important for Hg, but I'd like to understand what other situations occur frequently en

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

Hi. On 14.4.2014. 23:51, Brett Cannon wrote: Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. I would really love to have better startup times in production, but

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Wed Apr 16 2014 at 4:53:25 PM, Terry Reedy wrote: > > On Wednesday, April 16, 2014 2:57:35 PM, Terry Reedy > > wrote: > > > PS. In the user process sys.modules, there are numerous null > > entries like these: > > >>> sys.modules['idlelib.os'] > >

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

> On Wednesday, April 16, 2014 2:57:35 PM, Terry Reedy > wrote: > PS. In the user process sys.modules, there are numerous null > entries like these: > >>> sys.modules['idlelib.os'] > >>> sys.modules['idlelib.tokenize'__] > >>> sys.modules['idlel

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On 16 April 2014 12:25, "Martin v. Löwis" wrote: > Am 14.04.14 23:51, schrieb Brett Cannon: >> It was realized during PyCon that since we are freezing importlib we >> could now consider freezing all the modules to cut out having to stat or >> read them from disk. > [...] >> Thoughts? > > They stil

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

Is this Python 2 or 3? In Python 2 it means an attempt to perform a relative import failed but an absolute in succeeded, e.g. from idlelib you imported os, so import tried idlelib.is and then os. You should definitely consider using a future import to guarantee absolute imports. On Wednesday, Apri

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On 4/16/2014 12:25 PM, "Martin v. Löwis" wrote: Am 14.04.14 23:51, schrieb Brett Cannon: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. [...] Thoughts? They still get read from

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

Am 14.04.14 23:51, schrieb Brett Cannon: > It was realized during PyCon that since we are freezing importlib we > could now consider freezing all the modules to cut out having to stat or > read them from disk. [...] > Thoughts? They still get read from disk, except that it is the operating system

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

Brett Cannon, 14.04.2014 23:51: > It was realized during PyCon that since we are freezing importlib we could > now consider freezing all the modules to cut out having to stat or read > them from disk. So for day 1 of the sprints I I decided to hack up a > proof-of-concept to see what kind of perfor

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Wed, Apr 16, 2014 at 4:54 AM, Guido van Rossum wrote: > Can we please stop the argument about Hg vs. Git? My apologies. All I was saying was that hg is a use case where startup performance really does matter, as opposed to the ones presented earlier in the thread where a process stays in memor

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

Can we please stop the argument about Hg vs. Git? On Tue, Apr 15, 2014 at 12:54 PM, Chris Angelico wrote: > On Wed, Apr 16, 2014 at 2:40 AM, Antoine Pitrou > wrote: > > Le 15/04/2014 09:45, Chris Angelico a écrit : > > > >> > >> Specific use-case that I can see: Mercurial. In a git vs hg shoot

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Wed, Apr 16, 2014 at 2:40 AM, Antoine Pitrou wrote: > Le 15/04/2014 09:45, Chris Angelico a écrit : > >> >> Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, >> git will usually win on performance, and hg is using Py2; > > > Keep in mind those shoot-outs usually rely on lar

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

Le 14/04/2014 23:51, Brett Cannon a écrit : It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

Le 15/04/2014 09:45, Chris Angelico a écrit : Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; Keep in mind those shoot-outs usually rely on large repositories and/or non-trivial operations, so startup time is not

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Tue, Apr 15, 2014 at 11:19 AM, Daniel Holth wrote: > IIRC it is no longer the case that ZIP imports (involving only one > file for a lot of modules) are much faster than regular FS imports? > It's definitely minimized since Python 3.3 and the caching of stat results at the directory level for

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

IIRC it is no longer the case that ZIP imports (involving only one file for a lot of modules) are much faster than regular FS imports? On Tue, Apr 15, 2014 at 10:34 AM, Eric Snow wrote: > On Tue, Apr 15, 2014 at 1:45 AM, Chris Angelico wrote: >> Specific use-case that I can see: Mercurial. In a

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Tue, Apr 15, 2014 at 1:45 AM, Chris Angelico wrote: > Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, > git will usually win on performance, and hg is using Py2; migrating hg > to Py3 will (if I understand the above figures correctly) widen that > gap, so any improvement

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Mon, Apr 14, 2014 at 3:51 PM, Brett Cannon wrote: > It was realized during PyCon that since we are freezing importlib we could > now consider freezing all the modules to cut out having to stat or read them > from disk. So for day 1 of the sprints I I decided to hack up a > proof-of-concept to s

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On 15.04.2014 09:45, Chris Angelico wrote: > On Tue, Apr 15, 2014 at 8:21 AM, Brett Cannon wrote: >>> In my work environment (Python 2.7.2, all the heavy lifting done in >>> C++), startup costs are dominated by dynamic linking of all our C++ >>> libraries and their Boost wrappers: >> >> >> Sure, b

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On 14 Apr 2014 18:37, "Glenn Linderman" wrote: > > On 4/14/2014 2:51 PM, Brett Cannon wrote: >> >> Freezing everything except encodings.__init__, os, and _sysconfigdata, > > > I suppose these are omitted because they can vary in different environments? > > But isn't Python built for a particular e

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Tue, Apr 15, 2014 at 8:21 AM, Brett Cannon wrote: >> In my work environment (Python 2.7.2, all the heavy lifting done in >> C++), startup costs are dominated by dynamic linking of all our C++ >> libraries and their Boost wrappers: > > > Sure, but not everyone uses Boost or has long running proc

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On 4/14/2014 2:51 PM, Brett Cannon wrote: Freezing everything except encodings.__init__, os, and _sysconfigdata, I suppose these are omitted because they can vary in different environments? But isn't Python built for a particular environment... seems like os could be included? Seems like it

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Mon, Apr 14, 2014 at 6:07 PM, Glenn Linderman wrote: > On 4/14/2014 2:51 PM, Brett Cannon wrote: > > consider freezing all the modules > > ... > > Now the question is whether the maintenance cost of having to rebuild > Python for a select number of stdlib modules > > > "all" versus "select n

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Mon, Apr 14, 2014 at 6:15 PM, Skip Montanaro wrote: > On Mon, Apr 14, 2014 at 4:51 PM, Brett Cannon wrote: > > Thoughts? > > Interesting idea, but YAGNI? > Not at all. Think of every script you execute that's written in Python. One of the things the Mercurial folks say is hindering any motiv

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On Mon, Apr 14, 2014 at 4:51 PM, Brett Cannon wrote: > Thoughts? Interesting idea, but YAGNI? In my work environment (Python 2.7.2, all the heavy lifting done in C++), startup costs are dominated by dynamic linking of all our C++ libraries and their Boost wrappers: % time python -c 'import trad

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

On 4/14/2014 2:51 PM, Brett Cannon wrote: consider freezing all the modules ... Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules "all" versus "select number". So I'm guessing the proposal is to freeze all the modules that Pyth