Re: [Python-Dev] [numpy wishlist] PyMem_*Calloc
Indeed, that's very reasonable. Please open an issue on the tracker! ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Tue, Apr 15, 2014 at 8:21 AM, Brett Cannon bcan...@gmail.com wrote: In my work environment (Python 2.7.2, all the heavy lifting done in C++), startup costs are dominated by dynamic linking of all our C++ libraries and their Boost wrappers: Sure, but not everyone uses Boost or has long running processes where startup time is minuscule compared to the total execution time. Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; migrating hg to Py3 will (if I understand the above figures correctly) widen that gap, so any improvement done to startup performance will give a very real advantage. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On 14 Apr 2014 18:37, Glenn Linderman v+pyt...@g.nevcal.com wrote: On 4/14/2014 2:51 PM, Brett Cannon wrote: Freezing everything except encodings.__init__, os, and _sysconfigdata, I suppose these are omitted because they can vary in different environments? But isn't Python built for a particular environment... seems like os could be included? Seems like it would be helpful to have the utf8 encoding preloaded both to encourage people to use it rather than something else for the load-time performance gain (although likely minuscule for one encoding), and because they might as well, since they are spending the memory on it anyway! :) Via some moderately arcane hackery, UTF-8 support is already built in to the Py3 interpreter :) Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On 15.04.2014 09:45, Chris Angelico wrote: On Tue, Apr 15, 2014 at 8:21 AM, Brett Cannon bcan...@gmail.com wrote: In my work environment (Python 2.7.2, all the heavy lifting done in C++), startup costs are dominated by dynamic linking of all our C++ libraries and their Boost wrappers: Sure, but not everyone uses Boost or has long running processes where startup time is minuscule compared to the total execution time. Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; migrating hg to Py3 will (if I understand the above figures correctly) widen that gap, so any improvement done to startup performance will give a very real advantage. You might want to have a look at this project: http://pyrun.org/ It's currently Python 2 only, but we'll try to get it to work with Python 3.4 as well, now that freeze.py and some other bits have been fixed to make it work again. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Mon, Apr 14, 2014 at 3:51 PM, Brett Cannon bcan...@gmail.com wrote: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of performance gain it would get. Freezing everything except encodings.__init__, os, and _sysconfigdata, it speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14% from the slowdown (27% slower vs. 41% slower). The full results are at the end of the email. Nice. I was hoping it would be even bigger (given the hyper-focus people put on the impact of FS-access on startup time imports), but this is definitely a significant improvement. I wonder then where the remaining slowdown lies; are there any remaining low hanging fruit elsewhere? Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. Yeah. Definitely the big question. Who cares the most about startup time? Would this improvement please them? Does that alone make it worth the increased maintenance burden? Is that group big enough or important enough to justify it? At the very least it may be good for the PR value alone, but the maintenance cost will long outlive the PR benefit. :) My guess is the best approach would be adding a Lib/_frozen directory where any modules that we treat like this would be kept to act as a reminder that you need to rebuild for them (I would probably move importlib/_boostrap.py as well to make this consistent). That makes sense. I also wonder if we could accomplish the same thing with a marker (e.g. a comment) in each related module (and leave them where they are). A marker would allow for easily finding the freezable modules. Personally, I think the speedup would be worth it if it doesn't add significant to the maintenance burden. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Tue, Apr 15, 2014 at 1:45 AM, Chris Angelico ros...@gmail.com wrote: Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; migrating hg to Py3 will (if I understand the above figures correctly) widen that gap, so any improvement done to startup performance will give a very real advantage. Perhaps not so much a very real advantage as less of a distraction. It's still significantly slower than 2.7. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2migr8
On 14 Apr 2014 17:17, Donald Stufft don...@stufft.io wrote: Now I will admit I personally have probably had a harder time than some others because of the nature of the things I was trying to work on, and lately it’s gotten better (although I think that’s partially because I’m more known now, and I think in general the experience of contributing to CPython changes depending on who you are, so the more integrated into the culture you are, the less likely you are to see the issues and unfortunately those people are also the ones with the most power to change it). I do however think that just in general it might be getting better too? From my perspective, the main issue is that new contributors typically don't acknowledge the large number of competing constraints we're balancing in core development, and hence grow frustrated when we reject their obvious idea on the basis of a concern that seems completely irrelevant to them. (Due to my background, I'll occasionally phrase this along the lines of Supporting submarines and secure enclaves matters to me, but most open source devs don't care). Core development for a programming language is a genuinely hard problem, and one that often involves thinking on timescales of years and decades rather than the weeks and months involved in more typical development environments. There's a huge impedance mismatch in expectations that can lead to major communication problems if people don't take the time to lurk for a while and get a feel for the kinds of novel concerns that arise when sitting at the centre of a massive ecosystem like Python's. Specific details are hard because it’s nothing major and obvious like having Linus go off on rants and tearing things apart, it’s death by a thousand cuts so it’s hard to point a finger at one behavior (or a few behaviors) and look at them in isolation and “see” it. That being said I’m more than happy to *try* and explain it, but right this moment I don’t have a lot of time as I’m getting ready to step out the door, but I didn’t want to leave this email hanging without a reply. From my perspective, the heart of the issue is personal time management on the part of the core development team. We're highly cognizant of the limits of what can be sustained through volunteer development, and one of the big issues is that loading volunteers up with too many activities that they feel obliged to do but don't find inherently enjoyable is a recipe for burnout. So we default to no not just because the number of ways we can make Python worse is unbounded, but also because the time we have available to do anything at all is incredibly limited. Now, consider that we're operating in an environment where multi-billion dollar companies are relying on our software while making only relatively small contributions to its ongoing support and evolution, and where we have multiple prominent community members wishing vocally (and encouraging others to advocate) for the core development team to devote our volunteer efforts to improving a legacy language rather than the new one we shifted en masse to working on instead. (Note that the latter actually makes about as much sense to me as telling the Rust and Go developers they should spend their free time working on C compilers instead because the latter would be more immediately useful to commercial users) On top of this, various outreach efforts to encourage new contributors are working, and working well enough that they are actually *exceeding* the existing team's ability to effectively *absorb* those new contributors. So, tremendously high stress levels for the core development team, on top of whatever stress we have to deal with in our personal and professional lives. This stress then manifests as irritability, impatience and outright anger - my own stress levels reached sufficiently high levels earlier this year that I almost decided to walk away from my job. Red Hat management intervened to keep that from actually happening, but I think it's important to make that incident more public to help people understand how utterly unsustainable the status quo is when it comes to CPython - we can't keep relying on almost entirely volunteer effort to maintain 2.7 LTS, 3.x, all the python.org infrastructure *and* the PSF without also anticipating complete and total burnout of some highly invested contributors. PEP 462 describes some ideas to make more effective use of core developer time, and to potentially distribute particular tasks to better suited groups of people (such as the tutorial and HOWTO guides), but in itself involves a substantial amount of up front work. That's where Guido's suggestion of corporations offering more 50% jobs for core developers comes in. That 50% time wouldn't be about working on things we would have done anyway - it would be about working on things that are difficult to get jumpstarted with purely volunteer effort. More work on 2.7 maintenance, more work on CPython
Re: [Python-Dev] Appeal for reviews
On 14 Apr 2014 08:42, R. David Murray rdmur...@bitdance.com wrote: On Mon, 14 Apr 2014 08:18:13 -0400, Nick Coghlan ncogh...@gmail.com wrote: On 14 Apr 2014 01:56, Stephen J. Turnbull step...@xemacs.org wrote: mar...@v.loewis.de writes: For gaining commit access, it's really more important that the patch is factually finished, than that it's author believes it to. If people get it right the first time often enough, they get commit access. Yes, that's what I had in mind, but I guess I explained it poorly. We should capture this discussion clearly in the dev guide. Even if we switch to a core reviewer model at some point (as I propose in PEP 462), the criteria for core reviewer status will match those for core commiter status. There are actually a few things I'm personally looking for: * good judgement on when a patch is finished enough to merge * good judgement on whether a change is a new feature or a bug fix * good judgement whether a new feature is worth the additional cognitive burden * good ability to assess backwards compatibility risks * sufficient humility to answer I don't know to the above questions when appropriate and ask the relevant domain experts, their sponsoring mentor, the core-mentorship list or python-dev at large for advice on what to do When considering who we give commit access to, I think we would be well served to start giving more weight to the quality of the code reviews that someone does. Producing good patches is important, but even without moving the infrastructure to Nick's core reviewer model, doing those reviews is an important part of what committers do, and it is a different (although related) skill to that of writing good patches. Or to put it another way, I'd like to encourage contributors who want to get commit access to focus just as much on doing good reviews as they do on writing new patches. Currently the focus is all on getting patches accepted. Huh, I hadn't thought of it that way before, but it's a very good point. Cheers, Nick. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Appeal for reviews
On Tue, Apr 15, 2014 at 10:41 AM, Nick Coghlan ncogh...@gmail.com wrote: On 14 Apr 2014 08:42, R. David Murray rdmur...@bitdance.com wrote: When considering who we give commit access to, I think we would be well served to start giving more weight to the quality of the code reviews that someone does. Producing good patches is important, but even without moving the infrastructure to Nick's core reviewer model, doing those reviews is an important part of what committers do, and it is a different (although related) skill to that of writing good patches. Or to put it another way, I'd like to encourage contributors who want to get commit access to focus just as much on doing good reviews as they do on writing new patches. Currently the focus is all on getting patches accepted. Huh, I hadn't thought of it that way before, but it's a very good point. Indeed. This is also a reflection of a good hiring policy for software developers (wannabe startup founders pay attention): Don't just try to hire brilliant coders -- look for people who work really well with a team. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] PyMem_*Calloc
Hi, 2014-04-14 1:39 GMT-04:00 Nathaniel Smith n...@pobox.com: The new tracemalloc infrastructure in python 3.4 is super-interesting to numerical folks, because we really like memory profiling. Cool, thanks :-) calloc() is more awesome than malloc()+memset() (...) I had a discussion with someone about tracemalloc and numpy at Pycon, was it you? After this discussion, I realized that calloc() exists because the operating system can have a very efficient implementation for calloc() (as you described). SO, we'd like to route our allocations through PyMem_* in order to let tracemalloc see them, but because there is no PyMem_*Calloc, doing this would force us to give up on the calloc() optimizations. It would also be useful in Python because in many places, Python uses memset() to fill memory with zeros. The obvious solution is to add a PyMem_*Calloc to the API. Would this be possible? Unfortunately it would require adding a new field to the PyMemAllocator struct, which would be an ABI/API break; PyMemAllocator is exposed directly in the C API and passed by value: https://docs.python.org/3.5/c-api/memory.html#c.PyMemAllocator I don't want to change the structure in Python 3.4, but I'm interested to implement the change in Python 3.5. Please open an issue and add me to the nosy list. For Python 3.4, you can maybe add a compilation flag to use Python allocators but reimplementing calloc() which will be slower as you explained. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] PyMem_*Calloc
On Tue, Apr 15, 2014 at 9:31 AM, Charles-François Natali cf.nat...@gmail.com wrote: Indeed, that's very reasonable. Please open an issue on the tracker! Done! http://bugs.python.org/issue21233 I'll ping numpy-discussion and see if I can convince someone to do the work ;-). -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
IIRC it is no longer the case that ZIP imports (involving only one file for a lot of modules) are much faster than regular FS imports? On Tue, Apr 15, 2014 at 10:34 AM, Eric Snow ericsnowcurren...@gmail.com wrote: On Tue, Apr 15, 2014 at 1:45 AM, Chris Angelico ros...@gmail.com wrote: Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; migrating hg to Py3 will (if I understand the above figures correctly) widen that gap, so any improvement done to startup performance will give a very real advantage. Perhaps not so much a very real advantage as less of a distraction. It's still significantly slower than 2.7. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Tue, Apr 15, 2014 at 11:19 AM, Daniel Holth dho...@gmail.com wrote: IIRC it is no longer the case that ZIP imports (involving only one file for a lot of modules) are much faster than regular FS imports? It's definitely minimized since Python 3.3 and the caching of stat results at the directory level for a small amount of time. -Brett On Tue, Apr 15, 2014 at 10:34 AM, Eric Snow ericsnowcurren...@gmail.com wrote: On Tue, Apr 15, 2014 at 1:45 AM, Chris Angelico ros...@gmail.com wrote: Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; migrating hg to Py3 will (if I understand the above figures correctly) widen that gap, so any improvement done to startup performance will give a very real advantage. Perhaps not so much a very real advantage as less of a distraction. It's still significantly slower than 2.7. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
On Wed, Apr 16, 2014 at 1:34 AM, Skip Montanaro s...@pobox.com wrote: I find it hard to believe that freezing the stdlib is going to lower the barrier enough for the Mercurial folks, if, in fact, import slowness is their main reason for not moving to 3.x. I've no idea whether that's the case or not. All I know is, every time I need to work with a Mercurial repo it feels a lot slower than doing similar work on a similar size git repo [1], so any improvement (or reduction of penalty) will be noticeable. Every time you type 'hg something', it has to do those imports, so startup time is significant. ChrisA [1] One time I actually did a conversion of CPython from hg into git, just so I could do some analysis and digging. Worked out considerably faster, although that's only because I left the import/conversion process to run by itself while I made a hot chocolate, which meant I was waiting less time. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/15/2014 11:34 AM, Skip Montanaro wrote: I find it hard to believe that freezing the stdlib is going to lower the barrier enough for the Mercurial folks, if, in fact, import slowness is their main reason for not moving to 3.x. My understanding of what Matt said at the language summit is that the need to support really old versions of Python 2.x (back to 2.4) is a big part of the holdup (straddling is *much* more painful without constraining to Python2 = 2.6). As I heard it, the real reason for the inertia is that the Python3 port is a lot of effort / pain for zero perceived gain outside of because it is the Right Thing(TM). After my porting experience, I can sympathize with that sensibility, and my stuff gets an advantage (frameworks / libraries marketing to Python3 devs) that Hg doesn't (most users don't really care which Python is used to drive the standalone tool). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlNNVHAACgkQ+gerLs4ltQ4lpwCeJTYvfBBlE3cS+eq+kA4/zEi3 R+8AnRy4HYLRZ4DHhHDop/8A86MJt5Ei =fORL -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
Eric wrote: Perhaps not so much a very real advantage as less of a distraction. It's still significantly slower than 2.7. :) I'm still confused. I peeked in /usr/bin/hg. The only system modules it imports directly are os and system (maybe I'm using an ancient version?). After that, it imports its own lazy import module. This suggests to me that Mercurial's import slowness is mostly in its own code (I counted 104 Python modules and six shared objects in its mercurial package, which isn't going to be affected (much?) by freezing the Python standard modules. I'm not trying to be difficult here. I thought that way back in the day a huge amount of work was done to remove needless filesystem activity, and zip packaging has been around for quite awhile. As an experiment, I ran hg pull as /usr/bin/python -v /usr/bin/hg pull in my cpython repo then looked at the -v output. Summarizing the output I saw the following: 30 imports (0 dlopens) Python banner printed 86 imports (18 dlopens) adding changesets message 5 imports (2 dlopens) adding manifests message 1 import (0 dlopens) adding file changes message 7 imports (3 dlopens) added ... changesets message 4 imports (0 dlopens) run 'hg update' message (I missed a searching message in there somewhere.) That's a total of 133 imports, 23 of which were satisfied by loading an extension module. The imports break down as follows: 51 imports (4 dlopens) from the mercurial package 5 imports from the hgext package 7 imports from the encodings package Everything else is imported from the top level, and at a glance appears to be all Python stdlib stuff. The key period of execution looks to be between the printing of the Python banner and the printing of the adding changesets message. I see 46 imports (2 dlopens) from the mercurial or hgext packages. That leaves 40 stdlib imports, of which 16 were satisfied by dlopen. As a final check, I saved all the stdlib import statements from the -v output (77 statements) to a file and timed its execution: % time /usr/bin/python ~/tmp/pyimp.py real0m0.162s user0m0.034s sys 0m0.010s It doesn't take much time to import every stdlib module that Mercurial needs. I don't know how much slower all this import machinery is in 3.x (and can't test at work, as we don't have a copy laying about). It would probably have to be 3x or more slower for it to have much visible impact on even simple hg commands. I find it hard to believe that freezing the stdlib is going to lower the barrier enough for the Mercurial folks, if, in fact, import slowness is their main reason for not moving to 3.x. Skip ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
On Tue, Apr 15, 2014 at 10:40 AM, Chris Angelico ros...@gmail.com wrote: I've no idea whether that's the case or not. All I know is, every time I need to work with a Mercurial repo it feels a lot slower than doing similar work on a similar size git repo [1], so any improvement (or reduction of penalty) will be noticeable. Based on what I saw, I really don't think that startup slowness is in imports of Python stdlib modules, which is all Brett was aiming at. I *can* believe overall import slowness might be a problem, but in that case, Brett's work isn't going to help the Mercurial folks much. Skip ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
On Tue, Apr 15, 2014 at 10:42 AM, Daniel Holth dho...@gmail.com wrote: I wish it was less than 50 milliseconds (0.05 seconds) including running hg, which is the common threshold for instant. Instant for me is the blink of an eye, which Wikipedia reports as typically between 100ms and 400ms. http://en.wikipedia.org/wiki/Blink If you blink, you've missed Python 2.7 startup on a relatively modern machine. wink Skip ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
On Wed, Apr 16, 2014 at 1:56 AM, Skip Montanaro s...@pobox.com wrote: On Tue, Apr 15, 2014 at 10:42 AM, Daniel Holth dho...@gmail.com wrote: I wish it was less than 50 milliseconds (0.05 seconds) including running hg, which is the common threshold for instant. Instant for me is the blink of an eye, which Wikipedia reports as typically between 100ms and 400ms. http://en.wikipedia.org/wiki/Blink If you blink, you've missed Python 2.7 startup on a relatively modern machine. 400ms feels glitchy. 100ms is the absolute maximum for immediate interaction. If I can sense a musical beat between doing something and seeing its result, it's not instant. And if there's a comedic beat between them, it's... laughably slow. *dives for cover* ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Appeal for reviews
On 14 Apr 2014 08:42, R. David Murray rdmur...@bitdance.com wrote: Or to put it another way, I'd like to encourage contributors who want to get commit access to focus just as much on doing good reviews as they do on writing new patches. Currently the focus is all on getting patches accepted. Huh, I hadn't thought of it that way before, but it's a very good point. AFAICS Python already does very well at getting people to do reviewing by comparison to most projects, though. And it's *not* all about getting patches accepted -- newer people seem to do a lot of work on PEPs and testing compared to most projects I've seen, and not just because Python-Dev insists on them before getting code approved. I've always really liked MvL's 5-reviews-to-get-1 approach. In the context of this thread it has a number of nice properties. First, it makes it explicit that cooperative work (even if it's expressed as out-and-out horse-trading, it's still working together) is central to python-dev. Second, it makes that work visible if people post their requests to either python-dev or core-mentorship.[1][2] Third, it emphasizes reviewing as a good thing and an important contribution. People tend to think of reviews as criticism, or even us-against- them. Again, the activity and the idea that it is a Good Thing is (or can be) visible to the contributors in general. The only thing I don't like about it[3] is that it puts an explicit price on core developer time (my time is worth 5x as much as yours). I fear that it may be a little off-putting. In this vein, I wonder if a slot for new comments on old issues in the tracker report might not be useful. (Yeah, I know, the tracker reporting function is free software. :) Maybe a most active issues report, too. This isn't to deprecate the 50% core developer idea at all -- it's great! I just don't know enough bosses in the field to know how to sell that one. Footnotes: [1] Bazaar had a patch pilot program where an experienced developer was detailed to clean the patch queue by shepherding newer developers through their rather detailed process. But it had two disadvantages: first, it really was all about getting patches accepted, and second, the actual piloting tended to be done off-list. [2] I know some people don't like core-mentorship because it's somewhat less visible than python-dev. Let's not discuss that now, it's just an example. [3] Well, actually, when wearing my economist hat I like it a lot. :-) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
I find Python's startup time to be very sluggish. I wish it was less than 50 milliseconds (0.05 seconds) including running hg, which is the common threshold for instant. On my machine 'python -c ' takes about 0.05 seconds but 'python3 -c ' takes 0.125 seconds. I will be very happy to see any speedup. On Tue, Apr 15, 2014 at 11:34 AM, Skip Montanaro s...@pobox.com wrote: Eric wrote: Perhaps not so much a very real advantage as less of a distraction. It's still significantly slower than 2.7. :) I'm still confused. I peeked in /usr/bin/hg. The only system modules it imports directly are os and system (maybe I'm using an ancient version?). After that, it imports its own lazy import module. This suggests to me that Mercurial's import slowness is mostly in its own code (I counted 104 Python modules and six shared objects in its mercurial package, which isn't going to be affected (much?) by freezing the Python standard modules. I'm not trying to be difficult here. I thought that way back in the day a huge amount of work was done to remove needless filesystem activity, and zip packaging has been around for quite awhile. As an experiment, I ran hg pull as /usr/bin/python -v /usr/bin/hg pull in my cpython repo then looked at the -v output. Summarizing the output I saw the following: 30 imports (0 dlopens) Python banner printed 86 imports (18 dlopens) adding changesets message 5 imports (2 dlopens) adding manifests message 1 import (0 dlopens) adding file changes message 7 imports (3 dlopens) added ... changesets message 4 imports (0 dlopens) run 'hg update' message (I missed a searching message in there somewhere.) That's a total of 133 imports, 23 of which were satisfied by loading an extension module. The imports break down as follows: 51 imports (4 dlopens) from the mercurial package 5 imports from the hgext package 7 imports from the encodings package Everything else is imported from the top level, and at a glance appears to be all Python stdlib stuff. The key period of execution looks to be between the printing of the Python banner and the printing of the adding changesets message. I see 46 imports (2 dlopens) from the mercurial or hgext packages. That leaves 40 stdlib imports, of which 16 were satisfied by dlopen. As a final check, I saved all the stdlib import statements from the -v output (77 statements) to a file and timed its execution: % time /usr/bin/python ~/tmp/pyimp.py real0m0.162s user0m0.034s sys 0m0.010s It doesn't take much time to import every stdlib module that Mercurial needs. I don't know how much slower all this import machinery is in 3.x (and can't test at work, as we don't have a copy laying about). It would probably have to be 3x or more slower for it to have much visible impact on even simple hg commands. I find it hard to believe that freezing the stdlib is going to lower the barrier enough for the Mercurial folks, if, in fact, import slowness is their main reason for not moving to 3.x. Skip ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
Le 15/04/2014 09:45, Chris Angelico a écrit : Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; Keep in mind those shoot-outs usually rely on large repositories and/or non-trivial operations, so startup time is not necessarily a significant contributor in Mercurial being slower (when it actually is slower than git, which may not be all the time). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
Le 15/04/2014 17:42, Daniel Holth a écrit : I find Python's startup time to be very sluggish. I wish it was less than 50 milliseconds (0.05 seconds) including running hg, which is the common threshold for instant. On my machine 'python -c ' takes about 0.05 seconds but 'python3 -c ' takes 0.125 seconds. Please quote exact versions. Different versions of Python 3 will have different startup characteristics. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
Le 14/04/2014 23:51, Brett Cannon a écrit : It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of performance gain it would get. Freezing everything except encodings.__init__, os, and _sysconfigdata, it speeds up startup by 11% compared to default. That sounds like a rather small number for the amount of complication and opacity it adds into the build and startup process. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Wed, Apr 16, 2014 at 2:40 AM, Antoine Pitrou solip...@pitrou.net wrote: Le 15/04/2014 09:45, Chris Angelico a écrit : Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; Keep in mind those shoot-outs usually rely on large repositories and/or non-trivial operations, so startup time is not necessarily a significant contributor in Mercurial being slower (when it actually is slower than git, which may not be all the time). I'm talking also about the feel of actual daily use, partly on big repos like git (git), CPython (hg), and Pike (git), and partly on some smaller ones. Whether it's startup cost or operational cost I don't know, but if I want it consistently fast, I generally go for git. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
Le 15/04/2014 17:34, Skip Montanaro a écrit : This suggests to me that Mercurial's import slowness is mostly in its own code (I counted 104 Python modules and six shared objects in its mercurial package, which isn't going to be affected (much?) by freezing the Python standard modules. Skip is right. When trying to find out why the hgprompt extension (which is a rather nifty extension adding color-coded repository information into your bash prompt) made the shell so much slower, it turned out that most of the execution time comes from importing *Mercurial* modules, not stdlib modules. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
Skip Montanaro writes: Instant for me is the blink of an eye, which Wikipedia reports as typically between 100ms and 400ms. http://en.wikipedia.org/wiki/Blink If you blink, you've missed Python 2.7 startup on a relatively modern machine. This is what I see for Mac OS X Mavericks on a 2.7GHz Core i7: # Apple's /usr/bin/python, Python 2.7.5, built with clang 5.0.0 $ time python -c # cold warm warm real 0.967s0.020s0.022s user 0.025s0.011s0.012s sys 0.061s0.009s0.007s # MacPorts /opt/local/bin/python3.3, Python 3.3.5, same compiler $ time python3.3 -c real 1.034s0.041s0.037s user 0.065s0.030s0.028s sys 0.036s0.008s0.007s Further iterations of warm cache starts remain (coincidentally, I would guess, but it's indicative) in the ranges above. I don't feel like rebooting or otherwise figuring out how to evict python from the cache to get a sense of variation for cold cache startup, but obviously it's more than an order of magnitude slower than warm start for both Pythons. Warm start numbers are well within Daniel's 50ms spec. Granted user time for Python 3.3 is 2.5-3X that of Python 2.7 warm or cold, it's still below human JND (if you can see it, it's probably a slow display ;-). So it's all about waiting on the OS, it seems to me. By comparison, git: $ time git --version #cold warm warm real 0.430s0.017s0.021s user 0.007s0.006s0.006s sys 0.009s0.006s0.006s OK, Python 2.7 is slower than git and Python 3.3 much slower. But I don't think this explains anybody's feeling that hg is sluggish compared to git -- git also shows perceptible delay on a cold start, I didn't notice it being faster (I wasn't thinking about it, though, and I wasn't in a hurry to see the version string). I suspect Linus has spiked everybody's Kool-Aid and it's a mass hallucination. More seriously, I wouldn't be surprised if git is just better optimized for certain operations that people expect to be fast. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
In case you were wondering, I'm using Ubuntu's 2.7.5+ and 3.3.2+. My feeling has long been that the speed of getting at the --help option or any initial user feedback from Mercurial or git is a big driver in perceived speed as opposed to how long the entire operation might take. But for me any initial speed improvements from git are fully offset by the feeling of irritation afterwards. /troll For me Python's startup time (warm) takes about 1/4 of the hg startup time in the worst case. I expect to both notice and appreciate any speedups and encourage all optimizers to optimize. On Tue, Apr 15, 2014 at 12:47 PM, Antoine Pitrou solip...@pitrou.net wrote: Le 15/04/2014 17:42, Daniel Holth a écrit : I find Python's startup time to be very sluggish. I wish it was less than 50 milliseconds (0.05 seconds) including running hg, which is the common threshold for instant. On my machine 'python -c ' takes about 0.05 seconds but 'python3 -c ' takes 0.125 seconds. Please quote exact versions. Different versions of Python 3 will have different startup characteristics. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
Le 15/04/2014 19:09, Daniel Holth a écrit : In case you were wondering, I'm using Ubuntu's 2.7.5+ and 3.3.2+. My feeling has long been that the speed of getting at the --help option or any initial user feedback from Mercurial or git is a big driver in perceived speed as opposed to how long the entire operation might take. But for me any initial speed improvements from git are fully offset by the feeling of irritation afterwards. /troll For me Python's startup time (warm) takes about 1/4 of the hg startup time in the worst case. I expect to both notice and appreciate any speedups and encourage all optimizers to optimize. Well, if we optimize 11% out of that 1/4, I don't expect you to notice the speedup at all ;-) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
On 15 April 2014 18:09, Daniel Holth dho...@gmail.com wrote: For me Python's startup time (warm) takes about 1/4 of the hg startup time in the worst case. I expect to both notice and appreciate any speedups and encourage all optimizers to optimize. My experience is essentially irrelevant (as a Windows user, the OS process creation time makes any of the numbers people are quoting for Unix little more than a pipe dream for me :-)) but it seems to me that the key measure of sluggishness tends to be on the tiny query operations (the things people put in their prompt). Nobody cares about process startup time on a hg clone of a 1GB repo across the network. But hg status to get details of the current repo to show in your prompt has to be very fast, or people complain it's slow [1]. On that point, I suspect that git's approach of having a plethora of tiny focused commands each of which does only what it needs to, probably makes for a better simple things are fast experience than a single-application architecture like hg. (On the other hand, it utterly kills git's performance on Windows, so you win some, you lose some). So improving Python startup time will probably help with Mercurial's *perceived* speed - but architecture improvements focusing on running enquiry type commands really fast may help as much if not more (I don't know how much they've optimised for that case in the past). On 15 April 2014 18:29, Antoine Pitrou solip...@pitrou.net wrote: Well, if we optimize 11% out of that 1/4, I don't expect you to notice the speedup at all ;-) Yeah, in reality no one thing is ever going to be *that* perceptible. But as they say, every little helps. Paul [1] Windows process startup times affect this *a lot*. My powershell prompt function directly reads the files in .hg/.git because running the actual commands is far too slow for a prompt function. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
On Tue, Apr 15, 2014 at 1:29 PM, Antoine Pitrou solip...@pitrou.net wrote: Le 15/04/2014 19:09, Daniel Holth a écrit : In case you were wondering, I'm using Ubuntu's 2.7.5+ and 3.3.2+. My feeling has long been that the speed of getting at the --help option or any initial user feedback from Mercurial or git is a big driver in perceived speed as opposed to how long the entire operation might take. But for me any initial speed improvements from git are fully offset by the feeling of irritation afterwards. /troll For me Python's startup time (warm) takes about 1/4 of the hg startup time in the worst case. I expect to both notice and appreciate any speedups and encourage all optimizers to optimize. Well, if we optimize 11% out of that 1/4, I don't expect you to notice the speedup at all ;-) Regards Antoine. No one expects the Spanish Inquisition. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
On 15/04/2014 18:32, Daniel Holth wrote: On Tue, Apr 15, 2014 at 1:29 PM, Antoine Pitrou solip...@pitrou.net wrote: Le 15/04/2014 19:09, Daniel Holth a écrit : In case you were wondering, I'm using Ubuntu's 2.7.5+ and 3.3.2+. My feeling has long been that the speed of getting at the --help option or any initial user feedback from Mercurial or git is a big driver in perceived speed as opposed to how long the entire operation might take. But for me any initial speed improvements from git are fully offset by the feeling of irritation afterwards. /troll For me Python's startup time (warm) takes about 1/4 of the hg startup time in the worst case. I expect to both notice and appreciate any speedups and encourage all optimizers to optimize. Well, if we optimize 11% out of that 1/4, I don't expect you to notice the speedup at all ;-) Regards Antoine. No one expects the Spanish Inquisition. Except those who expect Python 2.8? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Appeal for reviews
On 4/15/2014 12:15 PM, Stephen J. Turnbull wrote: I've always really liked MvL's 5-reviews-to-get-1 approach. The only thing I don't like about it[3] is that it puts an explicit price on core developer time (my time is worth 5x as much as yours). Not really true since any of the 5 could be one or two sentence comments while his 1 might include committing, or a second review after revision. Besides which, a person accepting the offer got to choose both the 5 to review and the 1 to be reviewed. Choice has value. I fear that it may be a little off-putting. Perhaps, but it is also attractive enough that there have been several requests for someone to renew the offer, and no core developer currently willing to make the offer (because giving up choice is a cost). -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
Can we please stop the argument about Hg vs. Git? On Tue, Apr 15, 2014 at 12:54 PM, Chris Angelico ros...@gmail.com wrote: On Wed, Apr 16, 2014 at 2:40 AM, Antoine Pitrou solip...@pitrou.net wrote: Le 15/04/2014 09:45, Chris Angelico a écrit : Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; Keep in mind those shoot-outs usually rely on large repositories and/or non-trivial operations, so startup time is not necessarily a significant contributor in Mercurial being slower (when it actually is slower than git, which may not be all the time). I'm talking also about the feel of actual daily use, partly on big repos like git (git), CPython (hg), and Pike (git), and partly on some smaller ones. Whether it's startup cost or operational cost I don't know, but if I want it consistently fast, I generally go for git. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Wed, Apr 16, 2014 at 4:54 AM, Guido van Rossum gu...@python.org wrote: Can we please stop the argument about Hg vs. Git? My apologies. All I was saying was that hg is a use case where startup performance really does matter, as opposed to the ones presented earlier in the thread where a process stays in memory longer. It wasn't meant to devolve like that. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Appeal for reviews
Terry Reedy writes: On 4/15/2014 12:15 PM, Stephen J. Turnbull wrote: The only thing I don't like about it[3] is that it puts an explicit price on core developer time (my time is worth 5x as much as yours). Not really true But that is *not* your call! It's for the would-be contributor to decide, because they're the one to decide whether to take the offer. I fear that it may be a little off-putting. Perhaps, but it is also attractive enough that there have been several requests for someone to renew the offer, and no core developer currently willing to make the offer (because giving up choice is a cost). Sure. That doesn't mean there wouldn't be more people to take up the offer if it didn't implicitly deprecate the value of their time. The issue here is not the reality of value of choice, it's whether would-be developers *perceive* the price negatively or not. If they don't, or can easily be convinced that it makes sense, great! Unfortunately the point is moot at the moment. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Timing breakdown of Py_InitializeEx_Private()
To finish my timing work I decided to see where Py_InitializeEx_Private() spends its time. The following is a breakdown measured in microseconds running using -E: INIT: setlocale: 11 envvar: 2 random init: 2 interp creation: 15 thread creation: 6 GIL: 10 _Py_ReadyTypes(): 930 more types: 44 builtins: 141 exceptions: 287 sys: 258 _PyImport_Init: 15 import hooks: 4 _PyWarnings_Init(): 15 ENTERING import_init(): PyImport_ImportFrozenModule(_frozen_importlib): 1186 interp-importlib: 6 PyInit_imp(): 15 _imp: 3 importlib._install(): 876 _PyImportZip_Init(): 130 _PyFaulthandler_Init(): 13 time: 3 ENTERING initfsencoding(): codec lookup: 2179 signals: 120 tracemalloc: 7 __main__: 13 initstdio(): 1568 warnings module: 4 initsite(): 9552 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()
Are you going to post your code (or a link to it)? On Tue, Apr 15, 2014 at 2:26 PM, Brett Cannon bcan...@gmail.com wrote: To finish my timing work I decided to see where Py_InitializeEx_Private() spends its time. The following is a breakdown measured in microseconds running using -E: INIT: setlocale: 11 envvar: 2 random init: 2 interp creation: 15 thread creation: 6 GIL: 10 _Py_ReadyTypes(): 930 more types: 44 builtins: 141 exceptions: 287 sys: 258 _PyImport_Init: 15 import hooks: 4 _PyWarnings_Init(): 15 ENTERING import_init(): PyImport_ImportFrozenModule(_frozen_importlib): 1186 interp-importlib: 6 PyInit_imp(): 15 _imp: 3 importlib._install(): 876 _PyImportZip_Init(): 130 _PyFaulthandler_Init(): 13 time: 3 ENTERING initfsencoding(): codec lookup: 2179 signals: 120 tracemalloc: 7 __main__: 13 initstdio(): 1568 warnings module: 4 initsite(): 9552 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()
On Tue, Apr 15, 2014 at 5:40 PM, Guido van Rossum gu...@python.org wrote: Are you going to post your code (or a link to it)? I basically wrote a function that uses gettimeofday() and just calculates the delta between the calls. Greg Smith verified that I wasn't doing anything stupid. =) Anyway, the diff can be found at https://gist.github.com/brettcannon/9cd3960dd067bb7a45bd . -Brett On Tue, Apr 15, 2014 at 2:26 PM, Brett Cannon bcan...@gmail.com wrote: To finish my timing work I decided to see where Py_InitializeEx_Private() spends its time. The following is a breakdown measured in microseconds running using -E: INIT: setlocale: 11 envvar: 2 random init: 2 interp creation: 15 thread creation: 6 GIL: 10 _Py_ReadyTypes(): 930 more types: 44 builtins: 141 exceptions: 287 sys: 258 _PyImport_Init: 15 import hooks: 4 _PyWarnings_Init(): 15 ENTERING import_init(): PyImport_ImportFrozenModule(_frozen_importlib): 1186 interp-importlib: 6 PyInit_imp(): 15 _imp: 3 importlib._install(): 876 _PyImportZip_Init(): 130 _PyFaulthandler_Init(): 13 time: 3 ENTERING initfsencoding(): codec lookup: 2179 signals: 120 tracemalloc: 7 __main__: 13 initstdio(): 1568 warnings module: 4 initsite(): 9552 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2migr8
Apologies for the slow reply, I was travelling back from PyCon. From Guido: - I'd prefer a name that plays on 2 and 3, not 2 and 8. :-) I didn't really expect the name to stick, but Nick and I had such a good laugh that it would have been unfair not to share it :-) (though I laughed even more at 'migr2**3') Really, I'd expect the name to be pretty boring - Python Straddle, Python 2 Strict or Python Migration Edition (if anyone can live with Python ME ;-) ), since the aim is to discourage general use. Users can (and should) keep using Python 2.7 while the development team is using the stricter version - or however this looks for the organisations that want to use it. - Are you sure this isn't better directed to python-ideas first? Most ideas have to prove their worth in that list before python-dev will give them the light of day. It was a fifty-fifty choice between here and there. Python-dev had the more recent discussion about this issue and it was also raised at the language summit. - When it comes to purely syntactic issues (e.g. except x, y:) a linter or some other separate tool can handle this well (heck, you can build it into an import hook probably :-). True, but that isn't quite the same forcing function that interpreter errors would give. Code reviews are great until all of your potential contributors start with commit rights (and you are stuck on a VCS that has no concept of commit hooks... yes, there is also a tooling issue for the people who want this, but there also isn't a blessed tool for this purpose). Eric Snow suggested that this entire idea could probably be handled as an import hook, which may mean that the whole thing could just be a PyPI package that works with 2.7. If so, great! But it still needs the blessing of python-dev, and probably at least one contributor who is also a core developer. - When it's about backported modules, a sumo distribution is probably the way to go; when it's about renamed stdlib modules, six (perhaps an extended version) should cover you. Agreed. The advantage of the sumo distribution is that not every enterprise finds it easy to add more dependencies (I think this discussion happened on distutils-sig) - when it's just part of Python and Python is already approved by our lawyers/IT guys/whoever things are significantly easier. - Regarding warning about the changed dict API, I wonder how you plan to implement that if you allow passing dict object back and forth between code that has opted in to single-source and code that hasn't yet. Please think through some specific examples before responding. I haven't thought through specific examples, so instead I'll describe my vision for how this solves the issue (it finally dawned on my during a jet-lagged dream - let's see if it still makes sense :-) ): Migrating from 2.7 to 3.3[1] causes a lot of pain - let's call it 100% pain. You have to do 100% of the migration work before it will work. Phase 1: Migrating from 2.7 to the straddle-edition causes a small amount of unconditional warnings and performance hit (these would be for things considered poor form anyway, since they are errors in 3.3), but no behaviour changes. There is some pain in fixing these, let's say 10%, but it isn't pain-pain because your program still runs. You can defer this 10% pain entirely into Phase 2 if you want. Phase 2: Migrating from straddle-edition files without the flag to files with the flag is 70-80% of pain, distributed over a long period of time, during which your program still runs. Phase 3: Migrating from straddle-edition where every file is flagged to 3.3 covers the last 10-20% of pain. This pain is real, since your program will not run against 3.3 until it is fixed, but it is significantly smaller than going directly from 2.7. The last phase includes the final pass of should this string literal be bytes? work, but even that could be partially moved into Phase 2 by failing comparisons between byte literals and strings (apologies for the simplistic idea that is probably close to impossible to implement - the fact that nobody has a good solution to this is really the core of the entire issue...). Basically, this idea allows the migration pain to be distributed over a longer period of time. (1: I keep using 3.3 as my target because 3.4 has no new syntax over 3.3 - the new features in 3.4 are basically irrelevant to porting. The target version is also basically irrelevant to this discussion.) - But the biggest issue is of course bytes vs. text. You would have to first do a careful analysis of the *whole* problem before you can even think about proposing a solution. Too many people think there is an easy solution for this; but almost everybody is focused on only part of the problem (the part that causes them immediate pain) without realizing that other people's pain may be different. Agreed. Maybe there is a need for different levels of
Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()
On 4/15/2014 5:26 PM, Brett Cannon wrote: To finish my timing work I decided to see where Py_InitializeEx_Private() spends its time. The following is a breakdown measured in microseconds running using -E: INIT: setlocale: 11 envvar: 2 random init: 2 interp creation: 15 thread creation: 6 GIL: 10 _Py_ReadyTypes(): 930 more types: 44 builtins: 141 exceptions: 287 sys: 258 _PyImport_Init: 15 import hooks: 4 _PyWarnings_Init(): 15 ENTERING import_init(): PyImport_ImportFrozenModule(_frozen_importlib): 1186 interp-importlib: 6 PyInit_imp(): 15 _imp: 3 importlib._install(): 876 _PyImportZip_Init(): 130 _PyFaulthandler_Init(): 13 time: 3 ENTERING initfsencoding(): codec lookup: 2179 signals: 120 tracemalloc: 7 __main__: 13 initstdio(): 1568 warnings module: 4 initsite(): 9552 It looks like initsite takes half the time. Can it be sped up? -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2migr8
On Apr 15, 2014, at 10:55 PM, Steve Dower wrote: Really, I'd expect the name to be pretty boring - Python Straddle, Python 2 Strict or Python Migration Edition (if anyone can live with Python ME ;-) ) We often call code that works in both Python 2 and 3 from a single source bi-lingual. Maybe we've just been mispronouncing py-lingual :). -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2migr8
On 15 Apr 2014 18:56, Steve Dower steve.do...@microsoft.com wrote: From Eric: This should be doable with an import hook. For that matter it would be pretty straight-forward to provide a script around the Python binary that installs the import hook. All this could be bundled up and distributed through the cheeseshop. I love this solution. If it really is doable this way then it is even better than having a fork. My name suggestion for a PyPI packages would be PyStraddle. Eric and I spent some time discussing this today, and don't see any insurmountable barriers to doing it as an import hook. It may mean doing a 2.7 compatible full importlib backport, but that's also useful for other reasons. We may also want to consider/discuss integrating this idea with the python-future project rather than making it a new independent project, as that provides ready access to Python 3 style builtins that a hook could inject as globals in flagged files. Regardless, I think we're now drifting off topic for python-dev - Eric and I are still at the sprints, so we'll take a look at the feasibility of proposing this to Ed and the other python-future devs (I also have some ideas around using sitecustomize to put an entire Python installation into enforcement mode for flagged files). Any other related suggestions should probably be directed to python-ideas at this point. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()
Well, that's the part that does import site. Anything that speeds up the code in Lib/site.py might help. :-) On Tue, Apr 15, 2014 at 5:23 PM, Terry Reedy tjre...@udel.edu wrote: On 4/15/2014 5:26 PM, Brett Cannon wrote: To finish my timing work I decided to see where Py_InitializeEx_Private() spends its time. The following is a breakdown measured in microseconds running using -E: INIT: setlocale: 11 envvar: 2 random init: 2 interp creation: 15 thread creation: 6 GIL: 10 _Py_ReadyTypes(): 930 more types: 44 builtins: 141 exceptions: 287 sys: 258 _PyImport_Init: 15 import hooks: 4 _PyWarnings_Init(): 15 ENTERING import_init(): PyImport_ImportFrozenModule(_frozen_importlib): 1186 interp-importlib: 6 PyInit_imp(): 15 _imp: 3 importlib._install(): 876 _PyImportZip_Init(): 130 _PyFaulthandler_Init(): 13 time: 3 ENTERING initfsencoding(): codec lookup: 2179 signals: 120 tracemalloc: 7 __main__: 13 initstdio(): 1568 warnings module: 4 initsite(): 9552 It looks like initsite takes half the time. Can it be sped up? -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
Brett Cannon, 14.04.2014 23:51: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of performance gain it would get. Freezing everything except encodings.__init__, os, and _sysconfigdata, it speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14% from the slowdown (27% slower vs. 41% slower). The full results are at the end of the email. Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. My guess is the best approach would be adding a Lib/_frozen directory where any modules that we treat like this would be kept to act as a reminder that you need to rebuild for them (I would probably move importlib/_boostrap.py as well to make this consistent). Thoughts? Alternatively, the modules could be compiled with Cython. That would not only speed up the loading time but also the initialisation time and runtime. And since you'd keep the original .py files next to the .so files, you'd still get proper tracebacks. Compiling the modules natively would also enable linking them right into the interpreter core, BTW. But that would substantially increase its size. Maybe some of them could still be worth being linked in. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
