Re: [Python-Dev] Computed Goto dispatch for Python 2
On 1 June 2015 at 00:44, Terry Reedy tjre...@udel.edu wrote: On 5/31/2015 6:59 AM, Alexander Walters wrote: A better course of action would be to deprecate the non-portable version. Other than setting the PATH envvar, why do we need to continue even touching the system on install? It is highly annoying for those of us that maintain several installs of python on a single windows system, and it really should stop. Some people want the right-click context menu entries -- Run (also double click) and Edit with Idle, which should be Edit with Idle x.y. And system administrators responsible for deploying and maintaining Standard Operating Environments want the MSI integration. In that regard, the default behaviour of the python.org installers is the rough equivalent of the system Python on Linux distributions (with the added complexity of needing to deal with the Windows registry). Portable installations are often good for developers, but they come at the cost of failing to integrate properly with the underlying operating system. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
We are calling it 'embedable', but the rest of the world would call it 'portable', as in, runable from a usb stick I called it embeddable because it's not intended for direct use and is not complete. There's no test suite, no documentation, no tkinter (pending high demand), no pip, no site-packages, and no folder structure. It really is meant to be a component in another application that provides the rest of the layout for its own needs. (I probably ought to blog about it so there's at least one detailed example of what it's for...) A nice side-effect is that you can make a regular per-user install portable by adding a pyvenv.cfg with applocal = True, which disables regular path resolution (and also ignores PYTHONPATH, which is a feature or a bug, depending on your point of view). This only works on Windows right now, but could probably be ported from getpathp.c into getpath.c easily. Cheers, Steve Top-posted from my Windows Phone From: Alexander Waltersmailto:tritium-l...@sdamon.com Sent: 5/31/2015 6:39 To: python-dev@python.orgmailto:python-dev@python.org Subject: Re: [Python-Dev] Computed Goto dispatch for Python 2 A better course of action would be to deprecate the non-portable version. Other than setting the PATH envvar, why do we need to continue even touching the system on install? It is highly annoying for those of us that maintain several installs of python on a single windows system, and it really should stop. The only use I can think of for ever touching the registry in the first place is to tell distutils installers where python is. I can tell you right now, that design choice is a bug. There are some mighty hacks you have to go through to correct that behavior when you happen to be using a virtualenv. (We are calling it 'embedable', but the rest of the world would call it 'portable', as in, runable from a usb stick) On 5/31/2015 06:47, Paul Moore wrote: On 31 May 2015 at 11:41, Paul Moore p.f.mo...@gmail.com wrote: On 31 May 2015 at 10:14, Xavier Combelle xavier.combe...@gmail.com wrote: +1. The new embeddable Python distribution for Windows is a great step forward for this. It's not single-file, but it's easy to produce a single-directory self-contained application with it. I don't know if there's anything equivalent for Linux/OSX - maybe it's something we should look at for them as well (although the whole static binaries concept seems to be fairly frowned on in the Unix world, from what I've seen). Just curious What is the new embeddable Python distribution for Windows ? Python 3.5 ships a zipfile which contains a self-contained Python installation, intended for embedding. The idea is that you unzip it into your application directory, and use it from within your application (either via the embedding API, or using the included python.exe/pythonw.exe). It doesn't use the registry, or any global resources, so it's independent of any installed python that might be present. By the way, IMO the new embeddable distribution is a pretty big deal on Windows. To make sure that it doesn't end up unnoticed, can I suggest we include a prominent What's New entry for it, and a section in Python Setup and Usage under Using Python on Windows for it? I'd hate to find that 3 or 4 versions from now, we're still trying to remind people that they can use the embeddable distribution, in the same way that executable zipfiles ended up an almost unknown feature for ages. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium-list%40sdamon.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
+1. The new embeddable Python distribution for Windows is a great step forward for this. It's not single-file, but it's easy to produce a single-directory self-contained application with it. I don't know if there's anything equivalent for Linux/OSX - maybe it's something we should look at for them as well (although the whole static binaries concept seems to be fairly frowned on in the Unix world, from what I've seen). Just curious What is the new embeddable Python distribution for Windows ? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 31 May 2015 at 10:14, Xavier Combelle xavier.combe...@gmail.com wrote: +1. The new embeddable Python distribution for Windows is a great step forward for this. It's not single-file, but it's easy to produce a single-directory self-contained application with it. I don't know if there's anything equivalent for Linux/OSX - maybe it's something we should look at for them as well (although the whole static binaries concept seems to be fairly frowned on in the Unix world, from what I've seen). Just curious What is the new embeddable Python distribution for Windows ? Python 3.5 ships a zipfile which contains a self-contained Python installation, intended for embedding. The idea is that you unzip it into your application directory, and use it from within your application (either via the embedding API, or using the included python.exe/pythonw.exe). It doesn't use the registry, or any global resources, so it's independent of any installed python that might be present. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 31 May 2015 at 11:41, Paul Moore p.f.mo...@gmail.com wrote: On 31 May 2015 at 10:14, Xavier Combelle xavier.combe...@gmail.com wrote: +1. The new embeddable Python distribution for Windows is a great step forward for this. It's not single-file, but it's easy to produce a single-directory self-contained application with it. I don't know if there's anything equivalent for Linux/OSX - maybe it's something we should look at for them as well (although the whole static binaries concept seems to be fairly frowned on in the Unix world, from what I've seen). Just curious What is the new embeddable Python distribution for Windows ? Python 3.5 ships a zipfile which contains a self-contained Python installation, intended for embedding. The idea is that you unzip it into your application directory, and use it from within your application (either via the embedding API, or using the included python.exe/pythonw.exe). It doesn't use the registry, or any global resources, so it's independent of any installed python that might be present. By the way, IMO the new embeddable distribution is a pretty big deal on Windows. To make sure that it doesn't end up unnoticed, can I suggest we include a prominent What's New entry for it, and a section in Python Setup and Usage under Using Python on Windows for it? I'd hate to find that 3 or 4 versions from now, we're still trying to remind people that they can use the embeddable distribution, in the same way that executable zipfiles ended up an almost unknown feature for ages. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
A better course of action would be to deprecate the non-portable version. Other than setting the PATH envvar, why do we need to continue even touching the system on install? It is highly annoying for those of us that maintain several installs of python on a single windows system, and it really should stop. The only use I can think of for ever touching the registry in the first place is to tell distutils installers where python is. I can tell you right now, that design choice is a bug. There are some mighty hacks you have to go through to correct that behavior when you happen to be using a virtualenv. (We are calling it 'embedable', but the rest of the world would call it 'portable', as in, runable from a usb stick) On 5/31/2015 06:47, Paul Moore wrote: On 31 May 2015 at 11:41, Paul Moore p.f.mo...@gmail.com wrote: On 31 May 2015 at 10:14, Xavier Combelle xavier.combe...@gmail.com wrote: +1. The new embeddable Python distribution for Windows is a great step forward for this. It's not single-file, but it's easy to produce a single-directory self-contained application with it. I don't know if there's anything equivalent for Linux/OSX - maybe it's something we should look at for them as well (although the whole static binaries concept seems to be fairly frowned on in the Unix world, from what I've seen). Just curious What is the new embeddable Python distribution for Windows ? Python 3.5 ships a zipfile which contains a self-contained Python installation, intended for embedding. The idea is that you unzip it into your application directory, and use it from within your application (either via the embedding API, or using the included python.exe/pythonw.exe). It doesn't use the registry, or any global resources, so it's independent of any installed python that might be present. By the way, IMO the new embeddable distribution is a pretty big deal on Windows. To make sure that it doesn't end up unnoticed, can I suggest we include a prominent What's New entry for it, and a section in Python Setup and Usage under Using Python on Windows for it? I'd hate to find that 3 or 4 versions from now, we're still trying to remind people that they can use the embeddable distribution, in the same way that executable zipfiles ended up an almost unknown feature for ages. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium-list%40sdamon.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 5/31/2015 6:59 AM, Alexander Walters wrote: A better course of action would be to deprecate the non-portable version. Other than setting the PATH envvar, why do we need to continue even touching the system on install? It is highly annoying for those of us that maintain several installs of python on a single windows system, and it really should stop. Some people want the right-click context menu entries -- Run (also double click) and Edit with Idle, which should be Edit with Idle x.y. The only use I can think of for ever touching the registry in the first place is to tell distutils installers where python is. I can tell you right now, that design choice is a bug. There are some mighty hacks you have to go through to correct that behavior when you happen to be using a virtualenv. (We are calling it 'embedable', but the rest of the world would call it 'portable', as in, runable from a usb stick) -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 29 May 2015 11:01 am, Victor Stinner victor.stin...@gmail.com wrote: Why not continue to enhance Python 3 instead of wasting our time with Python 2? We have limited resources in term of developers to maintain Python. (I'm not talking about fixing *bugs* in Python 2 which is fine with me.) I'm actually OK with volunteers deciding that even fixing bugs in 2.7 isn't inherently rewarding enough for them to be willing to do it for free on their own time. Stepping up to extrinsically reward activities that are beneficial for customers but aren't intrinsically interesting enough for people to be willing to do for free is one of the key reasons commercial open source redistributors get paid. That more explicitly commercial presence is a dynamic we haven't historically had to deal with in core development, so there are going to be some growing pains as we find an arrangement that everyone is comfortable with (or is at least willing to tolerate, but I'm optimistic we can do better than that). Cheers, Nick. -- By the way, I just wrote sixer, a new tool to generate patches to port OpenStack to Python 3 :-) https://pypi.python.org/pypi/sixer It's based on regex, so it's less reliable than 2to3, 2to6 or modernize, but it's just enough for my specific use case. On OpenStack, it's not possible to send one giant patch hello, this is python 3. Code is modified by small and incremental changes. Come on in the Python 3 world and... always look on the bright side of life ( https://www.youtube.com/watch?v=VOAtCOsNuVM )! Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
Op 28 mei 2015 om 21:37 heeft Chris Barker chris.bar...@noaa.gov het volgende geschreven: On Thu, May 28, 2015 at 12:25 PM, Sturla Molden sturla.mol...@gmail.com wrote: The system Python should be left alone as it is. absolutely! By the way, py2app will build an application bundle that depends on the system python, indeed, that's all it will do if you run it with the system python, as Apple has added some non-redistributable bits in there. That's not quite the reason. It's more that I don't want to guess whether or not it is valid to bundle binaries from a system location. Furthermore bundling files from a base install of the OS is pretty useless, especially when those binaries won't run on earlier releases anyway due to the compilation options used. Ronald___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 29 May 2015 00:52, Paul Moore p.f.mo...@gmail.com wrote: +1. The new embeddable Python distribution for Windows is a great step forward for this. It's not single-file, but it's easy to produce a single-directory self-contained application with it. I don't know if there's anything equivalent for Linux/OSX - maybe it's something we should look at for them as well (although the whole static binaries concept seems to be fairly frowned on in the Unix world, from what I've seen). Correct - in the absence of the capacity to rebuild and redeploy the world at the click of a button, widespread deployment of static binaries poses an appallingly high security risk. It isn't an accident that Linux container orchestration is co-evolving with Linux container formats. Those efforts are mostly focused on network services GUI applications, though. For portable console applications, Go is still one of the nicest options currently available, as the relatively limited ability to interoperate with the system provided C/C++ libraries makes it much harder to create dependencies between the app and the platform. It's similar to Java in that respect, but without the dependency on a large language runtime like the JVM. In that vein, it might be interesting to see what could be done with MicroPython in terms of providing a lightweight portable Python runtime without CPython's extensive integration with the underlying OS. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 28/05/15 21:37, Chris Barker wrote: I think it's great for it to be used by end users as a system library / utility. i.e. like you would a the system libc -- so if you can write a little python script that only uses the stdlib -- you can simply deliver that script. No it is not, because someone will be 'clever' and try to upgrade it with pip or install packages into it. Sturla ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On Thu, May 28, 2015 at 12:25 PM, Sturla Molden sturla.mol...@gmail.com wrote: Many Unix tools need Python, so Mac OS X (like Linux distros and FreeBSD) will always need a system Python. Yes, it would be great if could be called spython or something else than python. But the main problem is that it is used by end-users as well, not just the operating system. I think it's great for it to be used by end users as a system library / utility. i.e. like you would a the system libc -- so if you can write a little python script that only uses the stdlib -- you can simply deliver that script. But if you want to go an install a bunch of extra non-standard packages (or, for heaven's sake, want a version with bug fixes!), they you really are better off installing a new python you can control. The system Python should be left alone as it is. absolutely! By the way, py2app will build an application bundle that depends on the system python, indeed, that's all it will do if you run it with the system python, as Apple has added some non-redistributable bits in there. But things get kin dof confusing if you want to rely on non-system packages... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On Thu, May 28, 2015 at 9:08 AM Guido van Rossum gu...@python.org wrote: Wow. Such thread. :-) This patch could save companies like Dropbox a lot of money. We run a ton of Python code in large datacenters, and while we are slow in moving to Python 3, we're good at updating to the latest 2.7. Dropbox should be compiling its own interpreter with whatever patches it deems appropriate. The people it'll save resources for are companies not enlightened enough to do that: thousands of them, generally small or non-tech focused :) The patch is forward and backward compatible.I'm strongly in favor. +1 I'm in favor as well. I mostly wanted to make sure that people were aware of profile-opt builds and that it was being compared. Sounds like both benefit, even used together. Win win. This is a 100% API compatible change. It just rearranges the interpreter loop on compilers enlightened enough to allow it. I was always bummed that it didn't make it into 2.7 itself. But given the world+dog is going to have 2.7 around and kicking for a long time, lets save the world some CPU cycles (read: carbon) for little effort. Very practical. Good for the world. People who need to save orders of magnitude more cycles shouldn't use an interpreter. ie: PyPy. Or consider the costs of moving to a compiled language. -gps ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 5/28/2015 10:55 AM, Steve Dower wrote: And it would look like a 20MB+ file just for a simple 1KB Python script... For Windows at least, I'd prefer to have some app-style installer generation (e.g. http://pynsist.readthedocs.org/en/latest/) which, combined with the embeddable Python distro (new for 3.5.0b1 in case anyone missed it), can simply extract everything into an install directory and run it from there. None of the items on the list above are needed for or would help with this. What I wish, of course, is that Windows just came with Python3, the way that DOS came with BASIC, so people could publish and trade Python programs the way we once did with BASIC programs. Then a simple 1KB Python script would just take an extra 1KB on disk. To me, the removal of a simple, builtin programming language for everyone was the biggest Windows mistake. Failing that, maybe PSF edu-sig could somehow encourage universities that requires students to have a computer to recommend or even require that Python be loaded so students could at least assume that others students have Python already loaded. Python is the BASIC of the 21 century. Somehow, trying to make it easier to have 50 duplicate copies of Python on a system seems the wrong direction to go. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
Donald Stufft don...@stufft.io wrote: Honestly, I’m on an OS that *does* ship Python (OS X) and part of me hopes that they stop shipping it. It’s very rare that someone ships Python as part of their OS without modifying it in some way, and those modifications almost always cause pain to some set of users (and since I work on pip, they tend to come to us with the weirdo problems). Case in point: Python on OS X adds some preinstalled software, but they put this pre-installed software before site-packages in sys.path, so pip can’t upgrade those pre-installed software packages at all. Many Unix tools need Python, so Mac OS X (like Linux distros and FreeBSD) will always need a system Python. Yes, it would be great if could be called spython or something else than python. But the main problem is that it is used by end-users as well, not just the operating system. Anyone who use Python on OSX should install their own Python. The system Python should be left alone as it is. If the system Python needs updating, it is the responsibility of Apple to distribute the upgrade. Nobody should attempt to use pip to update the system Python. Who knows what side-effects it might have. Preferably pip should have a check for it and bluntly refuse to do it. Sturla ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
Sorry for missing Julian's question. The GCC version used for the benchmarks is 4.8.2 Will look into the discussion at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=39284 and will investigate it. Julian Taylor jtaylor.debian at googlemail.com Thu May 28 13:30:59 CEST 2015 won't this need python compiled with gcc 5.1 to have any effect? Which compiler version was used for the benchmark? the issue that negated most computed goto improvements (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=39284) was only closed very recently (r212172, 9f4ec746affbde1) -Original Message- From: Matthias Klose [mailto:d...@ubuntu.com] Sent: Thursday, May 28, 2015 5:01 AM To: Parasa, Srinivas Vamsi; 'python-dev@python.org' Subject: Re: [Python-Dev] Computed Goto dispatch for Python 2 On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote: Hi All, This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation. Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email). Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP CPU with 18 cores, hyper-threading off, turbo off). Below is a summary of the performance we saw on the grand unified python benchmarks suite (available at https://hg.python.org/benchmarks/). We made 3 rigorous runs of the following benchmarks. In each rigorous run, a benchmark is run 100 times with and without the computed goto patch. Below we show the average performance boost for the 3 rigorous runs. Python 2.7.10 (original) vs Computed Goto performance Benchmark -1 As Gregory pointed out, there are other options to build the interpreter, and we are missing data how these compare with your patch. I assume, you tested with the Intel compiler, so it would be good to see results for other compilers as well (GCC, clang). Please could you provide the data for LTO and profile guided optimized builds (maybe combined too)? I'm happy to work with you on setting up these builds, but currently don't have the machine resources to do so myself. If the benefits show up for these configurations too, then I'm +/-0 on this patch. Matthias ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On May 28, 2015 at 2:11:02 PM, Terry Reedy (tjre...@udel.edu) wrote: On 5/28/2015 10:55 AM, Steve Dower wrote: And it would look like a 20MB+ file just for a simple 1KB Python script... For Windows at least, I'd prefer to have some app-style installer generation (e.g. http://pynsist.readthedocs.org/en/latest/) which, combined with the embeddable Python distro (new for 3.5.0b1 in case anyone missed it), can simply extract everything into an install directory and run it from there. None of the items on the list above are needed for or would help with this. What I wish, of course, is that Windows just came with Python3, the way that DOS came with BASIC, so people could publish and trade Python programs the way we once did with BASIC programs. Then a simple 1KB Python script would just take an extra 1KB on disk. To me, the removal of a simple, builtin programming language for everyone was the biggest Windows mistake. Failing that, maybe PSF edu-sig could somehow encourage universities that requires students to have a computer to recommend or even require that Python be loaded so students could at least assume that others students have Python already loaded. Python is the BASIC of the 21 century. Somehow, trying to make it easier to have 50 duplicate copies of Python on a system seems the wrong direction to go. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Honestly, I’m on an OS that *does* ship Python (OS X) and part of me hopes that they stop shipping it. It’s very rare that someone ships Python as part of their OS without modifying it in some way, and those modifications almost always cause pain to some set of users (and since I work on pip, they tend to come to us with the weirdo problems). Case in point: Python on OS X adds some preinstalled software, but they put this pre-installed software before site-packages in sys.path, so pip can’t upgrade those pre-installed software packages at all. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
Hello, On Fri, 29 May 2015 08:38:44 +1000 Nick Coghlan ncogh...@gmail.com wrote: [] In that vein, it might be interesting to see what could be done with MicroPython in terms of providing a lightweight portable Python runtime without CPython's extensive integration with the underlying OS. Thanks for mentioning that. That's definitely what I have on my mind, actually I wanted to do long ago a Lua-esque hack of being able to cat together an interpreter and a script, so resulting executable would just run the script. What stopped me is that it would be Lua-esque-ly useless, as how much useful one can do with a bare script without dependencies? And MicroPython definitely has some puzzle pieces for a generic solution, but so far not a complete picture: 1. There're frozen modules support, but they're that - modules, packages not supported for freezing so far. 2. Then frozen modules require recompiling, and that's not real-world scalable. 3. Standard library is already distributed (vs CPython's monolithic), but half of modules are dummy so far. That said, making a demo of self-contained webapp server in 350-400K is definitely on my TODO list (package support for frozen modules is the only blocker for that). Cheers, Nick. -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 05/28/2015 05:58 PM, Victor Stinner wrote: Why not continue to enhance Python 3 instead of wasting our time with Python 2? We have limited resources in term of developers to maintain Python. Uh, guys, you might as well call off the debate. Benjamin already checked it in. https://hg.python.org/cpython/rev/17d3bbde60d2 //arry/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
2015-05-28 18:07 GMT+02:00 Guido van Rossum gu...@python.org: This patch could save companies like Dropbox a lot of money. We run a ton of Python code in large datacenters, and while we are slow in moving to Python 3, we're good at updating to the latest 2.7. I'm not sure that backporting optimizations would motivate companies like Dropbox to drop Python 2. It's already hard to convince someone to migrate to Python 3. Why not continue to enhance Python 3 instead of wasting our time with Python 2? We have limited resources in term of developers to maintain Python. (I'm not talking about fixing *bugs* in Python 2 which is fine with me.) -- By the way, I just wrote sixer, a new tool to generate patches to port OpenStack to Python 3 :-) https://pypi.python.org/pypi/sixer It's based on regex, so it's less reliable than 2to3, 2to6 or modernize, but it's just enough for my specific use case. On OpenStack, it's not possible to send one giant patch hello, this is python 3. Code is modified by small and incremental changes. Come on in the Python 3 world and... always look on the bright side of life ( https://www.youtube.com/watch?v=VOAtCOsNuVM )! Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On Thu, May 28, 2015 at 5:58 PM, Victor Stinner victor.stin...@gmail.com wrote: 2015-05-28 18:07 GMT+02:00 Guido van Rossum gu...@python.org: This patch could save companies like Dropbox a lot of money. We run a ton of Python code in large datacenters, and while we are slow in moving to Python 3, we're good at updating to the latest 2.7. I'm not sure that backporting optimizations would motivate companies like Dropbox to drop Python 2. It's already hard to convince someone to migrate to Python 3. Why not continue to enhance Python 3 instead of wasting our time with Python 2? We have limited resources in term of developers to maintain Python. As a matter of fact (unknown to me earlier) Dropbox is already using a private backport of this patch. So it won't affect Dropbox's decision anyway. But it might make life easier for other, smaller companies that don't have the same resources. However this talk of wasting our time with Python 2 needs to stop, and if you think that making Python 2 less attractive will encourage people to migrate to Python 3, think again. Companies like Intel are *contributing* by offering this backport up publicly. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
YESSS!!! On Thu, May 28, 2015 at 8:09 PM, Larry Hastings la...@hastings.org wrote: On 05/28/2015 05:58 PM, Victor Stinner wrote: Why not continue to enhance Python 3 instead of wasting our time with Python 2? We have limited resources in term of developers to maintain Python. Uh, guys, you might as well call off the debate. Benjamin already checked it in. https://hg.python.org/cpython/rev/17d3bbde60d2 */arry* ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On Thu, May 28, 2015 at 5:01 PM, Paul Sokolovsky pmis...@gmail.com wrote: That said, making a demo of self-contained webapp server in 350-400K is definitely on my TODO list (package support for frozen modules is the only blocker for that). It may be worth taking this over to import-...@python.org for more discussion. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On Thu, May 28, 2015 at 6:10 PM Larry Hastings la...@hastings.org wrote: On 05/28/2015 05:58 PM, Victor Stinner wrote: Why not continue to enhance Python 3 instead of wasting our time with Python 2? We have limited resources in term of developers to maintain Python. Uh, guys, you might as well call off the debate. Benjamin already checked it in. https://hg.python.org/cpython/rev/17d3bbde60d2 Thanks Benjamin! :) */arry* ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/greg%40krypto.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 28.05.2015 02:17, Parasa, Srinivas Vamsi wrote: Hi All, This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation. Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email). +1. It's been successful for Python 3, doesn't change semantics and increases performance. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 28 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On Thu, May 28, 2015 at 3:17 AM, Parasa, Srinivas Vamsi srinivas.vamsi.par...@intel.com wrote: Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon – Haswell EP CPU with 18 cores, hyper-threading off, turbo off). Hi Vamsi, Thank you for your work and your detailed email. I'm -1 on the idea because: * Performance improvements are not bug fixes * The patch doesn't make the migration process from Python 2 to Python 3 easier * In long term, it would be nice to work on making Python 3 better: See http://bugs.python.org/issue11549 for an example task. --Berker ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 28 May 2015 at 19:47, Nick Coghlan ncogh...@gmail.com wrote: That's going to be a negotiation process - companies don't typically contribute paid development time to open source projects out of the kindness of their hearts, they do it either because they're using the project themselves, because of deals they've made with individual contributors around how they spend their time, or because it helps them influence the direction of upstream development in ways that help them and their customers. (And sometimes it's a mix of all 3 of those factors) And to be completely transparent about this: this probably won't be the last of these kinds of discussions we're likely to see. Various folks (including me) have been negotiating to have their employers fund paid CPython contribution positions and as we coach colleagues that take up these roles through the core team's contribution processes, one of the consequences will be that we will sometimes advocate for acceptance of changes that we would have historically rejected as imposing too high a maintenance burden for an all-volunteer development team to be expected to deal with. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
won't this need python compiled with gcc 5.1 to have any effect? Which compiler version was used for the benchmark? the issue that negated most computed goto improvements (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=39284) was only closed very recently (r212172, 9f4ec746affbde1) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 28 May 2015 at 18:54, Berker Peksağ berker.pek...@gmail.com wrote: On Thu, May 28, 2015 at 3:17 AM, Parasa, Srinivas Vamsi srinivas.vamsi.par...@intel.com wrote: Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon – Haswell EP CPU with 18 cores, hyper-threading off, turbo off). Hi Vamsi, Thank you for your work and your detailed email. I'm -1 on the idea because: * Performance improvements are not bug fixes The nothing except backwards compatible bug fixes in maintenance releases rule was adopted for the *benefit of Python users*. When a new feature release can be reliably expected every 18-24 months, it makes sense to err heavily on the side of minimising risks to stability when it comes to making judgement calls on whether or not a change is appropriate to a maintenance release or not. Python 2.7 is an unusual case, as even though there *are* newer feature releases available, the barriers to migration are much higher than they would otherwise be. Each progressive 3.x release has brought those barriers down a bit, and 3.5 and the static type checking work being done through mypy et al will bring them down even further, but version migration work is still work where end users don't see any immediate practical benefit - they only see the benefit *after* they're able to drop Python 2 compatibility, and can start using Python 3 only features like matrix multiplication and the async/await syntax. *Adding* features to Python 2.7 is quite rightly still controversial, as they add complexity to the compatibility matrix for testing purposes. Code that runs correctly with the PEP 466 and 476 changes to SSL handling, may fail on earlier versions. Internal performance improvements, by contrast, don't hurt end users at all beyond the stability risks, and in this case, the request to make the change is being accompanied by the offer to assist with ongoing maintenance (including engaging an experienced core developer to help coach Intel contributors through the contribution process). So when folks ask What changed? in relation to this request, what changed is the fact that it isn't expected to be a one off contribution, but rather part of a broader effort focused on improving the performance of both Python 2 and Python 3, including contributions to ongoing maintenance activities. * The patch doesn't make the migration process from Python 2 to Python 3 easier * In long term, it would be nice to work on making Python 3 better: Indeed, but we also need help living up to the Python 2.7 will be supported to 2020 commitment. Python 2.7 maintenance is *not* a particularly exciting task, and it's only going to get less interesting as Python 3 adoption climbs, so we're going to need paid contributors to start filling the gap as volunteers (quite reasonably) move on to the more inherently rewarding task of working to move Python 3 forward. That's going to be a negotiation process - companies don't typically contribute paid development time to open source projects out of the kindness of their hearts, they do it either because they're using the project themselves, because of deals they've made with individual contributors around how they spend their time, or because it helps them influence the direction of upstream development in ways that help them and their customers. (And sometimes it's a mix of all 3 of those factors) Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
I'm -1 on the idea because: * Performance improvements are not bug fixes * The patch doesn't make the migration process from Python 2 to Python 3 easier And this is why people have been porting Python applications to Go. Maybe addressing Python performance and making Python (2 or 3) a better language/platform would mitigate that. Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote: Hi All, This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation. Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email). Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP CPU with 18 cores, hyper-threading off, turbo off). Below is a summary of the performance we saw on the grand unified python benchmarks suite (available at https://hg.python.org/benchmarks/). We made 3 rigorous runs of the following benchmarks. In each rigorous run, a benchmark is run 100 times with and without the computed goto patch. Below we show the average performance boost for the 3 rigorous runs. Python 2.7.10 (original) vs Computed Goto performance Benchmark -1 As Gregory pointed out, there are other options to build the interpreter, and we are missing data how these compare with your patch. I assume, you tested with the Intel compiler, so it would be good to see results for other compilers as well (GCC, clang). Please could you provide the data for LTO and profile guided optimized builds (maybe combined too)? I'm happy to work with you on setting up these builds, but currently don't have the machine resources to do so myself. If the benefits show up for these configurations too, then I'm +/-0 on this patch. Matthias ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
Hi Matthias and Gregory, The results shown were run on Python 2.7.10 built using gcc. The goal of our team is to make long-term open source contributions with emphasis on performance optimization and support for the larger community and hence icc wasn't used. We've experimented with gcc profile-guided optimization (PGO) and LTO a month ago. PGO being an independent/orthogonal optimization, it shows improvement for both the stock version (i.e. current switch based dispatch) and the computed-goto version. We ran PGO optimized Python on the workloads available at language benchmarks game (http://benchmarksgame.alioth.debian.org/u64/python.php) and found that PGO benefits computed-goto version more than the stock version. I haven't run PGO optimized Python with the grand unified python benchmarks (GUPB) suite ...please give me a day or two and will get back to you with PGO (and LTO) numbers as well. (LTO hasn't shown much benefit so far on the language benchmarks game workloads). Also, in our analysis using CPU performance counters, we found that python workloads (in general) have higher CPU front-end issues (mainly I-cache misses) and PGO is very helpful in mitigating those issues. We're also investigating and working on ways to further reduce those front-end issues and speedup Python workloads. Thanks, Vamsi -Original Message- From: Matthias Klose [mailto:d...@ubuntu.com] Sent: Thursday, May 28, 2015 5:01 AM To: Parasa, Srinivas Vamsi; 'python-dev@python.org' Subject: Re: [Python-Dev] Computed Goto dispatch for Python 2 On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote: Hi All, This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation. Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email). Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP CPU with 18 cores, hyper-threading off, turbo off). Below is a summary of the performance we saw on the grand unified python benchmarks suite (available at https://hg.python.org/benchmarks/). We made 3 rigorous runs of the following benchmarks. In each rigorous run, a benchmark is run 100 times with and without the computed goto patch. Below we show the average performance boost for the 3 rigorous runs. Python 2.7.10 (original) vs Computed Goto performance Benchmark -1 As Gregory pointed out, there are other options to build the interpreter, and we are missing data how these compare with your patch. I assume, you tested with the Intel compiler, so it would be good to see results for other compilers as well (GCC, clang). Please could you provide the data for LTO and profile guided optimized builds (maybe combined too)? I'm happy to work with you on setting up these builds, but currently don't have the machine resources to do so myself. If the benefits show up for these configurations too, then I'm +/-0 on this patch. Matthias ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 28 May 2015 at 22:00, Matthias Klose d...@ubuntu.com wrote: On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote: Hi All, This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation. Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email). Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP CPU with 18 cores, hyper-threading off, turbo off). Below is a summary of the performance we saw on the grand unified python benchmarks suite (available at https://hg.python.org/benchmarks/). We made 3 rigorous runs of the following benchmarks. In each rigorous run, a benchmark is run 100 times with and without the computed goto patch. Below we show the average performance boost for the 3 rigorous runs. Python 2.7.10 (original) vs Computed Goto performance Benchmark -1 As Gregory pointed out, there are other options to build the interpreter, and we are missing data how these compare with your patch. That's shifting the goal posts: suggesting that we should optimise Python 2 differently from the way we optimised Python 3 isn't a reasonable request to make in the context of a backporting proposal. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On May 28, 2015 at 10:10:03 AM, Nick Coghlan (ncogh...@gmail.com) wrote: On 28 May 2015 at 21:55, Maciej Fijalkowski wrote: I'm -1 on the idea because: * Performance improvements are not bug fixes * The patch doesn't make the migration process from Python 2 to Python 3 easier And this is why people have been porting Python applications to Go. For folks hitting the kinds of scalability problems that Go is designed to help with, a few percentage points here and there in CPython performance aren't going to make a big difference - they'll need the kinds of speed multipliers that PyPy can offer. Given that Go can't run Python C extensions any more than PyPy can, and involves a rewrite in a different programming language to boot, we'd do well to ponder what Go currently offers that PyPy doesn't. If we ignore the fashion-driven aspect of Google wrote it, so it must be cool (which we can't do anything about), and if we ignore the multi-vendor commercial support question (which tends to significantly lag community adoption for true community driven projects like PyPy), then one of the big keys in my view is the easy redistributability of Go binaries. For Linux based network services (and even Windows these days), Docker containers offer a potentially compelling way of bundling the PyPy runtime with Python applications, and Docker, Inc already maintain a set of PyPy base images at https://registry.hub.docker.com/_/pypy/ Docker's image layering model then means that applications sharing a PyPy runtime shouldn't need to download the interpreter runtime itself more than once. As a result, I personally suspect that better documenting and promoting the CPython-Docker+PyPy migration option is likely to offer a more effective alternative to CPython-Go migrations than the more modest performance improvements we can make to the CPython runtime itself. (I still think the latter are a good idea, though - there's no point leaving Python 2.7 slower than it needs to be given the offer of assistance in maintaining it) Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io I think docker is a pretty crummy answer to Go’s static binaries. What I would love is for Python to get: * The ability to import .so modules via zipzimport (ideally without a temporary directory, but that might require newer APIs from libc and such). * The ability to create a “static” Python that links everything it needs into the binary to do a zipimport of everything else (including the stdlib). * The ability to execute a zipfile that has been concat onto the end of the Python binary. I think that if we get all of that, we could easily create a single file executable with real, native support from Python by simply compiling Python in that static mode and then appending a zip file containing the standard library and any other distributions we need to the end of it. We’d probably want some more quality of life improvements around accessing resources from within that zip file as well, but that can be done as a library easier than the above three things can. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 28 May 2015 at 21:55, Maciej Fijalkowski fij...@gmail.com wrote: I'm -1 on the idea because: * Performance improvements are not bug fixes * The patch doesn't make the migration process from Python 2 to Python 3 easier And this is why people have been porting Python applications to Go. For folks hitting the kinds of scalability problems that Go is designed to help with, a few percentage points here and there in CPython performance aren't going to make a big difference - they'll need the kinds of speed multipliers that PyPy can offer. Given that Go can't run Python C extensions any more than PyPy can, and involves a rewrite in a different programming language to boot, we'd do well to ponder what Go currently offers that PyPy doesn't. If we ignore the fashion-driven aspect of Google wrote it, so it must be cool (which we can't do anything about), and if we ignore the multi-vendor commercial support question (which tends to significantly lag community adoption for true community driven projects like PyPy), then one of the big keys in my view is the easy redistributability of Go binaries. For Linux based network services (and even Windows these days), Docker containers offer a potentially compelling way of bundling the PyPy runtime with Python applications, and Docker, Inc already maintain a set of PyPy base images at https://registry.hub.docker.com/_/pypy/ Docker's image layering model then means that applications sharing a PyPy runtime shouldn't need to download the interpreter runtime itself more than once. As a result, I personally suspect that better documenting and promoting the CPython-Docker+PyPy migration option is likely to offer a more effective alternative to CPython-Go migrations than the more modest performance improvements we can make to the CPython runtime itself. (I still think the latter are a good idea, though - there's no point leaving Python 2.7 slower than it needs to be given the offer of assistance in maintaining it) Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On Thu, May 28, 2015 at 8:47 AM Raymond Hettinger raymond.hettin...@gmail.com wrote: On May 28, 2015, at 1:54 AM, Berker Peksağ berker.pek...@gmail.com wrote: * Performance improvements are not bug fixes Practicality beats purity here. Recognize that a huge number of Python users will remain in the Python2.7 world for some time. We have a responsibility to the bulk of our users (my estimate is that adoption rate for Python 3 is under 2%). Where does that number come from? I have not seen numbers less than 5% for the overall community adoption (and all of them are extremely rough and all skewed towards Python 2 for various technical reasons). The computed goto patch makes substantial performance improvements. It is callous to deny the improvement to 2.7 users. But you could argue that Special cases aren't special enough to break the rules and that's what we are proposing here by claiming Python 2.7 is a special case and thus we should accept a patch that is not a one-liner change to gain some performance in a bugfix release. * The patch doesn't make the migration process from Python 2 to Python 3 easier Sorry, that is a red-herring (an orthogonal issue). If you care about 2-to-3 migration, then start opposing proposals for API changes that increase the semantic difference between 2 and 3. That's misdirection for Berker's point that the proposal at hand does not help with getting to people to Python 3 even though what is proposed is an enhancement and not a bugfix (since we are not fixing a performance regression). I had someone on Twitter earlier this month point out that Python 3 was slower than Python 2 on some benchmark and that was a reason they weren't switching. Doing this is not going to help make that case (although I think arguing about performance between 2 and 3 is misleading when I've seen other workloads win out in Python 3). I'm -0 on this because I would like to stick to our policy of no enhancements in a bugfix release, but in the end it's Benjamin's call as 2.7 RM as to whether this is appropriate for 2.7.11. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 28 May 2015 at 23:37, Nick Coghlan ncogh...@gmail.com wrote: On 28 May 2015 at 22:00, Matthias Klose d...@ubuntu.com wrote: On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote: Hi All, This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation. Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email). Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP CPU with 18 cores, hyper-threading off, turbo off). Below is a summary of the performance we saw on the grand unified python benchmarks suite (available at https://hg.python.org/benchmarks/). We made 3 rigorous runs of the following benchmarks. In each rigorous run, a benchmark is run 100 times with and without the computed goto patch. Below we show the average performance boost for the 3 rigorous runs. Python 2.7.10 (original) vs Computed Goto performance Benchmark -1 As Gregory pointed out, there are other options to build the interpreter, and we are missing data how these compare with your patch. That's shifting the goal posts: suggesting that we should optimise Python 2 differently from the way we optimised Python 3 isn't a reasonable request to make in the context of a backporting proposal. Sorry, I misread your email. I thought you were talking about the part of Greg's email where he suggested different optimisation techniques, but you were actually referring to the part where he requested more detail on the compiler used and the number for gcc and clang. *That* request I agree with. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On Thu, May 28, 2015 at 10:47 AM Skip Montanaro skip.montan...@gmail.com wrote: On Thu, May 28, 2015 at 9:02 AM, Brett Cannon br...@python.org wrote: But you could argue that Special cases aren't special enough to break the rules and that's what we are proposing here by claiming Python 2.7 is a special case and thus we should accept a patch that is not a one-liner change to gain some performance in a bugfix release. One can read anything he wants into the Zen. I could respond with this retort: Although practicality beats purity, but I won't. :-) Because Raymond already said that. =) And you explicitly made the point I was trying to implicitly make: in this instance there is no clear argument for or against that is going to make this an obvious decision based on PEP 20. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On Thu, May 28, 2015 at 9:02 AM, Brett Cannon br...@python.org wrote: But you could argue that Special cases aren't special enough to break the rules and that's what we are proposing here by claiming Python 2.7 is a special case and thus we should accept a patch that is not a one-liner change to gain some performance in a bugfix release. One can read anything he wants into the Zen. I could respond with this retort: Although practicality beats purity, but I won't. :-) Skip ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 28 May 2015 at 15:37, Donald Stufft don...@stufft.io wrote: I think docker is a pretty crummy answer to Go’s static binaries. What I would love is for Python to get: * The ability to import .so modules via zipzimport (ideally without a temporary directory, but that might require newer APIs from libc and such). * The ability to create a “static” Python that links everything it needs into the binary to do a zipimport of everything else (including the stdlib). * The ability to execute a zipfile that has been concat onto the end of the Python binary. I think that if we get all of that, we could easily create a single file executable with real, native support from Python by simply compiling Python in that static mode and then appending a zip file containing the standard library and any other distributions we need to the end of it. We’d probably want some more quality of life improvements around accessing resources from within that zip file as well, but that can be done as a library easier than the above three things can. +1. The new embeddable Python distribution for Windows is a great step forward for this. It's not single-file, but it's easy to produce a single-directory self-contained application with it. I don't know if there's anything equivalent for Linux/OSX - maybe it's something we should look at for them as well (although the whole static binaries concept seems to be fairly frowned on in the Unix world, from what I've seen). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 28/05/2015 15:47, Skip Montanaro wrote: On Thu, May 28, 2015 at 9:02 AM, Brett Cannon br...@python.org wrote: But you could argue that Special cases aren't special enough to break the rules and that's what we are proposing here by claiming Python 2.7 is a special case and thus we should accept a patch that is not a one-liner change to gain some performance in a bugfix release. One can read anything he wants into the Zen. I could respond with this retort: Although practicality beats purity, but I won't. :-) Skip That's good, otherwise you'd just be repeating what Raymond said further up this subthread two hours and one minute before you didn't say it :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On May 28, 2015 at 10:55:08 AM, Steve Dower (steve.do...@microsoft.com) wrote: Donald Stufft wrote: I think docker is a pretty crummy answer to Go’s static binaries. What I would love is for Python to get: * The ability to import .so modules via zipzimport (ideally without a temporary directory, but that might require newer APIs from libc and such). * The ability to create a “static” Python that links everything it needs into the binary to do a zipimport of everything else (including the stdlib). * The ability to execute a zipfile that has been concat onto the end of the Python binary. I think that if we get all of that, we could easily create a single file executable with real, native support from Python by simply compiling Python in that static mode and then appending a zip file containing the standard library and any other distributions we need to the end of it. And it would look like a 20MB+ file just for a simple 1KB Python script... For Windows at least, I'd prefer to have some app-style installer generation (e.g. http://pynsist.readthedocs.org/en/latest/) which, combined with the embeddable Python distro (new for 3.5.0b1 in case anyone missed it), can simply extract everything into an install directory and run it from there. None of the items on the list above are needed for or would help with this. (Some other Windows-specific advantages of the latter - installers get special compatibility treatment when the OS does stuff to break them, modifying official Python binaries breaks the signatures, signed executables are fully scanned before running (slow if the file is big), IT departments know how to deal with installers and users know how to deal with installed binaries, and probably more.) Alright everyone, back on topic now unless you want to rename the thread :) Cheers, Steve Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python script + whatever other pieces you need. It would be entirely possible to only include the parts of the standard library you actually need. There's no rule that if your single file executable doesn't use xmlrpc that you have to include xmlrpc just for purities sake. This isn't something that can't be done today using something like PyInstaller, it's just super janky and finnicky because it's being hacked in after the fact by PyInstaller and because it's not an officially supported thing a lot of projects simply don't support it. A CLI I worked on that uses PyInstaller is 5MB. It's certainly a trade off but it's not nearly as big of a trade off as you say. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
Donald Stufft wrote: I think docker is a pretty crummy answer to Go’s static binaries. What I would love is for Python to get: * The ability to import .so modules via zipzimport (ideally without a temporary directory, but that might require newer APIs from libc and such). * The ability to create a “static” Python that links everything it needs into the binary to do a zipimport of everything else (including the stdlib). * The ability to execute a zipfile that has been concat onto the end of the Python binary. I think that if we get all of that, we could easily create a single file executable with real, native support from Python by simply compiling Python in that static mode and then appending a zip file containing the standard library and any other distributions we need to the end of it. And it would look like a 20MB+ file just for a simple 1KB Python script... For Windows at least, I'd prefer to have some app-style installer generation (e.g. http://pynsist.readthedocs.org/en/latest/) which, combined with the embeddable Python distro (new for 3.5.0b1 in case anyone missed it), can simply extract everything into an install directory and run it from there. None of the items on the list above are needed for or would help with this. (Some other Windows-specific advantages of the latter - installers get special compatibility treatment when the OS does stuff to break them, modifying official Python binaries breaks the signatures, signed executables are fully scanned before running (slow if the file is big), IT departments know how to deal with installers and users know how to deal with installed binaries, and probably more.) Alright everyone, back on topic now unless you want to rename the thread :) Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On May 28, 2015, at 1:54 AM, Berker Peksağ berker.pek...@gmail.com wrote: * Performance improvements are not bug fixes Practicality beats purity here. Recognize that a huge number of Python users will remain in the Python2.7 world for some time. We have a responsibility to the bulk of our users (my estimate is that adoption rate for Python 3 is under 2%). The computed goto patch makes substantial performance improvements. It is callous to deny the improvement to 2.7 users. * The patch doesn't make the migration process from Python 2 to Python 3 easier Sorry, that is a red-herring (an orthogonal issue). If you care about 2-to-3 migration, then start opposing proposals for API changes that increase the semantic difference between 2 and 3. Raymond ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
Wow. Such thread. :-) This patch could save companies like Dropbox a lot of money. We run a ton of Python code in large datacenters, and while we are slow in moving to Python 3, we're good at updating to the latest 2.7. The patch is forward and backward compatible.I'm strongly in favor. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On May 27, 2015, at 6:31 PM, Nick Coghlan ncogh...@gmail.com wrote: On 28 May 2015 at 10:17, Parasa, Srinivas Vamsi srinivas.vamsi.par...@intel.com wrote: Hi All, This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation. Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email). +1 from me, for basically the same reasons Guido gives: Python 2.7 is going to be with us for a long time, and this particular change shouldn't have any externally visible impacts at either an ABI or API level. +1 from me a well. We probably should have done this long ago. Raymond Hettinger ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 5/27/2015 9:31 PM, Nick Coghlan wrote: +1 from me, for basically the same reasons Guido gives: Python 2.7 is going to be with us for a long time, and this particular change shouldn't have any externally visible impacts at either an ABI or API level. Immediately after a release, giving the patch plenty of time to be be tested, seems like a good time. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
Why now? We intentionally decided not to do this for 2.7 in the past because it was too late for the release cutoff. Has anyone benchmarked compiling python in profile-opt mode vs having the computed goto patch? I'd *expect* the benefits to be the roughly the same. Has this been compared to that? (Anyone not compiling their Python interpreter in profile-opt mode is doing themselves a major disservice.) Does it shows noteworthy improvements even when used with a profile-opt build using a PROFILE_TASK of regrtest.py with the same test exclusion list as the debian python2.7 package? Could you please rerun benchmarks including the profile-opt build with and without the patch for comparsion. -gps PS I recommend attaching the up to date patch against 2.7.10 to issue4753. That is where anyone will go looking for it, not buried in a mailing list archive. On Wed, May 27, 2015 at 10:01 PM Terry Reedy tjre...@udel.edu wrote: On 5/27/2015 9:31 PM, Nick Coghlan wrote: +1 from me, for basically the same reasons Guido gives: Python 2.7 is going to be with us for a long time, and this particular change shouldn't have any externally visible impacts at either an ABI or API level. Immediately after a release, giving the patch plenty of time to be be tested, seems like a good time. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/greg%40krypto.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Computed Goto dispatch for Python 2
On 28 May 2015 at 10:17, Parasa, Srinivas Vamsi srinivas.vamsi.par...@intel.com wrote: Hi All, This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation. Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email). +1 from me, for basically the same reasons Guido gives: Python 2.7 is going to be with us for a long time, and this particular change shouldn't have any externally visible impacts at either an ABI or API level. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
