Re: [Python-Dev] PEP 488: elimination of PYO files
Here is the latest draft of the PEP. I have closed the issue of file name
formatting thanks to the informal poll results being very clear on the
preferred format and also closed the idea of embedding the optimization
level in the bytecode file metadata (that can be another PEP if someone
cares to write that one). The only remaining open issue is whether to
special-case non-optimized bytecode file names and completely leave out the
optimization level in that instance.
I think this PEP is close enough to be finished to ask whether Guido wants
to wield his BDFL stick for this PEP or if he wants to delegate to a BDFAP.
PEP: 488
Title: Elimination of PYO files
Version: $Revision$
Last-Modified: $Date$
Author: Brett Cannon
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Feb-2015
Post-History:
2015-03-06
2015-03-13
Abstract
This PEP proposes eliminating the concept of PYO files from Python.
To continue the support of the separation of bytecode files based on
their optimization level, this PEP proposes extending the PYC file
name to include the optimization level in bytecode repository
directory (i.e., the ``__pycache__`` directory).
Rationale
=
As of today, bytecode files come in two flavours: PYC and PYO. A PYC
file is the bytecode file generated and read from when no
optimization level is specified at interpreter startup (i.e., ``-O``
is not specified). A PYO file represents the bytecode file that is
read/written when **any** optimization level is specified (i.e., when
``-O`` is specified, including ``-OO``). This means that while PYC
files clearly delineate the optimization level used when they were
generated -- namely no optimizations beyond the peepholer -- the same
is not true for PYO files. Put in terms of optimization levels and
the file extension:
- 0: ``.pyc``
- 1 (``-O``): ``.pyo``
- 2 (``-OO``): ``.pyo``
The reuse of the ``.pyo`` file extension for both level 1 and 2
optimizations means that there is no clear way to tell what
optimization level was used to generate the bytecode file. In terms
of reading PYO files, this can lead to an interpreter using a mixture
of optimization levels with its code if the user was not careful to
make sure all PYO files were generated using the same optimization
level (typically done by blindly deleting all PYO files and then
using the `compileall` module to compile all-new PYO files [1]_).
This issue is only compounded when people optimize Python code beyond
what the interpreter natively supports, e.g., using the astoptimizer
project [2]_.
In terms of writing PYO files, the need to delete all PYO files
every time one either changes the optimization level they want to use
or are unsure of what optimization was used the last time PYO files
were generated leads to unnecessary file churn. The change proposed
by this PEP also allows for **all** optimization levels to be
pre-compiled for bytecode files ahead of time, something that is
currently impossible thanks to the reuse of the ``.pyo`` file
extension for multiple optimization levels.
As for distributing bytecode-only modules, having to distribute both
``.pyc`` and ``.pyo`` files is unnecessary for the common use-case
of code obfuscation and smaller file deployments.
Proposal
To eliminate the ambiguity that PYO files present, this PEP proposes
eliminating the concept of PYO files and their accompanying ``.pyo``
file extension. To allow for the optimization level to be unambiguous
as well as to avoid having to regenerate optimized bytecode files
needlessly in the `__pycache__` directory, the optimization level
used to generate a PYC file will be incorporated into the bytecode
file name. Currently bytecode file names are created by
``importlib.util.cache_from_source()``, approximately using the
following expression defined by PEP 3147 [3]_, [4]_, [5]_::
'{name}.{cache_tag}.pyc'.format(name=module_name,
cache_tag=sys.implementation.cache_tag)
This PEP proposes to change the expression to::
'{name}.{cache_tag}.opt-{optimization}.pyc'.format(
name=module_name,
cache_tag=sys.implementation.cache_tag,
optimization=str(sys.flags.optimize))
The "opt-" prefix was chosen so as to provide a visual separator
from the cache tag. The placement of the optimization level after
the cache tag was chosen to preserve lexicographic sort order of
bytecode file names based on module name and cache tag which will
not vary for a single interpreter. The "opt-" prefix was chosen over
"o" so as to be somewhat self-documenting. The "opt-" prefix was
chosen over "O" so as to not have any confusion with "0" while being
so close to the interpreter version number.
A period was chosen over a hyphen as a separator so as to distinguish
clearly that the optimization level is not part of the interpreter
version as specified by the cache tag. It also
Re: [Python-Dev] PEP 488: elimination of PYO files
Hi Brett, On 6 March 2015 at 19:11, Brett Cannon wrote: > I disagree with your premise that .pyo files don't have a noticeable effect > on performance. If you don't use asserts a lot then there is no effect, but > if you use them heavily or have them perform expensive calculations then > there is an impact. Maybe you'd be interested to learn that PyPy (at least the 2.x branch) uses a new bytecode, JUMP_IF_NOT_DEBUG, to conditionally jump over the "assert" line. In "optimized" mode PyPy follows the jumps; in "non-optimized" mode it doesn't. This mode is initialized with the -O flag but can be changed dynamically, as the bytecode is the same. We introduced it as a simple solution to the mess of .pyc versus .pyo. (We didn't consider the case of -OO very closely because PyPy is much bigger than CPython as a binary to start with, so the demand for that is lower.) A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Wed, Mar 11, 2015 at 05:34:10PM +, Brett Cannon wrote: > I have a poll going on G+ to see what people think of the various proposed > file name formats at > https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm . Feel free to > vote if you have an opinion. G+ hates my browser and won't let me vote. I click on the button and nothing happens. I have Javascript enabled and I'm not using any ad blockers. For the record, I think only the first two options importlib.cpython-35.opt-0.pyc importlib.cpython-35.opt0.pyc are sane, and I prefer the first. I'm mildly inclined to leave out the opt* part for default, unoptimized code. In other words, the file name holds two or three '.' delimited fields, plus the extension: .-.[opt-].pyc where [...] is optional and the optimization codes for CPython will be 1 for -O and 2 for -OO. And 0 for unoptimized, if you decide that it should be mandatory. Thank you for moving forward on this, I think it is a good plan. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Wed, Mar 11, 2015 at 5:29 PM Armin Rigo wrote: > Hi Brett, > > On 6 March 2015 at 19:11, Brett Cannon wrote: > > I disagree with your premise that .pyo files don't have a noticeable > effect > > on performance. If you don't use asserts a lot then there is no effect, > but > > if you use them heavily or have them perform expensive calculations then > > there is an impact. > > Maybe you'd be interested to learn that PyPy (at least the 2.x branch) > uses a new bytecode, JUMP_IF_NOT_DEBUG, to conditionally jump over the > "assert" line. In "optimized" mode PyPy follows the jumps; in > "non-optimized" mode it doesn't. This mode is initialized with the -O > flag but can be changed dynamically, as the bytecode is the same. We > introduced it as a simple solution to the mess of .pyc versus .pyo. > (We didn't consider the case of -OO very closely because PyPy is much > bigger than CPython as a binary to start with, so the demand for that > is lower.) > Interesting, so you simply merged the optimization levels 0 and 1 in the bytecode and basically drop .pyo files thanks to it. That might be some motivation to support the default file name not having any specified optimization level at all. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 6:49 PM Benjamin Peterson wrote: > > > On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote: > > > > OK, but that doesn't influence the PEP's goal of dropping .pyo files. > > Correct. > > > > > Are you suggesting that the tag be changed to be less specific to > > optimizations and more free-form? Like > > `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff > > like this gets baked into the .pyc file itself instead of the file name, > > but I don't think we should just drop the ability to switch off asserts > > and > > docstrings like Mark seemed to be suggesting. > > Basically, though the filename strings could perhaps be more compact. > I have a poll going on G+ to see what people think of the various proposed file name formats at https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm . Feel free to vote if you have an opinion. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
For the record here: +1 on the PEP from me (the comments I made on import-sig have already incorporated into this version of the PEP) On 8 March 2015 at 08:03, Brett Cannon wrote: > > > On Sat, Mar 7, 2015 at 12:39 PM Scott Dial > wrote: >> >> On 2015-03-06 11:34 AM, Brett Cannon wrote: >> > This PEP proposes eliminating the concept of PYO files from Python. >> > To continue the support of the separation of bytecode files based on >> > their optimization level, this PEP proposes extending the PYC file >> > name to include the optimization level in bytecode repository >> > directory (i.e., the ``__pycache__`` directory). >> >> As a packager, this PEP is a bit silent on it's expectations about what >> will happen with (for instance) Debian and Fedora packages for Python. >> My familiarity is with Fedora, and on that platform, we ship .pyc and >> .pyo files (using -O for the .pyo). Is it your expectation that such >> platforms will still distribute -O only? Or also -OO? In my world, all >> of the __pycache__ directories are owned by root. > > > I assume they will generate all .pyc files at all levels, but I don't if > it's my place to dictate such a thing since bytecode files are an > optimization to Python itself and do not influence how people interact with > the interpreter like with PEP 394 (The "python" Command on Unix-Like > Systems). = Fedora near term = Nothing significant will change for RPMs with policy compliant (https://fedoraproject.org/wiki/Packaging:Python#Byte_compiling) spec files, even after switching to Python 3.5. The only different will be that the current .pyo files will be replaced with files using the new PEP 488 naming scheme. Folks using custom spec files with their own pattern matches that assume "pyo" may need to adjust them (e.g. to grab everything in __pycache__, as recommended in the packaging guidelines) = Fedora long term = After the switch to Python 3.5 (which, given the release schedule, I currently expect will be in Fedora 24 early next year), the RPM Python build macros *might* be changed to generate -OO files in addition to -O ones, making it easier to run Python applications in -OO mode to reduce memory usage. The PEP doesn't mandate such a change, but it does enable it. This would have the downside of making every Python package in the distro slightly larger (all 15k+ of them), so there'd need to be a clear and compelling pay-off to justify that cost. While folks may be tempted to say "disk space is cheap", VM and container provisioning latency are performance critical in a number of use cases, so making the Fedora Cloud and Atomic Host images even slightly larger isn't likely to win us any friends, so actually making such a change would require compelling benchmarks demonstrating reductions in runtime memory usage that justify the larger image size, as well as confirming that the change doesn't place undue pressure on the DVD ISO image contents. This isn't really the right list for more in-depth discussion of what Fedora *might* do though - any discussion will be entirely theoretical until after Python 3.5 comes out, and even then the appropriate list would be python-de...@lists.fedoraproject.org rather than here. Regards, Nick. P.S. For those that aren't already aware, I became a voting member of Fedora's Environments & Stacks Working Group several months ago and now work closely with Slavek Kabrda (the Fedora Python maintainer, amongst other things) on Python's integration into Fedora and its derivatives. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Mar 07, 2015, at 12:30 PM, Scott Dial wrote: >As a packager, this PEP is a bit silent on it's expectations about what >will happen with (for instance) Debian and Fedora packages for Python. >My familiarity is with Fedora, and on that platform, we ship .pyc and >.pyo files (using -O for the .pyo). Is it your expectation that such >platforms will still distribute -O only? Or also -OO? In my world, all >of the __pycache__ directories are owned by root. On Debuntu, we don't create pyo files by default, although IIRC it's an option. The problem has been the collision between -O and -OO, so this PEP may open the door to us generating both optimization levels automatically. It's not something that's very high on my todo list, but if there's enough interest it probably wouldn't be difficult to add. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Sat, Mar 7, 2015 at 12:39 PM Scott Dial wrote: > On 2015-03-06 11:34 AM, Brett Cannon wrote: > > This PEP proposes eliminating the concept of PYO files from Python. > > To continue the support of the separation of bytecode files based on > > their optimization level, this PEP proposes extending the PYC file > > name to include the optimization level in bytecode repository > > directory (i.e., the ``__pycache__`` directory). > > As a packager, this PEP is a bit silent on it's expectations about what > will happen with (for instance) Debian and Fedora packages for Python. > My familiarity is with Fedora, and on that platform, we ship .pyc and > .pyo files (using -O for the .pyo). Is it your expectation that such > platforms will still distribute -O only? Or also -OO? In my world, all > of the __pycache__ directories are owned by root. > I assume they will generate all .pyc files at all levels, but I don't if it's my place to dictate such a thing since bytecode files are an optimization to Python itself and do not influence how people interact with the interpreter like with PEP 394 (The "python" Command on Unix-Like Systems). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On 2015-03-06 11:34 AM, Brett Cannon wrote: > This PEP proposes eliminating the concept of PYO files from Python. > To continue the support of the separation of bytecode files based on > their optimization level, this PEP proposes extending the PYC file > name to include the optimization level in bytecode repository > directory (i.e., the ``__pycache__`` directory). As a packager, this PEP is a bit silent on it's expectations about what will happen with (for instance) Debian and Fedora packages for Python. My familiarity is with Fedora, and on that platform, we ship .pyc and .pyo files (using -O for the .pyo). Is it your expectation that such platforms will still distribute -O only? Or also -OO? In my world, all of the __pycache__ directories are owned by root. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Sat, Mar 7, 2015 at 9:29 AM Ron Adam wrote: > > > On 03/07/2015 04:58 AM, Steven D'Aprano wrote: > > On Fri, Mar 06, 2015 at 08:00:20PM -0500, Ron Adam wrote: > > > >> >Have you considered doing this by having different magic numbers in the > >> >.pyc file for standard, -O, and -O0 compiled bytecode files? Python > >> >already checks that number and recompiles the files if it's not what > it's > >> >expected to be. And it wouldn't require any naming conventions or new > >> >cache directories. It seems to me it would be much easier to do as > well. > > And it would fail to solve the problem. The problem isn't just that the > > .pyo file can contain the wrong byte-code for the optimization level, > > that's only part of the problem. Another issue is that you cannot have > > pre-compiled byte-code for multiple different optimization levels. You > > can have a "no optimization" byte-code file, the .pyc file, but only one > > "optimized" byte-code file at the same time. > > > > Brett's proposal will allow -O optimized and -OO optimized byte-code > > files to co-exist, as well as setting up a clear naming convention for > > future optimizers in either the Python compiler or third-party > > optimizers. > > So all the different versions can be generated ahead of time. I think that > is the main difference. > > My suggestion would cause a recompile of all dependent python files when > different optimisation levels are used in different projects. Which may be > worse than not generating bytecode files at all. OK > > > A few questions... > > Can a submodule use an optimazation level that is different from the file > that imports it? (Other than the case this is trying to solve.) > Currently yes, with this PEP no (without purposefully doing it with some custom loader). > > Is there way to specify that an imported module not use any optimisation > level, or to always use a specific optimisation level? > Not without a custom loader. > > Is there a way to run tests with all the different optimisation levels? > You have to remember you can't change the optimization levels of the interpreter once you have started it up. The change in semantics is handled deep in the AST compiler and there is no exposed way to flip-flop the setting once the interpreter starts. So to test the different optimization levels would require either (a) implementing the optimizations are part of some AST optimizer and doing the right thing in terms of reloading the modules, or (b) simply running the tests again by running the interpreter again with different flags (this is when something like tox is useful). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On 03/07/2015 04:58 AM, Steven D'Aprano wrote: On Fri, Mar 06, 2015 at 08:00:20PM -0500, Ron Adam wrote: >Have you considered doing this by having different magic numbers in the >.pyc file for standard, -O, and -O0 compiled bytecode files? Python >already checks that number and recompiles the files if it's not what it's >expected to be. And it wouldn't require any naming conventions or new >cache directories. It seems to me it would be much easier to do as well. And it would fail to solve the problem. The problem isn't just that the .pyo file can contain the wrong byte-code for the optimization level, that's only part of the problem. Another issue is that you cannot have pre-compiled byte-code for multiple different optimization levels. You can have a "no optimization" byte-code file, the .pyc file, but only one "optimized" byte-code file at the same time. Brett's proposal will allow -O optimized and -OO optimized byte-code files to co-exist, as well as setting up a clear naming convention for future optimizers in either the Python compiler or third-party optimizers. So all the different versions can be generated ahead of time. I think that is the main difference. My suggestion would cause a recompile of all dependent python files when different optimisation levels are used in different projects. Which may be worse than not generating bytecode files at all. OK A few questions... Can a submodule use an optimazation level that is different from the file that imports it? (Other than the case this is trying to solve.) Is there way to specify that an imported module not use any optimisation level, or to always use a specific optimisation level? Is there a way to run tests with all the different optimisation levels? Cheers, Ron ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 06, 2015 at 08:00:20PM -0500, Ron Adam wrote: > Have you considered doing this by having different magic numbers in the > .pyc file for standard, -O, and -O0 compiled bytecode files? Python > already checks that number and recompiles the files if it's not what it's > expected to be. And it wouldn't require any naming conventions or new > cache directories. It seems to me it would be much easier to do as well. And it would fail to solve the problem. The problem isn't just that the .pyo file can contain the wrong byte-code for the optimization level, that's only part of the problem. Another issue is that you cannot have pre-compiled byte-code for multiple different optimization levels. You can have a "no optimization" byte-code file, the .pyc file, but only one "optimized" byte-code file at the same time. Brett's proposal will allow -O optimized and -OO optimized byte-code files to co-exist, as well as setting up a clear naming convention for future optimizers in either the Python compiler or third-party optimizers. No new cache directories are needed. The __pycache__ directory has been used since Python 3.3 (or was it 3.2? I forget which). -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On 03/06/2015 11:34 AM, Brett Cannon wrote: There are currently two open issues, although one is purely a bikeshed topic on formatting of file names so I don't really consider it open for change from what is proposed in the PEP without Guido saying he hates my preference or someone having a really good argument for some alternative. The second open issue on the common case file name is something to reasonably debate and come to consensus on. === PEP: 488 Title: Elimination of PYO files +1 And... Have you considered doing this by having different magic numbers in the .pyc file for standard, -O, and -O0 compiled bytecode files? Python already checks that number and recompiles the files if it's not what it's expected to be. And it wouldn't require any naming conventions or new cache directories. It seems to me it would be much easier to do as well. Cheers, Ron ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 6:49 PM Benjamin Peterson
wrote:
>
>
> On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote:
> >
> > OK, but that doesn't influence the PEP's goal of dropping .pyo files.
>
> Correct.
>
> >
> > Are you suggesting that the tag be changed to be less specific to
> > optimizations and more free-form? Like
> > `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff
> > like this gets baked into the .pyc file itself instead of the file name,
> > but I don't think we should just drop the ability to switch off asserts
> > and
> > docstrings like Mark seemed to be suggesting.
>
> Basically, though the filename strings could perhaps be more compact.
>
That's fine. Do you have a file name format you want to propose then
instead of "opt-{}" (which is what I'm assuming your "basically" is
referring to)?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 5:47 PM Antoine Pitrou wrote: > On Sat, 7 Mar 2015 09:34:20 +1100 > Steven D'Aprano wrote: > > > On Fri, Mar 06, 2015 at 09:37:05PM +0100, Antoine Pitrou wrote: > > > On Fri, 06 Mar 2015 18:11:19 + > > > Brett Cannon wrote: > > > > And the dropping of docstrings does have an impact on > > > > memory usage when you use Python at scale. > > > > > > What kind of "scale" are you talking about? Do you have any numbers > > > about such impact? > > > > > > > You're also assuming that we will never develop an AST optimizer > > > > > > No, the assumption is that we don't have such an optimizer *right now*. > > > Having command-line options because they might be useful some day is > > > silly. > > > > Quoting the PEP: > > > > This issue is only compounded when people optimize Python > > code beyond what the interpreter natively supports, e.g., > > using the astoptimizer project [2]_. > > The astoptimizer project is not part of Python. It's third-party > software that has no relationship to .pyo files. > Directly, no. But the point is that the PEP enables the astoptimizer project to write out .pyc files specifying different optimizations that won't clash with -O or -OO .pyc files. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 3:37 PM Antoine Pitrou wrote: > On Fri, 06 Mar 2015 18:11:19 + > Brett Cannon wrote: > > And the dropping of docstrings does have an impact on > > memory usage when you use Python at scale. > > What kind of "scale" are you talking about? Do you have any numbers > about such impact? > I know YouTube at least uses -OO and I don't have numbers to share (numbers I were last shown were years ago and I wouldn't be authorized to share anyway, but I do know they still use -OO). > > > You're also assuming that we will never develop an AST optimizer > > No, the assumption is that we don't have such an optimizer *right now*. > Having command-line options because they might be useful some day is > silly. > I'm not talking about changing any command-line option in the PEP so I don't know what you're referring to. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote: > > OK, but that doesn't influence the PEP's goal of dropping .pyo files. Correct. > > Are you suggesting that the tag be changed to be less specific to > optimizations and more free-form? Like > `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff > like this gets baked into the .pyc file itself instead of the file name, > but I don't think we should just drop the ability to switch off asserts > and > docstrings like Mark seemed to be suggesting. Basically, though the filename strings could perhaps be more compact. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015, at 15:13, Chris Angelico wrote: > On Sat, Mar 7, 2015 at 6:09 AM, Benjamin Peterson > wrote: > > I think it would be preferable deprecate -O and -OO and replace them > > with flags like --no-docstrings or --no-asserts. Ideally, "optimization" > > levels shouldn't change program semantics. > > Plenty of C compilers have optimization levels that can change > behaviour (replacing division with mult-by-reciprocal and such), so I > don't see any reason for Python to object. Yes, but in the C world, you have pass scary flags like -ffast-math. Just -O2 generally won't reach outside of standard semantics. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Sat, 7 Mar 2015 09:34:20 +1100 Steven D'Aprano wrote: > On Fri, Mar 06, 2015 at 09:37:05PM +0100, Antoine Pitrou wrote: > > On Fri, 06 Mar 2015 18:11:19 + > > Brett Cannon wrote: > > > And the dropping of docstrings does have an impact on > > > memory usage when you use Python at scale. > > > > What kind of "scale" are you talking about? Do you have any numbers > > about such impact? > > > > > You're also assuming that we will never develop an AST optimizer > > > > No, the assumption is that we don't have such an optimizer *right now*. > > Having command-line options because they might be useful some day is > > silly. > > Quoting the PEP: > > This issue is only compounded when people optimize Python > code beyond what the interpreter natively supports, e.g., > using the astoptimizer project [2]_. The astoptimizer project is not part of Python. It's third-party software that has no relationship to .pyo files. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 06, 2015 at 09:37:05PM +0100, Antoine Pitrou wrote: > On Fri, 06 Mar 2015 18:11:19 + > Brett Cannon wrote: > > And the dropping of docstrings does have an impact on > > memory usage when you use Python at scale. > > What kind of "scale" are you talking about? Do you have any numbers > about such impact? > > > You're also assuming that we will never develop an AST optimizer > > No, the assumption is that we don't have such an optimizer *right now*. > Having command-line options because they might be useful some day is > silly. Quoting the PEP: This issue is only compounded when people optimize Python code beyond what the interpreter natively supports, e.g., using the astoptimizer project [2]_. Brett, I'm a very strong +1 on the PEP. It's well-written and gives a good explanation for why such a thing is needed. The current behaviour of re-using the same .pyo file for two distinct sets of bytecode is out-and-out buggy: [steve@ando ~]$ python3.3 -O -c "import dis; print(dis.__doc__[:32])" Disassembler of Python byte code [steve@ando ~]$ python3.3 -OO -c "import dis; print(dis.__doc__[:32])" Disassembler of Python byte code The second should fail, since doc strings should be removed under -OO optimization, but because the .pyo file already exists it doesn't. Even if CPython drops -O and -OO altogether, this PEP should still be accepted to allow third party optimizers like astoptimizer to interact without getting in each other's way. (And for the record, I'm an equally strong -1 on dropping -O and -OO.) Thank you. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, 06 Mar 2015 18:11:19 + Brett Cannon wrote: > And the dropping of docstrings does have an impact on > memory usage when you use Python at scale. What kind of "scale" are you talking about? Do you have any numbers about such impact? > You're also assuming that we will never develop an AST optimizer No, the assumption is that we don't have such an optimizer *right now*. Having command-line options because they might be useful some day is silly. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 2:09 PM Benjamin Peterson wrote: > > > On Fri, Mar 6, 2015, at 13:34, Brett Cannon wrote: > > On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar > > wrote: > > > > > On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon wrote: > > > > > >> > > >> > > >> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: > > >> > > >>> > > >>> On 06/03/15 16:34, Brett Cannon wrote: > > >>> > Over on the import-sig I proposed eliminating the concept of .pyo > files > > >>> > since they only signify that /some/ optimization took place, not > > >>> > /what/ optimizations took place. Everyone on the SIG was positive > with > > >>> > the idea so I wrote a PEP, got positive feedback from the SIG > again, > > >>> and > > >>> > so now I present to you PEP 488 for discussion. > > >>> > > > >>> [snip] > > >>> > > >>> Historically -O and -OO have been the antithesis of optimisation, > they > > >>> change the behaviour of the program with no noticeable effect on > > >>> performance. > > >>> If a change is to be made, why not just drop .pyo files and be done > with > > >>> it? > > >>> > > >> > > >> I disagree with your premise that .pyo files don't have a noticeable > > >> effect on performance. If you don't use asserts a lot then there is no > > >> effect, but if you use them heavily or have them perform expensive > > >> calculations then there is an impact. And the dropping of docstrings > does > > >> have an impact on memory usage when you use Python at scale. > > >> > > >> You're also assuming that we will never develop an AST optimizer that > > >> will go beyond what the peepholer can do based on raw bytecode, or > > >> something that involves a bit of calculation and thus something you > > >> wouldn't want to do at startup. > > >> > > > > > > I don't want to speak for him, but you're going to get the best results > > > optimizing ASTs at runtime, which is what I thought he was suggesting. > > > Trying to optimize Python at compile time is setting your sights really > > > low. You have so little information then. > > > > > > > OK, I don't want to derail the discussion of the PEP into one over how > > best > > to optimize CPython's performance relative to bytecode vs. runtime like > > PyPy. The point is that we have -O and -OO and people do have uses for > > those flags. People can also do custom optimizations thanks to the > > flexibility of loaders. > > I think it would be preferable deprecate -O and -OO and replace them > with flags like --no-docstrings or --no-asserts. Ideally, "optimization" > levels shouldn't change program semantics. > OK, but that doesn't influence the PEP's goal of dropping .pyo files. Are you suggesting that the tag be changed to be less specific to optimizations and more free-form? Like `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff like this gets baked into the .pyc file itself instead of the file name, but I don't think we should just drop the ability to switch off asserts and docstrings like Mark seemed to be suggesting. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Sat, Mar 7, 2015 at 6:09 AM, Benjamin Peterson wrote: > I think it would be preferable deprecate -O and -OO and replace them > with flags like --no-docstrings or --no-asserts. Ideally, "optimization" > levels shouldn't change program semantics. Plenty of C compilers have optimization levels that can change behaviour (replacing division with mult-by-reciprocal and such), so I don't see any reason for Python to object. The removal of docstrings will be a problem to only a handful of programs (eg [1]) which use them for more than introspection. And the removal of assert shouldn't be a semantic change to a well-written program. [1] https://github.com/Rosuav/LetMeKnow/blob/master/letmeknow.py ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015, at 13:34, Brett Cannon wrote: > On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar > wrote: > > > On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon wrote: > > > >> > >> > >> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: > >> > >>> > >>> On 06/03/15 16:34, Brett Cannon wrote: > >>> > Over on the import-sig I proposed eliminating the concept of .pyo files > >>> > since they only signify that /some/ optimization took place, not > >>> > /what/ optimizations took place. Everyone on the SIG was positive with > >>> > the idea so I wrote a PEP, got positive feedback from the SIG again, > >>> and > >>> > so now I present to you PEP 488 for discussion. > >>> > > >>> [snip] > >>> > >>> Historically -O and -OO have been the antithesis of optimisation, they > >>> change the behaviour of the program with no noticeable effect on > >>> performance. > >>> If a change is to be made, why not just drop .pyo files and be done with > >>> it? > >>> > >> > >> I disagree with your premise that .pyo files don't have a noticeable > >> effect on performance. If you don't use asserts a lot then there is no > >> effect, but if you use them heavily or have them perform expensive > >> calculations then there is an impact. And the dropping of docstrings does > >> have an impact on memory usage when you use Python at scale. > >> > >> You're also assuming that we will never develop an AST optimizer that > >> will go beyond what the peepholer can do based on raw bytecode, or > >> something that involves a bit of calculation and thus something you > >> wouldn't want to do at startup. > >> > > > > I don't want to speak for him, but you're going to get the best results > > optimizing ASTs at runtime, which is what I thought he was suggesting. > > Trying to optimize Python at compile time is setting your sights really > > low. You have so little information then. > > > > OK, I don't want to derail the discussion of the PEP into one over how > best > to optimize CPython's performance relative to bytecode vs. runtime like > PyPy. The point is that we have -O and -OO and people do have uses for > those flags. People can also do custom optimizations thanks to the > flexibility of loaders. I think it would be preferable deprecate -O and -OO and replace them with flags like --no-docstrings or --no-asserts. Ideally, "optimization" levels shouldn't change program semantics. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
Thanks! All suggestions applied to my local copy. On Fri, Mar 6, 2015 at 1:55 PM Ethan Furman wrote: > On 03/06/2015 08:34 AM, Brett Cannon wrote: > > Over on the import-sig I proposed eliminating the concept of .pyo files > since they only signify that /some/ optimization > > took place, not /what/ optimizations took place. Everyone on the SIG was > positive with the idea so I wrote a PEP, got > > positive feedback from the SIG again, and so now I present to you PEP > 488 for discussion. > > +1 overall, comments in-line. > > > Implementation > > == > > > > importlib > > - > > > > As ``importlib.util.cache_from_source()`` is the API that exposes > > bytecode file paths as while as being directly used by importlib, it > > requires the most critical change. > > Not sure what that sentence is supposed to say -- maybe "as well as" and > not "as while as" ? > > > > The ``debug_override`` parameter will be deprecated. As the parameter > > expects a boolean, the integer value of the boolean will be used as > > if it had been provided as the argument to ``optimization`` (a > > ``None`` argument will mean the same as for ``optimization``). A > > deprecation warning will be raised when ``debug_override`` is given a > > value other than ``None``, but there are no plans for the complete > > removal of the parameter as this time (but removal will be no later > > than Python 4). > > "at this time" not "as this time" > > > > Rest of the standard library > > > > > > The various functions exposed by the ``py_compile`` and > > ``compileall`` functions will be updated as necessary to make sure > > they follow the new bytecode file name semantics [6]_, [1]_. The CLI > > for the ``compileall`` module will not be directly affected (the > > ``-b`` flag will be implicitly as it will no longer generate ``.pyo`` > > files when ``-O`` is specified). > > "will be implicit" not "will be implicitly" > > -- > ~Ethan~ > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On 03/06/2015 08:34 AM, Brett Cannon wrote: > Over on the import-sig I proposed eliminating the concept of .pyo files since > they only signify that /some/ optimization > took place, not /what/ optimizations took place. Everyone on the SIG was > positive with the idea so I wrote a PEP, got > positive feedback from the SIG again, and so now I present to you PEP 488 for > discussion. +1 overall, comments in-line. > Implementation > == > > importlib > - > > As ``importlib.util.cache_from_source()`` is the API that exposes > bytecode file paths as while as being directly used by importlib, it > requires the most critical change. Not sure what that sentence is supposed to say -- maybe "as well as" and not "as while as" ? > The ``debug_override`` parameter will be deprecated. As the parameter > expects a boolean, the integer value of the boolean will be used as > if it had been provided as the argument to ``optimization`` (a > ``None`` argument will mean the same as for ``optimization``). A > deprecation warning will be raised when ``debug_override`` is given a > value other than ``None``, but there are no plans for the complete > removal of the parameter as this time (but removal will be no later > than Python 4). "at this time" not "as this time" > Rest of the standard library > > > The various functions exposed by the ``py_compile`` and > ``compileall`` functions will be updated as necessary to make sure > they follow the new bytecode file name semantics [6]_, [1]_. The CLI > for the ``compileall`` module will not be directly affected (the > ``-b`` flag will be implicitly as it will no longer generate ``.pyo`` > files when ``-O`` is specified). "will be implicit" not "will be implicitly" -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar wrote: > On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon wrote: > >> >> >> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: >> >>> >>> On 06/03/15 16:34, Brett Cannon wrote: >>> > Over on the import-sig I proposed eliminating the concept of .pyo files >>> > since they only signify that /some/ optimization took place, not >>> > /what/ optimizations took place. Everyone on the SIG was positive with >>> > the idea so I wrote a PEP, got positive feedback from the SIG again, >>> and >>> > so now I present to you PEP 488 for discussion. >>> > >>> [snip] >>> >>> Historically -O and -OO have been the antithesis of optimisation, they >>> change the behaviour of the program with no noticeable effect on >>> performance. >>> If a change is to be made, why not just drop .pyo files and be done with >>> it? >>> >> >> I disagree with your premise that .pyo files don't have a noticeable >> effect on performance. If you don't use asserts a lot then there is no >> effect, but if you use them heavily or have them perform expensive >> calculations then there is an impact. And the dropping of docstrings does >> have an impact on memory usage when you use Python at scale. >> >> You're also assuming that we will never develop an AST optimizer that >> will go beyond what the peepholer can do based on raw bytecode, or >> something that involves a bit of calculation and thus something you >> wouldn't want to do at startup. >> > > I don't want to speak for him, but you're going to get the best results > optimizing ASTs at runtime, which is what I thought he was suggesting. > Trying to optimize Python at compile time is setting your sights really > low. You have so little information then. > OK, I don't want to derail the discussion of the PEP into one over how best to optimize CPython's performance relative to bytecode vs. runtime like PyPy. The point is that we have -O and -OO and people do have uses for those flags. People can also do custom optimizations thanks to the flexibility of loaders. All of this leads to wanting different bytecode files for different optimization levels to make sure you're actually executing your code with the optimizations you expect. If people think that optimizing code and surfacing it in bytecode files is a waste and want to suggest either dropping .pyo files entirely or dropping -O and only having -OO that's fine, but that is not what this PEP is proposing nor a PEP I want to bother writing. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon wrote: > > > On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: > >> >> On 06/03/15 16:34, Brett Cannon wrote: >> > Over on the import-sig I proposed eliminating the concept of .pyo files >> > since they only signify that /some/ optimization took place, not >> > /what/ optimizations took place. Everyone on the SIG was positive with >> > the idea so I wrote a PEP, got positive feedback from the SIG again, and >> > so now I present to you PEP 488 for discussion. >> > >> [snip] >> >> Historically -O and -OO have been the antithesis of optimisation, they >> change the behaviour of the program with no noticeable effect on >> performance. >> If a change is to be made, why not just drop .pyo files and be done with >> it? >> > > I disagree with your premise that .pyo files don't have a noticeable > effect on performance. If you don't use asserts a lot then there is no > effect, but if you use them heavily or have them perform expensive > calculations then there is an impact. And the dropping of docstrings does > have an impact on memory usage when you use Python at scale. > > You're also assuming that we will never develop an AST optimizer that will > go beyond what the peepholer can do based on raw bytecode, or something > that involves a bit of calculation and thus something you wouldn't want to > do at startup. > I don't want to speak for him, but you're going to get the best results optimizing ASTs at runtime, which is what I thought he was suggesting. Trying to optimize Python at compile time is setting your sights really low. You have so little information then. > > >> >> Any worthwhile optimisation needs to be done at runtime or involve much >> more than tweaking bytecode. >> > > I disagree again. If you do something like whole program analysis and want > to use that to optimize something, you will surface that through bytecode > and not editing the source. So while you are doing "much more than tweaking > bytecode" externally to Python, you still have to surface to the > interpreter through bytecode. > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: > > On 06/03/15 16:34, Brett Cannon wrote: > > Over on the import-sig I proposed eliminating the concept of .pyo files > > since they only signify that /some/ optimization took place, not > > /what/ optimizations took place. Everyone on the SIG was positive with > > the idea so I wrote a PEP, got positive feedback from the SIG again, and > > so now I present to you PEP 488 for discussion. > > > [snip] > > Historically -O and -OO have been the antithesis of optimisation, they > change the behaviour of the program with no noticeable effect on > performance. > If a change is to be made, why not just drop .pyo files and be done with > it? > I disagree with your premise that .pyo files don't have a noticeable effect on performance. If you don't use asserts a lot then there is no effect, but if you use them heavily or have them perform expensive calculations then there is an impact. And the dropping of docstrings does have an impact on memory usage when you use Python at scale. You're also assuming that we will never develop an AST optimizer that will go beyond what the peepholer can do based on raw bytecode, or something that involves a bit of calculation and thus something you wouldn't want to do at startup. > > Any worthwhile optimisation needs to be done at runtime or involve much > more than tweaking bytecode. > I disagree again. If you do something like whole program analysis and want to use that to optimize something, you will surface that through bytecode and not editing the source. So while you are doing "much more than tweaking bytecode" externally to Python, you still have to surface to the interpreter through bytecode. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On 06/03/15 16:34, Brett Cannon wrote: Over on the import-sig I proposed eliminating the concept of .pyo files since they only signify that /some/ optimization took place, not /what/ optimizations took place. Everyone on the SIG was positive with the idea so I wrote a PEP, got positive feedback from the SIG again, and so now I present to you PEP 488 for discussion. [snip] Historically -O and -OO have been the antithesis of optimisation, they change the behaviour of the program with no noticeable effect on performance. If a change is to be made, why not just drop .pyo files and be done with it? Any worthwhile optimisation needs to be done at runtime or involve much more than tweaking bytecode. Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 9:34 AM, Brett Cannon wrote: > Not specifying the optimization level when it is at 0 > - > > It has been suggested that for the common case of when the > optimizations are at level 0 that the entire part of the file name > relating to the optimization level be left out. This would allow for > file names of ``.pyc`` files to go unchanged, potentially leading to > less backwards-compatibility issues (although Python 3.5 introduces a > new magic number for bytecode so all bytecode files will have to be > regenerated regardless of the outcome of this PEP). > > It would also allow a potentially redundant bit of information to be > left out of the file name if an implementation of Python did not > allow for optimizing bytecode. This would only occur, though, if the > interpreter didn't support ``-O`` **and** didn't implement the ast > module, else users could implement their own optimizations. The presence of the "opt-0" part in the filename could imply that there are other supported optimization levels, even when there aren't. So leaving that out when optimizations aren't supported may be a good idea. Perhaps add sys.implementation.supports_optimization or something like that? Then only leave "opt-0" off if the implementation does not support any optimization. --eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
