Thanks Greg for the great, detailed response I think I understand now better your proposal and I think is a good idea and I would like to explore that. I have some questions:
* One problem I see is that that will make the constructor of the code object depend on global options in the interpreter. Someone using the C-API and passing down that attribute will be surprised to find that it was modified by a global. I am not saying is bad but I can see some problems with that. * The alternative is to modify all calls to the code object constructor. This is easy to do in the compiler because code objects are constructed very close where the meta data is crated but is going to be a pain in other places, because the code objects are constructed in places where we would either need new APIs or to hide global state as the previous point. * Another alternative is to walk the graph and strip the fields but that's going to have a performance impact. I think that if we decide to offer an opt out, this is actually one of the best options but I am still slightly concerned about the extra complexity, potential new APIs and maintainability. On Sat, 8 May 2021, 22:55 Gregory P. Smith, <g...@krypto.org> wrote: > > > On Sat, May 8, 2021 at 2:09 PM Pablo Galindo Salgado <pablog...@gmail.com> > wrote: > >> > Why not put in it -O instead? Then -O means lose asserts and lose >> fine-grained tracebacks, while -OO continues to also >> strip out doc strings. >> >> What if someone wants to keep asserts but do not want the extra data? >> > > exactly my theme. our existing -O and -OO already don't serve all user > needs. (I've witnessed people who need asserts but don't want docstrings > wasting ram jump through hacky hoops to do that). Complicating these > options more by combining additional actions on them them doesn't help. > > The reason we have -O and -OO generate their own special opt-1 and opt-2 > pyc files is because both of those change the generated bytecode and > overall flow of the program by omitting instructions and data. code using > those will run slightly faster as there are fewer instructions. > > The change we're talking about here doesn't do that. It just adds > additional metadata to whatever instructions are generated. So it doesn't > feel -O related. > > While some people aren't going to like the overhead, I'm happy not > offering the choice. > > > Greg, what do you think if instead of not writing it to the pyc file > with -OO or adding a header entry to decide to read/write, we place None in > the field? That way we can > > leverage the option that we intend to add to deactivate displaying the > traceback new information to reduce the data in the pyc files. The only > problem > > is that there will be still a tiny bit of overhead: an extra object per > code object (None), but that's much much better than something that scales > with the > > number of instructions :) > > > > What's your opinion on this? > > I don't understand the pyc structure enough to comment on how that works, > but that sounds fine from a way to store less data if these are stored as a > side table rather than intermingled with each instruction itself. *If > anyone even cares about storing less data.* I would not activate > generation of that in py_compile and compileall based on the -X flag to > disable display of tracebacks though. A flag changing a setting of the > current runtime regarding traceback printing detail level should not change > the metadata in pyc files it emits. I realize -O and -OO behave this way, > but I don't view those as a great example. We're not writing new uniquely > named pyc files, I suggest making this an explicit option for py_compile > and compileall if we're going to support generation of pyc files without > column data at all. > > I'm unclear on what the specific goals are with all of these option > possibilities. > > Who non-hypothetically cares about a 22% pyc file size increase? I don't > think we should be concerned. I'm in favor of always writing them and the > 20% size increase that results in. If pyc size is an issue that should be > its own separate enhancement PEP. When it comes to pyc files there is more > data we may want to store in the future for performance reasons - I don't > see them shrinking without an independent effort. > > Caring about additional data retained in memory at runtime makes more > sense to me as ram cost is much greater than storage cost and is paid > repeatedly per process. Storing an additional reference to None on code > objects where a column information table is perfectly fine. That can be a > -X style interpreter startup option. It isn't something that needs to > impacted by the pyc files. Pass that option to the interpreter, and it > just discards column info tables on code objects after loading them or > compiling them. If people want to optimize for a shared pyc situation with > memory mapping techniques, that is also something that should be a separate > enhancement PEP and not involved here. People writing code to use the > column information should always check it for None first, that'd be > something we document with the new feature. > > -gps > > >> >> On Sat, 8 May 2021 at 22:05, Ethan Furman <et...@stoneleaf.us> wrote: >> >>> On 5/8/21 1:31 PM, Pablo Galindo Salgado wrote: >>> >> We can't piggy back on -OO as the only way to disable this, it needs >>> to >>> >> have an option of its own. -OO is unusable as code that relies on >>> "doc" >>> >> strings as application data such as >>> http://www.dabeaz.com/ply/ply.html >>> >> exists. >>> > >>> > -OO is the only sensible way to disable the data. There are two >>> things to disable: >>> > >>> > * The data in pyc files >>> > * Printing the exception highlighting >>> >>> Why not put in it -O instead? Then -O means lose asserts and lose >>> fine-grained tracebacks, while -OO continues to also >>> strip out doc strings. >>> >>> -- >>> ~Ethan~ >>> _______________________________________________ >>> Python-Dev mailing list -- python-dev@python.org >>> To unsubscribe send an email to python-dev-le...@python.org >>> https://mail.python.org/mailman3/lists/python-dev.python.org/ >>> Message archived at >>> https://mail.python.org/archives/list/python-dev@python.org/message/BEE4BGUZHXBTVDPOW5R4DC3S463XC3EJ/ >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> _______________________________________________ >> Python-Dev mailing list -- python-dev@python.org >> To unsubscribe send an email to python-dev-le...@python.org >> https://mail.python.org/mailman3/lists/python-dev.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-dev@python.org/message/WK4KXZPOSWYMI3C5AILQCEYVZRCDFL7N/ >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/37O2WY73BQMWMSSG6MOCMAZYVBPDBZ3H/ Code of Conduct: http://python.org/psf/codeofconduct/
