On Sat, 14 Apr 2018 at 17:01 Neil Schemenauer
wrote:
> On 2018-04-12, M.-A. Lemburg wrote:
> > This leaves the proposal to restructure pyc files into a sectioned
> > file and possibly indexed file to make access to (lazily) loaded
> > parts faster.
>
> I would like to see a format can hold one or
On 2018-04-12, M.-A. Lemburg wrote:
> This leaves the proposal to restructure pyc files into a sectioned
> file and possibly indexed file to make access to (lazily) loaded
> parts faster.
I would like to see a format can hold one or more modules in a
single file. Something like the zip format but
2018-04-11 2:03 GMT+02:00 Steven D'Aprano :
[snip]
> I shouldn't think that the number of files on disk is very important,
> now that they're hidden away in the __pycache__ directory where they can
> be ignored by humans. Even venerable old FAT32 has a limit of 65,534
> files in a single folder,
On Fri, 13 Apr 2018 at 03:47, M.-A. Lemburg wrote:
> I think moving data out of pyc files is going in a wrong direction:
> more stat calls means slower import and slower startup time.
>
> Trying to make pycs smaller also isn't really worth it (they
> compress quite well).
>
> Saving memory could
I think moving data out of pyc files is going in a wrong direction:
more stat calls means slower import and slower startup time.
Trying to make pycs smaller also isn't really worth it (they
compress quite well).
Saving memory could be done by disabling reading objects lazily
from the file - witho
I've been playing a bit with this trying to collect some data and measure
how useful this would be. You can take a look at the script I'm using at:
https://github.com/dmoisset/pycstats
What I'm measuring is:
1. Number of objects in the pyc, and how many of those are:
* docstrings (I'm using a h
One implementation difficulty specifically related to annotations, is that
they are quite hard to find/extract from the code objects. Both docstrings
and lnotab are within specific fields of the code object for their
function/class/module; annotations are spread as individual constants
(assuming PE
> Finally, loading docstrings and other optional components can be made lazy.
> This was not in my original idea, and this will significantly complicate the
> implementation, but in principle it is possible. This will require larger
> changes in the marshal format and bytecode.
I'm +1 on this idea
10.04.18 19:24, Antoine Pitrou пише:
2. Line numbers (lnotab). They are helpful for formatting tracebacks,
for tracing, and debugging with the debugger. Sources are helpful in
such cases too. If the program doesn't contain errors ;-) and is sipped
without sources, they could be removed.
What is
10.04.18 20:38, Chris Angelico пише:
On Wed, Apr 11, 2018 at 2:14 AM, Serhiy Storchaka wrote:
A deployed Python distribution generally has .pyc files for all of the
standard library. I don't think people want to lose the ability to
call help(), and unless I'm misunderstanding, that requires
docs
On Thu, Apr 12, 2018 at 11:59 AM, Steven D'Aprano wrote:
> On Thu, Apr 12, 2018 at 12:09:38AM +1000, Chris Angelico wrote:
>
> [...]
>> >> Consider a very common use-case: an OS-provided
>> >> Python interpreter whose files are all owned by 'root'. Those will be
>> >> distributed with .pyc files f
On Thu, Apr 12, 2018 at 12:09:38AM +1000, Chris Angelico wrote:
[...]
> >> Consider a very common use-case: an OS-provided
> >> Python interpreter whose files are all owned by 'root'. Those will be
> >> distributed with .pyc files for performance, but you don't want to
> >> deprive the users of he
On 4/11/2018 4:26 AM, Petr Viktorin wrote:
Currently in Fedora, we ship *both* optimized and non-optimized pycs to
make sure both -O and non--O will work nicely without root privilieges.
So splitting the docstrings into a separate file would be, for us, a
benefit in terms of file size.
Curre
On 11 April 2018 at 02:14, Serhiy Storchaka wrote:
> Currently pyc files contain data that is useful mostly for developing and is
> not needed in most normal cases in stable program. There is even an option
> that allows to exclude a part of this information from pyc files. It is
> expected that t
On Wed, Apr 11, 2018 at 4:06 PM, Steven D'Aprano wrote:
> On Wed, Apr 11, 2018 at 02:21:17PM +1000, Chris Angelico wrote:
>
> [...]
>> > Yes, it will double the number of files. Actually quadruple it, if the
>> > annotations and line numbers are in separate files too. But if most of
>> > those ext
On Tue, Apr 10, 2018 at 9:50 PM, Eric V. Smith wrote:
>
>>> 3. Annotations. They are used mainly by third party tools that
>>> statically analyze sources. They are rarely used at runtime.
>>
>> Even less used than docstrings probably.
>
> typing.NamedTuple and dataclasses use annotations at runtim
On 04/11/18 08:06, Steven D'Aprano wrote:
On Wed, Apr 11, 2018 at 02:21:17PM +1000, Chris Angelico wrote:
[...]
Yes, it will double the number of files. Actually quadruple it, if the
annotations and line numbers are in separate files too. But if most of
those extra files never need to be open
On 04/11/18 06:21, Chris Angelico wrote:
On Wed, Apr 11, 2018 at 1:02 PM, Steven D'Aprano wrote:
On Wed, Apr 11, 2018 at 10:08:58AM +1000, Chris Angelico wrote:
File system limits aren't usually an issue; as you say, even FAT32 can
store a metric ton of files in a single directory. I'm more i
On 10/04/2018 18:54, Zachary Ware wrote:
> On Tue, Apr 10, 2018 at 12:38 PM, Chris Angelico wrote:
>> A deployed Python distribution generally has .pyc files for all of the
>> standard library. I don't think people want to lose the ability to
>> call help(), and unless I'm misunderstanding, that
On Wed, Apr 11, 2018 at 02:21:17PM +1000, Chris Angelico wrote:
[...]
> > Yes, it will double the number of files. Actually quadruple it, if the
> > annotations and line numbers are in separate files too. But if most of
> > those extra files never need to be opened, then there's no cost to them.
>
On Wed, Apr 11, 2018 at 1:02 PM, Steven D'Aprano wrote:
> On Wed, Apr 11, 2018 at 10:08:58AM +1000, Chris Angelico wrote:
>
>> File system limits aren't usually an issue; as you say, even FAT32 can
>> store a metric ton of files in a single directory. I'm more interested
>> in how long it takes to
On Wed, Apr 11, 2018 at 10:08:58AM +1000, Chris Angelico wrote:
> File system limits aren't usually an issue; as you say, even FAT32 can
> store a metric ton of files in a single directory. I'm more interested
> in how long it takes to open a file, and whether doubling that time
> will have a meas
On Tue, Apr 10, 2018 at 5:03 PM, Steven D'Aprano
wrote:
>
> __pycache__/spam.cpython-38.pyc
> __pycache__/spam.cpython-38-doc.pyc
> __pycache__/spam.cpython-38-lno.pyc
> __pycache__/spam.cpython-38-ann.pyc
>
Our product uses the doc strings for auto-generated help, so we need to
On Tue, Apr 10, 2018 at 12:51 PM Eric V. Smith wrote:
>
> >> 3. Annotations. They are used mainly by third party tools that
> >> statically analyze sources. They are rarely used at runtime.
> >
> > Even less used than docstrings probably.
>
> typing.NamedTuple and dataclasses use annotations at r
On Wed, Apr 11, 2018 at 10:03 AM, Steven D'Aprano wrote:
> On Wed, Apr 11, 2018 at 03:38:08AM +1000, Chris Angelico wrote:
>> A deployed Python distribution generally has .pyc files for all of the
>> standard library. I don't think people want to lose the ability to
>> call help(), and unless I'm
On Wed, Apr 11, 2018 at 03:38:08AM +1000, Chris Angelico wrote:
> On Wed, Apr 11, 2018 at 2:14 AM, Serhiy Storchaka wrote:
> > Currently pyc files contain data that is useful mostly for developing and is
> > not needed in most normal cases in stable program. There is even an option
> > that allows
>> 3. Annotations. They are used mainly by third party tools that
>> statically analyze sources. They are rarely used at runtime.
>
> Even less used than docstrings probably.
typing.NamedTuple and dataclasses use annotations at runtime.
Eric
___
Pyt
On Tue, 10 Apr 2018 19:14:58 +0300
Serhiy Storchaka
wrote:
> Currently pyc files contain data that is useful mostly for developing
> and is not needed in most normal cases in stable program. There is even
> an option that allows to exclude a part of this information from pyc
> files. It is expe
On Tue, 10 Apr 2018 11:13:01 -0700
Ethan Furman wrote:
> On 04/10/2018 10:54 AM, Zachary Ware wrote:
> > On Tue, Apr 10, 2018 at 12:38 PM, Chris Angelico wrote:
> >> A deployed Python distribution generally has .pyc files for all of the
> >> standard library. I don't think people want to lose t
I'm not sure I understand the benefit of this, perhaps you can clarify.
What I see is two scenarios
Scenario A) External files are present
In this case, the data is loaded from the pyc and then from external file,
so there are no savings in memory, startup time, disk space, or network
load time,
There are libraries out there like this:
https://docopt.readthedocs.io/en/0.2.0/
which use docstrings for runtime info.
Today we already have -OO which allows us to create docstring-less bytecode
files in case we have, after careful consideration, established that it is
safe to do so.
I think t
On 04/10/2018 10:54 AM, Zachary Ware wrote:
On Tue, Apr 10, 2018 at 12:38 PM, Chris Angelico wrote:
A deployed Python distribution generally has .pyc files for all of the
standard library. I don't think people want to lose the ability to
call help(), and unless I'm misunderstanding, that requir
On Tue, Apr 10, 2018 at 12:38 PM, Chris Angelico wrote:
> A deployed Python distribution generally has .pyc files for all of the
> standard library. I don't think people want to lose the ability to
> call help(), and unless I'm misunderstanding, that requires
> docstrings. So this will mean twice
On Wed, Apr 11, 2018 at 2:14 AM, Serhiy Storchaka wrote:
> Currently pyc files contain data that is useful mostly for developing and is
> not needed in most normal cases in stable program. There is even an option
> that allows to exclude a part of this information from pyc files. It is
> expected
On Tue, 10 Apr 2018 19:14:58 +0300
Serhiy Storchaka
wrote:
> Currently pyc files contain data that is useful mostly for developing
> and is not needed in most normal cases in stable program. There is even
> an option that allows to exclude a part of this information from pyc
> files. It is expe
Currently pyc files contain data that is useful mostly for developing
and is not needed in most normal cases in stable program. There is even
an option that allows to exclude a part of this information from pyc
files. It is expected that this saves memory, startup time, and disk
space (or the t
36 matches
Mail list logo
