Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-20 Thread Brett Cannon
On Fri, 8 Sep 2017 at 10:53 Benjamin Peterson wrote: > Thank you all for the feedback. I've now updated the PEP to specify a > 4-word pyc header with a bit field in every case. > > On Fri, Sep 8, 2017, at 09:43, Nick Coghlan wrote: > > On 8 September 2017 at 07:55, Antoine Pitrou > wrote: > > >

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-08 Thread Benjamin Peterson
Thank you all for the feedback. I've now updated the PEP to specify a 4-word pyc header with a bit field in every case. On Fri, Sep 8, 2017, at 09:43, Nick Coghlan wrote: > On 8 September 2017 at 07:55, Antoine Pitrou wrote: > > On Fri, 8 Sep 2017 07:49:46 -0700 > > Nick Coghlan wrote: > >> > I'

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-08 Thread Nick Coghlan
On 8 September 2017 at 07:55, Antoine Pitrou wrote: > On Fri, 8 Sep 2017 07:49:46 -0700 > Nick Coghlan wrote: >> > I'd rather a single magic number and a separate bitfield that tells >> > what the header encodes exactly. We don't *have* to fight for a tiny >> > size reduction of pyc files. >> >>

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-08 Thread Antoine Pitrou
On Fri, 8 Sep 2017 07:49:46 -0700 Nick Coghlan wrote: > On 8 September 2017 at 03:04, Antoine Pitrou wrote: > > On Thu, 7 Sep 2017 18:47:20 -0700 > > Nick Coghlan wrote: > >> However, I do wonder whether we could encode *all* the mode settings > >> into the magic number, such that we did somet

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-08 Thread Nick Coghlan
On 8 September 2017 at 03:04, Antoine Pitrou wrote: > On Thu, 7 Sep 2017 18:47:20 -0700 > Nick Coghlan wrote: >> However, I do wonder whether we could encode *all* the mode settings >> into the magic number, such that we did something like reserving the >> top 3 bits for format flags: >> >> * num

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-08 Thread Antoine Pitrou
On Thu, 7 Sep 2017 18:47:20 -0700 Nick Coghlan wrote: > However, I do wonder whether we could encode *all* the mode settings > into the magic number, such that we did something like reserving the > top 3 bits for format flags: > > * number & 0x1FFF -> the traditional magic number > * number & 0x8

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Barry Warsaw
On Sep 7, 2017, at 16:58, Gregory P. Smith wrote: > Input from OS package distributors would be interesting. Would they use this? I suspect it won’t be that interesting to the Debian ecosystem, since we generate pyc files on package install. We do that because we can support multiple version

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Benjamin Peterson
On Thu, Sep 7, 2017, at 16:58, Gregory P. Smith wrote: > +1 on this PEP. Thanks! > Questions: > > Input from OS package distributors would be interesting. Would they use > this? Which way would it impact their startup time (loading the .py file > vs just statting it. does that even matter?

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Nick Coghlan
On 7 September 2017 at 16:58, Gregory P. Smith wrote: > +1 on this PEP. > > The TL;DR summary of this PEP: > The pyc date+length metadata check was a convenient hack. It still works > well for many people and use cases, it isn't going away. > PEP 552 proposes a new alternate hack that relies

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Gregory P. Smith
+1 on this PEP. The TL;DR summary of this PEP: The pyc date+length metadata check was a convenient hack. It still works well for many people and use cases, it isn't going away. PEP 552 proposes a new alternate hack that relies on file contents instead of os and filesystem date metadata. A

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Benjamin Peterson
On Thu, Sep 7, 2017, at 14:43, Guido van Rossum wrote: > On Thu, Sep 7, 2017 at 2:40 PM, Benjamin Peterson > wrote: > > > > > > > On Thu, Sep 7, 2017, at 14:19, Guido van Rossum wrote: > > > Nice one. > > > > > > It would be nice to specify the various APIs needed as well. > > > > The compileal

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Benjamin Peterson
On Thu, Sep 7, 2017, at 14:54, Antoine Pitrou wrote: > On Thu, 07 Sep 2017 14:32:19 -0700 > Benjamin Peterson wrote: > > > > > > Not sure how common that situation is (certainly the source tree wasn't > > > read-only when you checked it out or untar'ed it), but isn't it easily > > > circumvente

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Antoine Pitrou
On Thu, 07 Sep 2017 14:40:33 -0700 Benjamin Peterson wrote: > On Thu, Sep 7, 2017, at 14:19, Guido van Rossum wrote: > > Nice one. > > > > It would be nice to specify the various APIs needed as well. > > The compileall and py_compile ones? > > > > > Why do you keep the mtime-based format as

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Antoine Pitrou
On Thu, 07 Sep 2017 14:32:19 -0700 Benjamin Peterson wrote: > > > > Not sure how common that situation is (certainly the source tree wasn't > > read-only when you checked it out or untar'ed it), but isn't it easily > > circumvented by copying the source tree before building? > > Well, yes, in

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Guido van Rossum
On Thu, Sep 7, 2017 at 2:40 PM, Benjamin Peterson wrote: > > > On Thu, Sep 7, 2017, at 14:19, Guido van Rossum wrote: > > Nice one. > > > > It would be nice to specify the various APIs needed as well. > > The compileall and py_compile ones? > Yes, and the SipHash mod to specify the key you menti

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Benjamin Peterson
On Thu, Sep 7, 2017, at 14:19, Guido van Rossum wrote: > Nice one. > > It would be nice to specify the various APIs needed as well. The compileall and py_compile ones? > > Why do you keep the mtime-based format as an option? (Maybe because it's > faster? Did you measure it?) I haven't actual

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Benjamin Peterson
On Thu, Sep 7, 2017, at 14:21, Antoine Pitrou wrote: > On Thu, 07 Sep 2017 14:08:58 -0700 > Benjamin Peterson wrote: > > On Thu, Sep 7, 2017, at 14:00, Antoine Pitrou wrote: > > > On Thu, 07 Sep 2017 13:39:21 -0700 > > > Benjamin Peterson wrote: > > > > Hello, > > > > I've written a short PEP

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Benjamin Peterson
On Thu, Sep 7, 2017, at 14:19, Freddy Rietdijk wrote: > > The main objection to that model is that it requires modifying source > timestamps, which isn't possible for builds on read-only source trees. > > Why not set the source timestamps of the source trees to say 1 first? If the source-tree i

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Antoine Pitrou
On Thu, 07 Sep 2017 14:08:58 -0700 Benjamin Peterson wrote: > On Thu, Sep 7, 2017, at 14:00, Antoine Pitrou wrote: > > On Thu, 07 Sep 2017 13:39:21 -0700 > > Benjamin Peterson wrote: > > > Hello, > > > I've written a short PEP about an import extension to allow pycs to be > > > more determinist

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Guido van Rossum
Nice one. It would be nice to specify the various APIs needed as well. Why do you keep the mtime-based format as an option? (Maybe because it's faster? Did you measure it?) On Thu, Sep 7, 2017 at 1:39 PM, Benjamin Peterson wrote: > Hello, > I've written a short PEP about an import extension t

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Freddy Rietdijk
> The main objection to that model is that it requires modifying source timestamps, which isn't possible for builds on read-only source trees. Why not set the source timestamps of the source trees to say 1 first? That's what is done with the Nix package manager. The Python interpreter is patched (

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Benjamin Peterson
On Thu, Sep 7, 2017, at 14:00, Antoine Pitrou wrote: > On Thu, 07 Sep 2017 13:39:21 -0700 > Benjamin Peterson wrote: > > Hello, > > I've written a short PEP about an import extension to allow pycs to be > > more deterministic by optional replacing the timestamp with a hash of > > the source file

Re: [Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Antoine Pitrou
On Thu, 07 Sep 2017 13:39:21 -0700 Benjamin Peterson wrote: > Hello, > I've written a short PEP about an import extension to allow pycs to be > more deterministic by optional replacing the timestamp with a hash of > the source file: https://www.python.org/dev/peps/pep-0552/ Why isn't https://gith

[Python-Dev] PEP 552: deterministic pycs

2017-09-07 Thread Benjamin Peterson
Hello, I've written a short PEP about an import extension to allow pycs to be more deterministic by optional replacing the timestamp with a hash of the source file: https://www.python.org/dev/peps/pep-0552/ Thanks for reading, Benjamin P.S. I came up with the idea for this PEP while awake. __