[Python-Dev] Re: RFC: PEP 608: Coordinated Python release
I think it is more important to have CI that clearly shows the impact of dev versions of the interpreter and core packages. Some of us in the Nixpkgs community had this idea for Python core packages as well (and potentially scientific computing packages, but that's out of scope here). This would need funding though https://discuss.python.org/t/funding-for-ci-for-integrating-development-versions-of-packages-and-interpreter-in-package-set/2540 . On Fri, Oct 25, 2019 at 4:29 PM Victor Stinner wrote: > Hi, > > I just posted a new PEP for comments, please reply there, rather than by > email: > https://discuss.python.org/t/rfc-pep-608-coordinated-python-release/2539 > > PEP 608: Coordinated Python release > https://www.python.org/dev/peps/pep-0608/ > > Abstract: > > Block a Python release until a compatible version of selected projects > is available. > > The Python release manager can decide to release Python even if a > project is not compatible, if they decide that the project is going to > be fixed soon enough, or if the issue severity is low enough. > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/C3KM3LZISKZOXA62OA5NSXREQ2MPDRON/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/H3C6PTOJBAZG5OGOFC6IRQO6B4OPQFLF/ Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-Dev] PEP 552: deterministic pycs
> The main objection to that model is that it requires modifying source timestamps, which isn't possible for builds on read-only source trees. Why not set the source timestamps of the source trees to say 1 first? That's what is done with the Nix package manager. The Python interpreter is patched (mostly similar to the referred PR) and checks whether SOURCE_DATE_EPOCH is set, and if so, sets the mtime to 1. On Thu, Sep 7, 2017 at 11:08 PM, Benjamin Peterson wrote: > > > On Thu, Sep 7, 2017, at 14:00, Antoine Pitrou wrote: > > On Thu, 07 Sep 2017 13:39:21 -0700 > > Benjamin Peterson wrote: > > > Hello, > > > I've written a short PEP about an import extension to allow pycs to be > > > more deterministic by optional replacing the timestamp with a hash of > > > the source file: https://www.python.org/dev/peps/pep-0552/ > > > > Why isn't https://github.com/python/cpython/pull/296 a good enough > > solution to this problem? It has a simple implementation, and requires > > neither maintaining two different pyc formats nor reading the entire > > source file to check whether the pyc file is up to date. > > The main objection to that model is that it requires modifying source > timestamps, which isn't possible for builds on read-only source trees. > This proposal also allows reproducible builds even if the files are > being modified in an edit-run-tests cycle. > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > freddyrietdijk%40fridh.nl > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snap Python for simple distribution across multiple Linux distros
In my opinion the trend of creating these standalone packages is pretty bad. Developers that choose to support these kind of packages become responsible not only for the code they distribute, but also for whether the dependencies that are bundled in such a standalone package are up to date and secure. What libraries would have to be bundled? I guess that depends on how standalone you want to get. In Nixpkgs we get to 130 MB for Python 3.5, if it would bundle all its dependencies (glibc, openssl, ncurses, some X11 libs, ...). If I am correct you can assume certain libraries are available on the host system so you could skip bundling them. Furthermore, there's now at least three competing formats: Snappy, Flatpak and AppImage. On Tue, May 16, 2017 at 12:31 PM, Martin Wimpress < martin.wimpr...@canonical.com> wrote: > Hi all, > > I work at Canonical as part of the engineering team developing Ubuntu > and Snapcraft [1] and I'm a long time Python fan :-) > > We've created snaps, a platform that enables projects to directly > control delivery of software updates to users. This video of a > lightning talk by dlang developers at DConf2017 [2] shows how they've > made good use of snaps to distribute their compiler. They found the > release channels particularly useful so their users can track a > specific release. > > Is there someone here who'd be interested in doing the same for Python? > > [1] https://snapcraft.io/ > [2] https://www.youtube.com/watch?v=M-bDzr4gYUU > [3] https://snapcraft.io/docs/core/install > [4] https://build.snapcraft.io/ > > -- > Regards, Martin. > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > freddyrietdijk%40fridh.nl > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash randomization and deterministic bytecode
Thanks for the clarification. I'm glad we can keep hash randomization disabled during our builds. On Fri, May 12, 2017 at 5:22 PM, Guido van Rossum wrote: > Don't worry, the PYTHONHASHSEED setting does not get recorded in the > bytecode header and the generated bytecode (even if it sometimes differs in > trivial ways) is usable with all hash seed settings. > > --Guido > > On Fri, May 12, 2017 at 6:06 AM, Freddy Rietdijk > wrote: > >> Hi, >> >> On Nix we set PYTHONHASHSEED to 0 when building packages, disabling hash >> randomization. We do this to improve determinism of the builds because we >> store the bytecode next to the code. >> >> When one runs Python directly or via a script PYTHONHASHSEED is not set >> thus enabling hash randomization. Am I correct when I say that in this case >> Python still uses the reproducibly build bytecode and, because its now >> running with a random seed we wouldn't be vulnerable to >> http://www.ocert.org/advisories/ocert-2011-003.html ? Or would it also >> try to each time also recompile bytecode? >> >> Kind regards, >> >> Freddy >> >> >> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% >> 40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Hash randomization and deterministic bytecode
Hi, On Nix we set PYTHONHASHSEED to 0 when building packages, disabling hash randomization. We do this to improve determinism of the builds because we store the bytecode next to the code. When one runs Python directly or via a script PYTHONHASHSEED is not set thus enabling hash randomization. Am I correct when I say that in this case Python still uses the reproducibly build bytecode and, because its now running with a random seed we wouldn't be vulnerable to http://www.ocert.org/advisories/ocert-2011-003.html ? Or would it also try to each time also recompile bytecode? Kind regards, Freddy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Set program name through exec -a or environment variable
Hi, I would like to know if you're open to supporting `exec -a` or an environment variable for setting `argv[0]`, and have some pointers as to where that should be implemented. On Nixpkgs we typically use wrappers to set environment variables like PATH or PYTHONPATH for individual programs. Consider a program named `prog`. We move the original program `prog` to `.prog-wrapped` and then create a wrapper `prog` that does `exec -a prog .prog-wrapped`. Unfortunately `exec -a` does not work with Python. The process is still named `.prog-wrapped` (although that's not really a problem) but worse, `sys.argv[0]` is also `.prog-wrapped`. Currently we inject some code in programs that sets `sys.argv=[0] = "prog" but this is fragile and I would prefer to get rid of this. Kind regards, Frederik ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deterministic builds of the interpreter
Hi Armin, Thank you for your explanation. I've now managed to build 2.7 and 3.5 deterministic by recompiling the bytecode at the end of the build (and excluding 2to3). Freddy On Sun, Feb 19, 2017 at 9:30 AM, Armin Rigo wrote: > Hi Freddy, > > On 16 February 2017 at 18:03, Freddy Rietdijk > wrote: > > As I mentioned, it seems only sets cause unreproducible > > bytecode. Sets have no order. But when generating the bytecode, I would > > expect there would still be an order since the code isn't actually > executed, > > right? > > No, the sets are built as real sets and then marshalled to .pyc files > in a separate step. So on CPython an essentially random order will > end up in the .pyc file. Even CPython 3.6 gives a deterministic order > to dictionaries but not sets. You could ensure sets are marshalled in > a known order by changing the marshalling code, e.g. to emit them in > sorted order (on Python 2.x; on 3.x it is more messy because different > types are more often non-comparable). > > > A bientôt, > > Armin. > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deterministic builds of the interpreter
Hi, Are there anymore suggestions how to improve the determinism of the Python 3 interpreter? As I mentioned, it seems only sets cause unreproducible bytecode. Sets have no order. But when generating the bytecode, I would expect there would still be an order since the code isn't actually executed, right? On Fri, Feb 10, 2017 at 12:03 PM, INADA Naoki wrote: > On Fri, Feb 10, 2017 at 7:58 PM, Freddy Rietdijk > wrote: > > For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items > > still seem indeterministic. > > To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f > $out` > > where $out is the path where I installed Python. > > > > Do you have an idea why in [3], this is Python 2.7, the timestamps are > still > > incorrect? I think they're all required for `compileall` and somehow it > > doesn't seem capable of taking into account DETERMINISTIC_BUILD. > Explicitly > > removing those pyc and pyo files and recompiling them to bytecode still > > results in timestamp issues for these 4 files. > > Sorry, I have no motivation about Python 2 anymore. > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deterministic builds of the interpreter
That should have been `PYTHONHASHSEED=0 $out/bin/python -m compileall -f $out`. On Fri, Feb 10, 2017 at 11:58 AM, Freddy Rietdijk wrote: > For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items > still seem indeterministic. > To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f $out` > where $out is the path where I installed Python. > > Do you have an idea why in [3], this is Python 2.7, the timestamps are > still incorrect? I think they're all required for `compileall` and somehow > it doesn't seem capable of taking into account DETERMINISTIC_BUILD. > Explicitly removing those pyc and pyo files and recompiling them to > bytecode still results in timestamp issues for these 4 files. > > On Thu, Feb 9, 2017 at 6:51 PM, INADA Naoki > wrote: > >> On Fri, Feb 10, 2017 at 2:45 AM, Freddy Rietdijk >> wrote: >> > Correct, that was an older version from before I patched >> > `_bootstrap_external.py`. A more recent diff can be found at >> > >> > https://gist.github.com/anonymous/d40f24fd6b636ba40d345ff3f12a0aaa >> > >> > These all seem to be sets. >> >> Maybe, PYTHONHASHSEED help you. >> https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHASHSEED >> >> > >> > On Thu, Feb 9, 2017 at 6:04 PM, INADA Naoki >> wrote: >> >> >> >> As reading [4], mtime is not 0. >> >> >> >> data = bytearray(MAGIC_NUMBER) >> >> data.extend(_w_long(mtime)) >> >> data.extend(_w_long(source_size)) >> >> data.extend(marshal.dumps(code)) >> >> >> >> First 4 bytes are magic. >> >> Next 4 bytes are mtime. >> >> >> >> │ │ │ │ -: 160d 0d0a 6b2e 9c58 6c21 e300 >> >> k..Xl!.. >> >> │ │ │ │ +: 160d 0d0a e631 9c58 6c21 e300 >> >> .1.Xl!.. >> >> >> >> mtime is 6b2e9c58 vs e6319c53 (little endian) >> >> >> >> maybe, you failed to use customized py_compile when building? >> >> >> >> >> >> On Thu, Feb 9, 2017 at 6:27 PM, Freddy Rietdijk < >> freddyrietd...@fridh.nl> >> >> wrote: >> >> > Hi, >> >> > >> >> > I'm attempting to make the builds of the Python interpreters for >> Nixpkgs >> >> > [1] >> >> > deterministic. >> >> > >> >> > In the case of Python 2.7 we have a patch [2] that fixes the >> timestamp >> >> > used >> >> > in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We >> also >> >> > remove `wininst*.exe`. This works fine, although there are 4 small >> >> > issues >> >> > left [3]. Do you have any idea what is going on in these files that >> >> > could >> >> > make them indeterministic? >> >> > >> >> > For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and >> >> > modified >> >> > `py_compile` to use `0` instead of `source_stats['mtime']`. The >> builds >> >> > are >> >> > not yet deterministic [4]. Any suggestions what could be fixed here? >> >> > >> >> > Kind regards, >> >> > >> >> > Freddy >> >> > >> >> > >> >> > [1] https://github.com/NixOS/nixpkgs >> >> > [2] >> >> > >> >> > https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/developme >> nt/interpreters/python/cpython/2.7/deterministic-build.patch >> >> > [3] https://github.com/NixOS/nixpkgs/issues/22570#issuecomment- >> 278474082 >> >> > [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d11 >> 0f0e6b >> >> > >> >> > ___ >> >> > Python-Dev mailing list >> >> > Python-Dev@python.org >> >> > https://mail.python.org/mailman/listinfo/python-dev >> >> > Unsubscribe: >> >> > >> >> > https://mail.python.org/mailman/options/python-dev/songofaca >> ndy%40gmail.com >> >> > >> > >> > >> > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deterministic builds of the interpreter
For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items still seem indeterministic. To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f $out` where $out is the path where I installed Python. Do you have an idea why in [3], this is Python 2.7, the timestamps are still incorrect? I think they're all required for `compileall` and somehow it doesn't seem capable of taking into account DETERMINISTIC_BUILD. Explicitly removing those pyc and pyo files and recompiling them to bytecode still results in timestamp issues for these 4 files. On Thu, Feb 9, 2017 at 6:51 PM, INADA Naoki wrote: > On Fri, Feb 10, 2017 at 2:45 AM, Freddy Rietdijk > wrote: > > Correct, that was an older version from before I patched > > `_bootstrap_external.py`. A more recent diff can be found at > > > > https://gist.github.com/anonymous/d40f24fd6b636ba40d345ff3f12a0aaa > > > > These all seem to be sets. > > Maybe, PYTHONHASHSEED help you. > https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHASHSEED > > > > > On Thu, Feb 9, 2017 at 6:04 PM, INADA Naoki > wrote: > >> > >> As reading [4], mtime is not 0. > >> > >> data = bytearray(MAGIC_NUMBER) > >> data.extend(_w_long(mtime)) > >> data.extend(_w_long(source_size)) > >> data.extend(marshal.dumps(code)) > >> > >> First 4 bytes are magic. > >> Next 4 bytes are mtime. > >> > >> │ │ │ │ -: 160d 0d0a 6b2e 9c58 6c21 e300 > >> k..Xl!.. > >> │ │ │ │ +: 160d 0d0a e631 9c58 6c21 0000 e300 > >> .1.Xl!.. > >> > >> mtime is 6b2e9c58 vs e6319c53 (little endian) > >> > >> maybe, you failed to use customized py_compile when building? > >> > >> > >> On Thu, Feb 9, 2017 at 6:27 PM, Freddy Rietdijk < > freddyrietd...@fridh.nl> > >> wrote: > >> > Hi, > >> > > >> > I'm attempting to make the builds of the Python interpreters for > Nixpkgs > >> > [1] > >> > deterministic. > >> > > >> > In the case of Python 2.7 we have a patch [2] that fixes the timestamp > >> > used > >> > in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We > also > >> > remove `wininst*.exe`. This works fine, although there are 4 small > >> > issues > >> > left [3]. Do you have any idea what is going on in these files that > >> > could > >> > make them indeterministic? > >> > > >> > For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and > >> > modified > >> > `py_compile` to use `0` instead of `source_stats['mtime']`. The builds > >> > are > >> > not yet deterministic [4]. Any suggestions what could be fixed here? > >> > > >> > Kind regards, > >> > > >> > Freddy > >> > > >> > > >> > [1] https://github.com/NixOS/nixpkgs > >> > [2] > >> > > >> > https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/ > development/interpreters/python/cpython/2.7/deterministic-build.patch > >> > [3] https://github.com/NixOS/nixpkgs/issues/22570# > issuecomment-278474082 > >> > [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d110f0e > 6b > >> > > >> > ___ > >> > Python-Dev mailing list > >> > Python-Dev@python.org > >> > https://mail.python.org/mailman/listinfo/python-dev > >> > Unsubscribe: > >> > > >> > https://mail.python.org/mailman/options/python-dev/ > songofacandy%40gmail.com > >> > > > > > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deterministic builds of the interpreter
Correct, that was an older version from before I patched `_bootstrap_external.py`. A more recent diff can be found at https://gist.github.com/anonymous/d40f24fd6b636ba40d345ff3f12a0aaa These all seem to be sets. On Thu, Feb 9, 2017 at 6:04 PM, INADA Naoki wrote: > As reading [4], mtime is not 0. > > data = bytearray(MAGIC_NUMBER) > data.extend(_w_long(mtime)) > data.extend(_w_long(source_size)) > data.extend(marshal.dumps(code)) > > First 4 bytes are magic. > Next 4 bytes are mtime. > > │ │ │ │ -: 160d 0d0a 6b2e 9c58 6c21 e300 k..Xl!.. > │ │ │ │ +: 160d 0d0a e631 9c58 6c21 e300 .1.Xl!.. > > mtime is 6b2e9c58 vs e6319c53 (little endian) > > maybe, you failed to use customized py_compile when building? > > > On Thu, Feb 9, 2017 at 6:27 PM, Freddy Rietdijk > wrote: > > Hi, > > > > I'm attempting to make the builds of the Python interpreters for Nixpkgs > [1] > > deterministic. > > > > In the case of Python 2.7 we have a patch [2] that fixes the timestamp > used > > in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We also > > remove `wininst*.exe`. This works fine, although there are 4 small issues > > left [3]. Do you have any idea what is going on in these files that could > > make them indeterministic? > > > > For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and modified > > `py_compile` to use `0` instead of `source_stats['mtime']`. The builds > are > > not yet deterministic [4]. Any suggestions what could be fixed here? > > > > Kind regards, > > > > Freddy > > > > > > [1] https://github.com/NixOS/nixpkgs > > [2] > > https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/ > development/interpreters/python/cpython/2.7/deterministic-build.patch > > [3] https://github.com/NixOS/nixpkgs/issues/22570#issuecomment-278474082 > > [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d110f0e6b > > > > ___ > > Python-Dev mailing list > > Python-Dev@python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/ > songofacandy%40gmail.com > > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Deterministic builds of the interpreter
Hi, I'm attempting to make the builds of the Python interpreters for Nixpkgs [1] deterministic. In the case of Python 2.7 we have a patch [2] that fixes the timestamp used in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We also remove `wininst*.exe`. This works fine, although there are 4 small issues left [3]. Do you have any idea what is going on in these files that could make them indeterministic? For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and modified `py_compile` to use `0` instead of `source_stats['mtime']`. The builds are not yet deterministic [4]. Any suggestions what could be fixed here? Kind regards, Freddy [1] https://github.com/NixOS/nixpkgs [2] https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/development/interpreters/python/cpython/2.7/deterministic-build.patch [3] https://github.com/NixOS/nixpkgs/issues/22570#issuecomment-278474082 [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d110f0e6b ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
