[issue29708] support reproducible Python builds

Felix C. Stegerman added the comment: Hi! I've been working on reproducible builds for python-for-android [1,2,3]. Current issues with .pyc files are: * .pyc files differ depending on whether Python was compiled w/ liblzma-dev installed or not; * many .pyc files include build paths; * some

[issue29708] support reproducible Python builds

Change by Felix C. Stegerman : -- nosy: +obfusk ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue29708] support reproducible Python builds

Steve Dower added the comment: This doesn't seem to necessarily impact distutils, so I'm leaving it open despite PEP 632. -- components: -Distutils dependencies: -Reproducible pyc: FLAG_REF is not stable., Reproducible pyc: frozenset is not serialized in a deterministic order

[issue29708] support reproducible Python builds

Change by Brett Cannon : -- nosy: -brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue29708] support reproducible Python builds

Benjamin Peterson added the comment: PEP 552 was a necessary but not sufficient step on the road towards fully deterministic pycs. The PEP says: "(Note there are other problems [1] [2] we do not address here that can make pycs non-deterministic.)" where [1] and [2] are basically the issues

[issue29708] support reproducible Python builds

STINNER Victor added the comment: > note the optimized .pyc is deterministic. As far as I know only __debug__ is > set to False, or is there something else different? Hum, maybe there is a misunderstanding on the PEP 552 purpose. I understood that the main point of the PEP 552 is to compare

[issue29708] support reproducible Python builds

Change by Inada Naoki : -- dependencies: +Reproducible pyc: FLAG_REF is not stable. ___ Python tracker ___ ___ Python-bugs-list

[issue29708] support reproducible Python builds

Inada Naoki added the comment: > note the optimized .pyc is deterministic. As far as I know only __debug__ is > set to False, or is there something else different? There is no difference between normal pyc and optimized pyc. * frozenset is deterministic if PYTHONHASHSEED is set * FLAG_REF

[issue29708] support reproducible Python builds

Frederik Rietdijk added the comment: note the optimized .pyc is deterministic. As far as I know only __debug__ is set to False, or is there something else different? -- ___ Python tracker

[issue29708] support reproducible Python builds

Inada Naoki added the comment: See bpo-34093 too. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue29708] support reproducible Python builds

STINNER Victor added the comment: > tiny bytecode differences bpo-37596 "Reproducible pyc: frozenset is not serialized in a deterministic order" is not fixed yet. -- ___ Python tracker

[issue29708] support reproducible Python builds

Frederik Rietdijk added the comment: Building Python packages reproducibly has now basically been resolved with the reproducible bytecode as well as changes in tools such as pip. Unfortunately, the interpreters do not yet seem to be reproducible. After certain changes, a Nixpkgs build of

[issue29708] support reproducible Python builds

Change by Will Thompson : -- nosy: -Will Thompson ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue29708] support reproducible Python builds

Change by Éric Araujo : -- dependencies: -setup.py sdist --format=gztar should use (equivalent of) `gzip -n`, setup.py sdist should honor SOURCE_DATE_EPOCH ___ Python tracker

[issue29708] support reproducible Python builds

Change by Éric Araujo : -- dependencies: +Reproducible pyc: frozenset is not serialized in a deterministic order, setup.py sdist --format=gztar should use (equivalent of) `gzip -n`, setup.py sdist should honor SOURCE_DATE_EPOCH ___ Python tracker

[issue29708] support reproducible Python builds

Change by Jeffery To : -- nosy: +jefferyto ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue29708] support reproducible Python builds

STINNER Victor added the comment: I created bpo-37596 "Reproducible pyc: frozenset is not serialized in a deterministic order". -- ___ Python tracker ___

[issue29708] support reproducible Python builds

miss-islington added the comment: New changeset 24b51b1a4919e310d338629cc60371387f475a32 by Miss Islington (bot) in branch '3.7': bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH (GH-9607)

[issue29708] support reproducible Python builds

Change by miss-islington : -- pull_requests: +10019 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue29708] support reproducible Python builds

Change by Sascha Silbe : -- nosy: +sascha_silbe ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue29708] support reproducible Python builds

STINNER Victor added the comment: New changeset a6b3ec5b6d4f6387820fccc570eea08b9615620d by Victor Stinner (Elvis Pranskevichus) in branch 'master': bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH (GH-9607)

[issue29708] support reproducible Python builds

Change by Elvis Pranskevichus : -- pull_requests: +9005 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue29708] support reproducible Python builds

Bernhard M. Wiedemann added the comment: also related to this topic: https://github.com/pypa/pip/pull/5525 for pip's RECORD file. -- ___ Python tracker ___

[issue29708] support reproducible Python builds

STINNER Victor added the comment: I created bpo-34033: distutils is not reproducible. -- ___ Python tracker ___ ___

[issue29708] support reproducible Python builds

Change by Matej Cepl : -- nosy: +mcepl ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue29708] support reproducible Python builds

STINNER Victor added the comment: I created PR 8057 to upstream distutils-reproducible-compile.patch from OpenSUSE (context: see bpo-34022). -- ___ Python tracker ___

[issue29708] support reproducible Python builds

Change by STINNER Victor : -- pull_requests: +7666 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue29708] support reproducible Python builds

Alexandru Ardelean added the comment: PYTHONHASHSEED does help on 3.6.4 I'll use it during build. Thanks for help -- ___ Python tracker

[issue29708] support reproducible Python builds

INADA Naoki added the comment: 3e 02 00 00 00 is frozenset(size=2) 72 b6/b5 00 00 00 is reference to b5 or b6 So it seems set order changed. (or items in the set is appearance order is changed.) Did you set PYTHONHASHSEED? Anyway, I think Python 3.7 can't guarantee

[issue29708] support reproducible Python builds

Alexandru Ardelean added the comment: Yeah, I also see it with 3.6.4. I wanted to try 3.7 to see if it's fixed by chance. Otherwise I may have to start digging deep into compilation logic. Looking here: https://tests.reproducible-builds.org/lede/lede_ar71xx.html More

[issue29708] support reproducible Python builds

Will Thompson added the comment: For what it's worth, in Endless OS we still saw slight variations between builds in the .pyc files, even with all the source files' mtimes set to the epoch (ie. equivalent to setting & supporting SOURCE_DATE_EPOCH, I believe). Looking

[issue29708] support reproducible Python builds

Bernhard M. Wiedemann added the comment: Any chance we can get the (somewhat related) patch for https://bugs.python.org/issue30693 also merged? -- ___ Python tracker

[issue29708] support reproducible Python builds

Change by Brett Cannon : -- assignee: brett.cannon -> stage: resolved -> ___ Python tracker ___

[issue29708] support reproducible Python builds

Change by Brett Cannon : -- resolution: fixed -> status: closed -> open ___ Python tracker ___

[issue29708] support reproducible Python builds

Alexandru Ardelean added the comment: Hey, Sorry, if I'm a bit late to the party with this. The road to reproducible builds has a few more steps. The way I validate whether Python is reproducible is with this link:

[issue29708] support reproducible Python builds

Change by Alexandru Ardelean : -- pull_requests: +5159 ___ Python tracker ___ ___

[issue29708] support reproducible Python builds

Brett Cannon added the comment: New changeset cab0b2b053970982b760048acc3046363615a8dd by Brett Cannon in branch 'master': bpo-29708: Add What's New entries for SOURCE_DATE_EPOCH and py_compile (GH-5306)

[issue29708] support reproducible Python builds

Change by Brett Cannon : -- pull_requests: +5153 ___ Python tracker ___ ___

[issue29708] support reproducible Python builds

Brett Cannon added the comment: Just merged Bernhard's PR which forces hash-based .pyc files. Thanks to everyone who constructively helped reach this point. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed

[issue29708] support reproducible Python builds

Brett Cannon added the comment: New changeset ccbe5818af20f8c12043f5c30c277a74714405e0 by Brett Cannon (Bernhard M. Wiedemann) in branch 'master': bpo-29708: Setting SOURCE_DATE_EPOCH forces hash-based .pyc files (GH-5200)

[issue29708] support reproducible Python builds

Change by Chih-Hsuan Yen : -- nosy: -yan12125 ___ Python tracker ___ ___

[issue29708] support reproducible Python builds

Brett Cannon added the comment: Since Barry chose an option that wasn't listed, I'm planning on accepting Bernhard's https://github.com/python/cpython/pull/5200 at some point next week barring any new, unique objections. -- ___

[issue29708] support reproducible Python builds

Change by Bernhard M. Wiedemann : -- keywords: +patch pull_requests: +5054 stage: -> patch review ___ Python tracker ___

[issue29708] support reproducible Python builds

Barry A. Warsaw added the comment: On Jan 15, 2018, at 11:31, Brett Cannon wrote: > > 1. SOURCE_DATE_EPOCH acts as an environment variable flag to forcibly > generate hash-based .pyc files with the check_source bit set in py_compile > and compileall

[issue29708] support reproducible Python builds

Brett Cannon added the comment: Bernhard's idea of SOURCE_DATE_EPOCH being an implicit envvar to forcibly switch on hash-based .pyc files in py_compile is intriguing. I assume this would force the check_source bit to be set? Or since SOURCE_DATE_EPOCH should only be used in

[issue29708] support reproducible Python builds

Bernhard M. Wiedemann added the comment: I think, there is no single nice and clean solution with time-based .pyc files, but to get a whole distribution to build reproducibly, there are two other ways: 1) if the SOURCE_DATE_EPOCH environment variable is set, make

[issue29708] support reproducible Python builds

Brett Cannon added the comment: As Eli's comments are coming off as negative to/at me, I feel like I have to defend myself here. If you look at the commit there was actually two places where the timestamp was checked; one did an equality comparison and one did a >= comparison.

[issue29708] support reproducible Python builds

Eli Schwartz added the comment: So, a couple of things. It seems to me, that properly supporting SOURCE_DATE_EPOCH means using exactly that and nothing else. To that end, I'm not entirely sure why things like --clamp-mtime even exist, as the original timestamp of a

[issue29708] support reproducible Python builds

Brett Cannon added the comment: A disagreement has popped up over what the ideal solution is on the PR currently connected to this issue. I'm having the folks involved switch it over to here. IMO I think py_compile can respect SOURCE_DATE_EPOCH and just blindly use it for

[issue29708] support reproducible Python builds

Change by Antoine Pitrou : -- nosy: +Ray Donnelly ___ Python tracker ___ ___

[issue29708] support reproducible Python builds

Alexandru Ardelean added the comment: Thank you for the heads-up. I did not follow-up too in-depth on the resolution. I just stumbled over this last night. Will keep an eye for 3.7, and see about 2.7. -- ___ Python tracker

[issue29708] support reproducible Python builds

Benjamin Peterson added the comment: PEP 552 has been implemented for 3.7. -- ___ Python tracker ___

[issue29708] support reproducible Python builds

Alexandru Ardelean added the comment: Hey, Allow me to join the discussion here. Context: - I'm the maintainer of Python & Python3 in the OpenWrt distro, and (since a while) we also care about reproducible builds. - The person [Alexander Couzens] who's leading the

[issue29708] support reproducible Python builds

Benjamin Peterson added the comment: I have proposed PEP 552 to address this issue. -- nosy: +benjamin.peterson ___ Python tracker ___

[issue29708] support reproducible Python builds

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___

[issue29708] support reproducible Python builds

Changes by Brett Cannon : -- nosy: +brett.cannon ___ Python tracker ___ ___

[issue29708] support reproducible Python builds

Changes by Chi Hsuan Yen : -- nosy: +Chi Hsuan Yen ___ Python tracker ___ ___

[issue29708] support reproducible Python builds

Bernhard M. Wiedemann added the comment: backports are optional. It can help reduce duplicated work for the various distributions. Currently, I think master and 2.7 are the most relevant targets. -- versions: +Python 3.7 ___ Python tracker

[issue29708] support reproducible Python builds

Barry A. Warsaw added the comment: Shouldn't this at least also cover Python 3.7? And should it be officially backported? I would think that if https://github.com/python/cpython/pull/296 gets accepted for 3.7, then distros that care can cherry pick it back into whatever versions they still

[issue29708] support reproducible Python builds

Changes by Barry A. Warsaw : -- nosy: +barry ___ Python tracker ___ ___ Python-bugs-list

[issue29708] support reproducible Python builds

Eric V. Smith added the comment: -- Eric. > On Mar 3, 2017, at 6:36 AM, Bernhard M. Wiedemann > wrote: > > > New submission from Bernhard M. Wiedemann: > > See https://reproducible-builds.org/ and > https://reproducible-builds.org/docs/buy-in/ for why this is a

[issue29708] support reproducible Python builds

New submission from Bernhard M. Wiedemann: See https://reproducible-builds.org/ and https://reproducible-builds.org/docs/buy-in/ for why this is a good thing to have in general. Fedora, openSUSE and possibly other Linux distributions package .pyc files as part of their binary rpm packages