Re: Three bytes in a zip file

Larry Doolittle wrote: > Yes. The -X isn't needed, sometimes, and then when you least expect it, it > is. > Classic reproducible-builds gotcha. > >> I'm happy to update this document myself if need be. :) > > Go for it. Finally got around to doing this. Many thanks. :) Best wishes, --

Re: Three bytes in a zip file

Chris - On Wed, Mar 13, 2024 at 01:01:47AM +, Chris Lamb wrote: > > TZ=UTC zip -X --latest-time "$zipfile" fab/* > > # Note the -X flag; to be pedantic about timestamps, > > # that means you should unpack with TZ=UTC unzip "$zipfile". See > > # > >

Re: Three bytes in a zip file

Hey, > echo "Forcing timestamp $SOURCE_DATE_EPOCH" > touch --date="@$SOURCE_DATE_EPOCH" fab/* > TZ=UTC zip -X --latest-time "$zipfile" fab/* > # Note the -X flag; to be pedantic about timestamps, > # that means you should unpack with TZ=UTC unzip "$zipfile". See > # >

Re: Three bytes in a zip file

Friends - On Fri, Mar 08, 2024 at 05:21:43PM +0100, Fay Stegerman wrote: > * Chris Lamb [2024-03-08 12:16]: > > Oh this is great work! So, using your tool, did you manage to solve the > > underlying non-determinism? :) > The original reproducibility issue this thread started with was traced back

Re: Three bytes in a zip file

Hey Fay, > The original reproducibility issue this thread started with was traced back to > the atime back then, my tool just hopefully makes doing that a bit easier :) Oh! I somehow missed that this was an atime-related issue at the time… in addition to missing your PoC to call out to repro-apk

Re: Three bytes in a zip file

* Fay Stegerman [2024-03-08 02:37]: > * Larry Doolittle [2023-04-06 23:59]: > > Do you know of any tooling that can help decode zip file contents in > > general? > > I know this thread is almost a year old now, but I finally got to my backlog > working on diff-zip-meta.py [1], which is a tool

Re: Three bytes in a zip file

* Chris Lamb [2024-03-08 12:16]: > Oh this is great work! So, using your tool, did you manage to solve the > underlying non-determinism? :) > > Based on the output (which labels the field as an 'extra atime' or > similar), it seems like you've managed to work out which part of your > toolchain

Re: Three bytes in a zip file

Chris - On Fri, Mar 08, 2024 at 11:16:06AM +, Chris Lamb wrote: > Oh this is great work! As the (year-old) thread originator, I agree! This fills a real gap! > So, using your tool, did you manage to solve the > underlying non-determinism? :) We did figure it out, very tediously, a year

Re: Three bytes in a zip file

Hey Fay, Oh this is great work! So, using your tool, did you manage to solve the underlying non-determinism? :) Based on the output (which labels the field as an 'extra atime' or similar), it seems like you've managed to work out which part of your toolchain is making the build reproducible — or

Re: Three bytes in a zip file

* Larry Doolittle [2023-04-06 23:59]: > Do you know of any tooling that can help decode zip file contents in general? I know this thread is almost a year old now, but I finally got to my backlog working on diff-zip-meta.py [1], which is a tool specifically intended to elucidated differences in

Re: Three bytes in a zip file

On 06/04/2023 10.28, Larry Doolittle wrote: I'm trying to make a process to generate byte-for-byte reproducible zip files. Try adding the -X option to the zip call. It will suppress adding of extended attributes (atime/ctime). And with

Re: Three bytes in a zip file

Michael - On Fri, Apr 07, 2023 at 01:31:24PM +0200, Michael Schierl wrote: > Larry's script already called touch immediately before zip. But I assume > the nature of atime can mean that any other process may have "won the > race" and accessed the file just in between these two lines. That's my

Re: Three bytes in a zip file

Hello John, Am 07.04.2023 um 03:56 schrieb John Gilmore: Larry Doolittle wrote: $ diff <(ls --full-time -u fab-ea2bb52c-ld) <(ls --full-time -u fab-ea2bb52c-mb) 22c22 < -rw-r--r-- 1 redacted redacted 644661 2023-04-04 18:10:00.0 -0700 marble-ipc-d-356.txt --- -rw-r--r-- 1 redacted

Re: Three bytes in a zip file

Hello, Am 06.04.2023 um 23:59 schrieb Larry Doolittle: Do you know of any tooling that can help decode zip file contents in general? Ideally something that could be absorbed into diffoscope? Maybe that one-liner above would be a useful addition to diffoscope. I don't know. I would assume

Re: Three bytes in a zip file

Larry Doolittle wrote: > $ diff <(ls --full-time -u fab-ea2bb52c-ld) <(ls --full-time -u > fab-ea2bb52c-mb) > 22c22 > < -rw-r--r-- 1 redacted redacted 644661 2023-04-04 18:10:00.0 -0700 > marble-ipc-d-356.txt > --- > > -rw-r--r-- 1 redacted redacted 644661 2023-04-06

Re: Three bytes in a zip file

Michael - On Thu, Apr 06, 2023 at 12:11:38PM +0200, Michael Schierl wrote: > Am 06.04.2023 um 10:28 schrieb Larry Doolittle: > > I'm trying to make a process to generate byte-for-byte reproducible zip > > files. > > I got the contents identical, including timestamps and permissions. > > But

Re: Three bytes in a zip file

Hello, Am 06.04.2023 um 10:28 schrieb Larry Doolittle: Friends - I'm trying to make a process to generate byte-for-byte reproducible zip files. I got the contents identical, including timestamps and permissions. But three bytes at the 98.08% mark (bytes 5543078 to 5543081, out of a file size

Re: Three bytes in a zip file

Hi Larry, > TZ=UTC zip --latest-time "$zipfile" fab/* ^ Just as a quick glance, this may be expanding the '*' shell glob in a different order between your two systems due to the underlying filesystem order. Perhaps try "zip […] -r […] fab/"? > The diff

Three bytes in a zip file

Friends - I'm trying to make a process to generate byte-for-byte reproducible zip files. I got the contents identical, including timestamps and permissions. But three bytes at the 98.08% mark (bytes 5543078 to 5543081, out of a file size 5651451) differ between my run and a friend's run.