Bug#1005197: pcmemtest: please make the build partly reproducible
Am Freitag, dem 25.03.2022 um 14:13 -0700 schrieb Vagrant Cascadian: > On 2022-03-25, Felix Zielcke wrote: > > Am Mittwoch, dem 23.02.2022 um 09:36 +0100 schrieb Thomas Schmitt: > > I totally forgot about this. Sorry! > > > > I just tried to test it with TZ set. > > But I always get the same binaries. No matter to what I change TZ > > or > > /etc/timezone between the builds. > > > > So is this just a bullseye problem? unstable has a newer mtools > > > > I'll try to test this in stable chroot the next days > > I can reproduce the issue in a debian sid chroot with pcmemtest 1.5-3 > from debian, using: > > reprotest --verbose --min-cpus=1 --vary=-user_group,-domain_host,- > fileordering auto -- null > > Passing TZ=UTC in three different ways, exporting TZ=UTC from > debian/rules, exporting TZ=UTC from the upstream Makefiles, and > prepending the mcopy calls in the upstream Makefiles. All three > approaches are below: Thanks Thomas and Vagrant. I finally could test this. Seems like an export TZ=UTC directly before the mcopy call doestn't work. And I wrongly thought stuff like TZ=UTC+1 works too. I export it now globally at the start of both Makefiles Oh and by the way, in case you don't know it already: memtest86+ 6.0 will be based on pcmemtest. And this means development of pcmemtest will stop. But the reproducible work done here is also good for memtest86+. :) Where I'm the new co-maintainer. Cheers, Felix
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi,
Felix Zielcke wrote:
> > I just tried to test it with TZ set.
> > But I always get the same binaries. No matter to what I change TZ or
> > /etc/timezone between the builds.
Vagrant Cascadian wrote:
> I can reproduce the issue in a debian sid chroot with pcmemtest 1.5-3
> from debian
I don't have newest Debian in reach today and am too lazy to set up
pcmemtest package building anyways. :o)
So i rather tested near to the bottom, i.e. mcopy(1) and localtime(3).
Both react on TZ here.
If Felix Zielcke and Vagrant Cascadian get the same result, then i
wonder why TZ does not have an influence in the Makefile of pcmemtest.
---
Test of mcopy(1):
Using rather outdated libc (i.e. localtime(3)) and rather young locally
compiled mtools-4.0.37 (source of january 2022):
/sbin/mkdosfs -i 12345678 -n MEMTEST-ESP -F12 -C fat.img 1024
./mcopy -s -i fat.img x ::/x
TZ=UTF ./mcopy -s -i fat.img x ::/y
TZ=UTF+3 ./mcopy -s -i fat.img x ::/z
This yields after mounting fat.img :
-rwxr-xr-x 1 root root 9931 Mar 26 09:29 x
-rwxr-xr-x 1 root root 9931 Mar 26 08:29 y
-rwxr-xr-x 1 root root 9931 Mar 26 05:29 z
which is somewhat wrong because my local time (CET = UTF-1) was
$ date
Sat Mar 26 10:29:49 CET 2022
mdir from mtools does it better (but other than ls -l it does not
react on TZ):
$ mdir -i fat.img
...
x 9931 2022-03-26 10:29
y 9931 2022-03-26 9:29
z 9931 2022-03-26 6:29
Whatever, the mcopy command reacts on TZ.
---
Test of localtime(3):
In order to directly check the influence of TZ on localtime(3), which
does the time format conversion in mtools--4.0.37/directory.c function
mk_entry(), i wrote a test program:
--- File start
/*
cc -g -o localtime_test localtime_test.c
*/
#include
#include
int main()
{
time_t now;
struct tm *res;
now= time(NULL);
res= localtime(&now);
printf("%d/%2.2d/%2.2d %2.2d:%2.2d:%2.2d (%d)\n",
1900 + res->tm_year, 1 + res->tm_mon, res->tm_mday,
res->tm_hour, res->tm_min, res->tm_sec,
res->tm_isdst);
return(0);
}
--- File end
After
cc -g -o localtime_test localtime_test.c
i get plausible results:
$ ./localtime_test
2022/03/26 10:43:05 (0)
$ TZ=UTC ./localtime_test
2022/03/26 09:43:14 (0)
$ TZ=UTC+3 ./localtime_test
2022/03/26 06:43:18 (0)
---
Have a nice day :)
Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
On 2022-03-25, Felix Zielcke wrote: > Am Mittwoch, dem 23.02.2022 um 09:36 +0100 schrieb Thomas Schmitt: > I totally forgot about this. Sorry! > > I just tried to test it with TZ set. > But I always get the same binaries. No matter to what I change TZ or > /etc/timezone between the builds. > > So is this just a bullseye problem? unstable has a newer mtools > > I'll try to test this in stable chroot the next days I can reproduce the issue in a debian sid chroot with pcmemtest 1.5-3 from debian, using: reprotest --verbose --min-cpus=1 --vary=-user_group,-domain_host,-fileordering auto -- null Passing TZ=UTC in three different ways, exporting TZ=UTC from debian/rules, exporting TZ=UTC from the upstream Makefiles, and prepending the mcopy calls in the upstream Makefiles. All three approaches are below: diff --git a/debian/rules b/debian/rules index 9477c3e..860be75 100755 --- a/debian/rules +++ b/debian/rules @@ -1,5 +1,7 @@ #!/usr/bin/make -f +export TZ=UTC + %: dh $@ Alternately, by patching the upstream Makefiles: diff --git a/build32/Makefile b/build32/Makefile index 1d47859..93a671f 100644 --- a/build32/Makefile +++ b/build32/Makefile @@ -47,6 +47,8 @@ APP_OBJS = app/badram.o \ OBJS = boot/startup.o boot/efisetup.o $(SYS_OBJS) $(LIB_OBJS) $(TST_OBJS) $(APP_OBJS) +export TZ=UTC + all: memtest.bin memtest.efi -include boot/efisetup.d diff --git a/build64/Makefile b/build64/Makefile index 24e1ddc..0094868 100644 --- a/build64/Makefile +++ b/build64/Makefile @@ -46,6 +46,8 @@ APP_OBJS = app/badram.o \ OBJS = boot/startup.o boot/efisetup.o $(SYS_OBJS) $(LIB_OBJS) $(TST_OBJS) $(APP_OBJS) +export TZ=UTC + all: memtest.bin memtest.efi -include boot/efisetup.d Or patching upstream Makefile to only set the timezone on the mcopy calls: diff --git a/build32/Makefile b/build32/Makefile index 1d47859..38324a5 100644 --- a/build32/Makefile +++ b/build32/Makefile @@ -120,7 +120,7 @@ esp.img: memtest.efi cp memtest.efi iso/EFI/BOOT/bootia32.efi @rm -f esp.img /sbin/mkdosfs -i 12345678 --invariant -n MEMTEST-ESP -F12 -C esp.img 4096 - mcopy -s -i esp.img iso/EFI :: + TZ=UTC mcopy -s -i esp.img iso/EFI :: iso: memtest.mbr floppy.img esp.img @mkdir -p iso/boot diff --git a/build64/Makefile b/build64/Makefile index 24e1ddc..faa6402 100644 --- a/build64/Makefile +++ b/build64/Makefile @@ -119,7 +119,7 @@ esp.img: memtest.efi cp memtest.efi iso/EFI/BOOT/bootx64.efi @rm -f esp.img /sbin/mkdosfs -i 12345678 --invariant -n MEMTEST-ESP -F12 -C esp.img 4096 - mcopy -s -i esp.img iso/EFI :: + TZ=UTC mcopy -s -i esp.img iso/EFI :: memtest.iso: memtest.mbr floppy.img esp.img @mkdir -p iso/boot @@ -169,7 +169,7 @@ grub-esp.img: memtest.efi grub-bootx64.efi grub/grub-efi.cfg cp $(GRUB_LIB_DIR)/x86_64-efi/*.mod grub-iso/EFI/BOOT/grub/x86_64-efi/ @rm -f grub-esp.img /sbin/mkdosfs -n MEMTEST-ESP -F12 -C grub-esp.img 8192 - mcopy -s -i grub-esp.img grub-iso/EFI :: + TZ=UTC mcopy -s -i grub-esp.img grub-iso/EFI :: grub-memtest.iso: memtest.bin grub-eltorito.img grub/grub-legacy.cfg grub-esp.img @mkdir -p grub-iso/boot/grub/i386-pc grub-iso/boot/grub/fonts live well, vagrant signature.asc Description: PGP signature
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, Chris Lamb wrote: > More generally, I can't think of a scenario where the value of > SOURCE_DATE_EPOCH request a timezone implication, Alain Knaff mentioned mcopy option -m "Preserve the file modification time." which he wants to fulfill the user's expectation that the Y/M/D/h/m/s formatted timestamps show local time. (That's a similar problem as i perceived in xorriso with your wish to make the effects of SOURCE_DATE_EPOCH immutable in xorriso, although a design principle of xorriso is that any setting can be set and revoked by a xorriso command. My solution was to let SOURCE_DATE_EPOCH set the defaults of the various involved settings but to allow the commands to override those defaults, if really desired.) I plan to write a mail to rb-gene...@lists.reproducible-builds.org about my difficulties to justify the demand for a fixed time zone from the specs of https://reproducible-builds.org/specs/source-date-epoch/ I did not find a better motivation than by the specification "Formatting MUST be deferred until runtime if an end user should observe the value in their own locale or timezone." from which i argue: * FAT needs formatting of SOURCE_DATE_EPOCH. * One cannot do this at "runtime" of the FAT filesystem, which in the special case of pcmtest is when EFI reads the filesystem to find and run the \EFI\BOOT\BOOT*.EFI start program. => So one must not let the end user see their own locale or timezone. Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi Thomas, > i wrote to info-mtools about the problem that time zone influences the > result of mcopy even if SOURCE_TIME_EPOCH is set. > Alain Knaff then proposed to set the TZ variable to UTF when pcmemtest > gets built. > https://lists.gnu.org/archive/html/info-mtools/2022-02/msg2.html > > Would that be viable in pcmemtest ? > Are any problems to foresee if TZ is set inside build*/Makefile ? I do not envisage any problem in doing this, although it is somewhat strange for mcopy to use SOURCE_TIME_EPOCH in a timezone-dependent manner. More generally, I can't think of a scenario where the value of SOURCE_DATE_EPOCH request a timezone implication, so if mcopy supports this particular use-case (and, indeed, supports it as the default one), I might boldly suggest that this is a bug in mcopy. Just my £0.02. :) Thanks for doing a lot of the legwork here. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org 🍥 chris-lamb.co.uk `-
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, i wrote to info-mtools about the problem that time zone influences the result of mcopy even if SOURCE_TIME_EPOCH is set. Alain Knaff then proposed to set the TZ variable to UTF when pcmemtest gets built. https://lists.gnu.org/archive/html/info-mtools/2022-02/msg2.html Would that be viable in pcmemtest ? Are any problems to foresee if TZ is set inside build*/Makefile ? Fritz Zielke: Can you give it a try in your patch ? Vagrant Cascadian: Can you then test whether this helped ? Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, my favorite suspect in mtools is to see in function mk_entry(): https://sources.debian.org/src/mtools/4.0.33-1+really4.0.32-1/directory.c/?hl=98#L107 now = localtime(&date2); (I see no debian/patches directory for mtools. So this is probably really in effect with Debian's package.) I understand that this function gets the SOURCE_DATE_EPOCH value as time_t date2 = date; but then applies the local timezone when generating the timestamp components. From man localtime: The localtime() function converts the calendar time timep to broken- down time representation, expressed relative to the user's specified timezone. Unix time functions always confused me. So my idea of using gmtime(3) instead of localtime(3) has to be taken with a grain of salt. (Further it might be necessary to use gmtime(3) only when SOURCE_DATE_EPOCH is defined and not in normal operation mode.) Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, Vagrant Cascadian wrote: > I haven't looked into it too deeply, but got the basically the same > results as currently showing on: > > > https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/pcmemtest.html Looks again like the FAT directory entries. The byte address is sufficiently above the 1.4 MiB of ISO 9660 payload to be in partition 2. We see the directory names "BOOT" and "EFI", and the file name "BOOTX64" with extension "EFI". The differences are at offsets 0x0f, 0x10, 0x12, 0x17, 0x18 after the name start bytes. According to https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system#Directory_entry that would be timestamp fields. 0x0f : belongs to the 16 bit of "Create time" beginning at offset 0x0e. Values as little endian: 0x9a91 <-> 0xaa91 Hour:Minute:Second 19:20:34 <-> 21:20:34 0x10 : 16 bit of "Create date". Values as little endian: 0x544c <-> 0x544d Year/Month/Day 2022/02/12 <-> 2022/02/13 0x12 : 16 bit of "Last access date". 2022/02/12 <-> 2022/02/13 0x17 : 16 bit of "Last modified time" beginning at 0x16. 19:20:34 <-> 21:20:34 0x18 : 16 bit of "Last modified date". 2022/02/12 <-> 2022/02/13 So for some reasons the timestamps of files do not stay stable. Since the timestamps of the filesystem label (a pseudo file name) staid stable, i would now rather suspect mcopy and not mkdosfs as culprit. - So the task seems to be to find out how to force timestamps with this line from build*/Makefile: mcopy -s -i esp.img iso/EFI :: Google brings me to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837044 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900410 where Chris Lamb fought such effects and let mtools obey SOURCE_DATE_EPOCH. Was this change not enough ? Does tests.reproducible-builds.org use an older version of mcopy ? Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
On 2022-02-15, Thomas Schmitt wrote: > Vagrant Cascadian wrote: >> I noticed 1.5-3 is still affected by timezone differences, > > Can you already say whether the differences are in the range of > partition 1 or in the range of partition 2 of the .iso (as told by > fdisk -l) ? I haven't looked into it too deeply, but got the basically the same results as currently showing on: https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/pcmemtest.html Maybe you can figure out from that where in the file? I could probably publish the built .debs somewhere, if that would help. >> If I come up with an updated patch, I'll maybe submit a new bug... > > Please Cc me then. (At least i'm curious, if not being the culprit.) Will do, though not sure when I'll get a chance to take another look at it; anyone feel free to fix it first! live well, vagrant signature.asc Description: PGP signature
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, Vagrant Cascadian wrote: > I noticed 1.5-3 is still affected by timezone differences, Can you already say whether the differences are in the range of partition 1 or in the range of partition 2 of the .iso (as told by fdisk -l) ? > If I come up with an updated patch, I'll maybe submit a new bug... Please Cc me then. (At least i'm curious, if not being the culprit.) Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
On 2022-02-12, Felix Zielcke wrote:
> Am Freitag, dem 11.02.2022 um 22:46 +0100 schrieb Thomas Schmitt:
>> Chris Lamb wrote:
>> > Did you try the --invariant flag?
>>
>> Well, i step aside and let this question reach Felix. :))
>>
>> https://manpages.debian.org/unstable/dosfstools/mkdosfs.8.en.html
>> says that this is what we need if my theory about the second
>> difference
>> is correct:
>>
>> --invariant
>> Use constants for normally randomly generated or time based data
>> such as volume ID and creation time. Multiple runs of mkfs.fat on
>> the same device create identical results with this option.
>>
>>
>> So the options to be added to the mkdosfs runs in the hope for a
>> fully
>> reproducible ISO are:
>>
>> -i 12345678 --invariant
>>
>> If a more more individual -i number is desired, one could use
>> something
>> like
>>
>> -i $(printf '%8.8x' "$SOURCE_DATE_EPOCH" | head -c 8)
>>
>>
>
> Wow. Thanks very much to you both.
> This is the solution. I would never come up that fast with it.
I noticed 1.5-3 is still affected by timezone differences, but is very
nearly there!
Figured out using:
reprotest --auto-build --store-dir=$(mktemp -d ${HOME}/rb-test-XX)
--min-cpus=1 --vary=-user_group,-domain_host,-fileordering auto -- null
Which does a build for each different tested variation. Some variations
are trickier to set up with the null backend, so I just disable
them. The store-dir argument allows you to see all the various
diffoscope outputs and build artifacts.
If I come up with an updated patch, I'll maybe submit a new bug...
live well,
vagrant
signature.asc
Description: PGP signature
Bug#1005197: pcmemtest: please make the build partly reproducible
Control: tags -1 pending Am Freitag, dem 11.02.2022 um 22:46 +0100 schrieb Thomas Schmitt: > Hi, > > Chris Lamb wrote: > > Did you try the --invariant flag? > > Well, i step aside and let this question reach Felix. :)) > > https://manpages.debian.org/unstable/dosfstools/mkdosfs.8.en.html > says that this is what we need if my theory about the second > difference > is correct: > > --invariant > Use constants for normally randomly generated or time based data > such as volume ID and creation time. Multiple runs of mkfs.fat on > the same device create identical results with this option. > > > So the options to be added to the mkdosfs runs in the hope for a > fully > reproducible ISO are: > > -i 12345678 --invariant > > If a more more individual -i number is desired, one could use > something > like > > -i $(printf '%8.8x' "$SOURCE_DATE_EPOCH" | head -c 8) > > Wow. Thanks very much to you both. This is the solution. I would never come up that fast with it. > Have a nice day :) > Have a nice weekend :) Felix
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, Chris Lamb wrote: > Did you try the --invariant flag? Well, i step aside and let this question reach Felix. :)) https://manpages.debian.org/unstable/dosfstools/mkdosfs.8.en.html says that this is what we need if my theory about the second difference is correct: --invariant Use constants for normally randomly generated or time based data such as volume ID and creation time. Multiple runs of mkfs.fat on the same device create identical results with this option. So the options to be added to the mkdosfs runs in the hope for a fully reproducible ISO are: -i 12345678 --invariant If a more more individual -i number is desired, one could use something like -i $(printf '%8.8x' "$SOURCE_DATE_EPOCH" | head -c 8) Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
[merging two replies] Hi Thomas, > I wonder whether the "file list" in "data.tar.xz" of > https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/pcmemtest.html > is made from the files in the ISO or from the input files on hard disk. That can safely be ignored for our purposes. That "file list" is merely repeating that the two .debs have different contents... but they have different contents because the two ISO files within them are different; something we already know. > First an observation from digging in pcmemtest/1.5-2/build64/Makefile: > What about target grub-memtest.iso ? > It has its own xorriso run and EFI System Partition image. > I guess they need treatment for reproducibility, too. Ah, that does sound plausible, although I am now really quite out of my depth in terms of ISO/EFI/xorriso knowledge. > The other difference could be a neighbor of "directory volume label", > which Wikipedia mentions as a kind of mirror of "Partition Volume Label". > If i get it right then this is in the name and extension fields of > a "Directory entry". > The differing bytes would then be the timestamps "Create time" at offset > 0x0E and "Last Modified time" at offset 0x16. Probably the date fields at > 0x10 and 0x18 could show differences, too. > > But i am far off my usual playground now. Same. ;) > Maybe Chris already knows a good trick to enforce values for those time > and date fields in FAT filesystems. Hm. Well, I remember doing some work on dosfstools and/or mtools in order that they could generate deterministic output, but it has been quite a few years now. Did you try the --invariant flag? (There were also some patches flying around to fix mtools serialising/saving uninitialised memory as well, but I think most of them got applied, at least in Debian.) Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org 🍥 chris-lamb.co.uk `-
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, First an observation from digging in pcmemtest/1.5-2/build64/Makefile: What about target grub-memtest.iso ? It has its own xorriso run and EFI System Partition image. I guess they need treatment for reproducibility, too. Felix Zielcke wrote: > Here's the output for the 64bit ISO: > [...] > │ │ │ 00179000: eb3c 906d 6b66 732e 6661 7400 0204 0100 .<.mkfs.fat. > │ │ │ 00179010: 0200 0200 20f8 0600 2000 0200 ... ... > │ │ │ -00179020: 8000 296c 13ae 4d4d 454d 5445 ..)l..MMEMTE > │ │ │ +00179020: 8000 29d6 8629 594d 454d 5445 ..)..)YMEMTE > │ │ │ 00179030: 5354 2d45 5350 4641 5431 3220 2020 0e1f ST-ESPFAT12 .. > [...] > │ │ │ -0017aa00: 4d45 4d54 4553 542d 4553 5008 a750 MEMTEST-ESPP > │ │ │ -0017aa10: 4b54 4b54 a750 4b54 KTKT...PKT.. > │ │ │ +0017aa00: 4d45 4d54 4553 542d 4553 5008 0951 MEMTEST-ESPQ > │ │ │ +0017aa10: 4b54 4b54 0951 4b54 KTKT...QKT.. > │ │ │ 0017aa20: 4546 4920 2020 2020 2020 2010 186a EFI j The clear text looks like EFI System Partition (ESP). The byte address 0x00179000 = 1,544,192 would roughly match the end of an ISO 9660 filesystem which only has the 1440 KiB of /boot/floppy.img as payload. So it would be roughly at the start of the partition in the ISO created by -append_partition 2 0xef ./esp.img A partition editor listing like /sbin/fdisk -l memtestx64.iso or a run of xorriso -indev memtestx64.iso -report_system_area plain would tell where partition 2 begins and whether the byte addresses are located in there. Assumed that the differences are indeed inside partition 2: The content of the appended partition image does not get altered by xorriso. So i would expect that differences stem from the creation of ./esp.img. Do i get it right from https://sources.debian.org/src/pcmemtest/1.5-2/build64/Makefile/#L121 that esp.img stems from /sbin/mkdosfs -n MEMTEST-ESP -F12 -C esp.img 4096 mcopy -s -i esp.img iso/EFI :: ? Assumed yes: We see in the first difference the string "MEMTEST-ESPFAT12" which would match the "Partition Volume Label" and "File system type" fields as of https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system#Extended_BIOS_Parameter_Block The differing byte would then be a part of "Volume ID (serial number)". In man mkdosfs i read -i volume-id Sets the volume ID of the newly created filesystem; volume-id is a 32-bit hexadecimal number (for example, 2e24ec82). The default is a number which depends on the filesystem creation time. So consider to add an option -i to the mkdosfs run and give it either a constant number or a number which gets reproducibly derived from variable SOURCE_DATE_EPOCH. -- The other difference could be a neighbor of "directory volume label", which Wikipedia mentions as a kind of mirror of "Partition Volume Label". If i get it right then this is in the name and extension fields of a "Directory entry". The differing bytes would then be the timestamps "Create time" at offset 0x0E and "Last Modified time" at offset 0x16. Probably the date fields at 0x10 and 0x18 could show differences, too. But i am far off my usual playground now. Maybe Chris already knows a good trick to enforce values for those time and date fields in FAT filesystems. Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Am Freitag, dem 11.02.2022 um 11:41 +0100 schrieb Thomas Schmitt: > Hi, > > Felix Zielcke wrote: > > I modifed the patch now to use --set_all_file_dates. > > I wonder whether the "file list" in "data.tar.xz" of > > https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffosc > ope-results/pcmemtest.html > is made from the files in the ISO or from the input files on hard > disk. > > In the latter case, it would be better to use the solution of Chris > which enforces dates on hard disk. > But then this solution should also enforce user id and group id > numbers > on hard disk. > > > Ah ok. I can try that. > > diffoscope still tells me some differences in the ISOs. > > Is there a chance to quote some of the found differences here ? > If it's about meta data of the ISO then i would be the person to > decode > them and to make first theories from where the deviations might come. > > Here's the output for the 64bit ISO: │ ├── ./usr/lib/pcmemtest/memtestx64.iso │ │ │┄ Format-specific differences are supported for ISO 9660 CD images but no file-specific differences were detected; falling back to a binary diff. file(1) reports: ISO 9660 CD-ROM filesystem data (DOS/MBR boot sector) 'PCMemTest64' (bootable) │ │ │ @@ -96508,15 +96508,15 @@ │ │ │ 00178fb0: │ │ │ 00178fc0: │ │ │ 00178fd0: │ │ │ 00178fe0: │ │ │ 00178ff0: │ │ │ 00179000: eb3c 906d 6b66 732e 6661 7400 0204 0100 .<.mkfs.fat. │ │ │ 00179010: 0200 0200 20f8 0600 2000 0200 ... ... │ │ │ -00179020: 8000 296c 13ae 4d4d 454d 5445 ..)l..MMEMTE │ │ │ +00179020: 8000 29d6 8629 594d 454d 5445 ..)..)YMEMTE │ │ │ 00179030: 5354 2d45 5350 4641 5431 3220 2020 0e1f ST-ESPFAT12 .. │ │ │ 00179040: be5b 7cac 22c0 740b 56b4 0ebb 0700 cd10 .[|.".t.V... │ │ │ 00179050: 5eeb f032 e4cd 16cd 19eb fe54 6869 7320 ^..2...This │ │ │ 00179060: 6973 206e 6f74 2061 2062 6f6f 7461 626c is not a bootabl │ │ │ 00179070: 6520 6469 736b 2e20 2050 6c65 6173 6520 e disk. Please │ │ │ 00179080: 696e 7365 7274 2061 2062 6f6f 7461 626c insert a bootabl │ │ │ 00179090: 6520 666c 6f70 7079 2061 6e64 0d0a 7072 e floppy and..pr │ │ │ @@ -96922,16 +96922,16 @@ │ │ │ 0017a990: │ │ │ 0017a9a0: │ │ │ 0017a9b0: │ │ │ 0017a9c0: │ │ │ 0017a9d0: │ │ │ 0017a9e0: │ │ │ 0017a9f0: │ │ │ -0017aa00: 4d45 4d54 4553 542d 4553 5008 a750 MEMTEST-ESPP │ │ │ -0017aa10: 4b54 4b54 a750 4b54 KTKT...PKT.. │ │ │ +0017aa00: 4d45 4d54 4553 542d 4553 5008 0951 MEMTEST-ESPQ │ │ │ +0017aa10: 4b54 4b54 0951 4b54 KTKT...QKT.. │ │ │ 0017aa20: 4546 4920 2020 2020 2020 2010 186a EFIj │ │ │ 0017aa30: 4954 4954 186a 4954 0200 ITIT...jIT.. │ │ │ 0017aa40: │ │ │ 0017aa50: │ │ │ 0017aa60: │ │ │ 0017aa70: │ │ │ 0017aa80:
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, Felix Zielcke wrote: > I modifed the patch now to use --set_all_file_dates. I wonder whether the "file list" in "data.tar.xz" of https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffosc ope-results/pcmemtest.html is made from the files in the ISO or from the input files on hard disk. In the latter case, it would be better to use the solution of Chris which enforces dates on hard disk. But then this solution should also enforce user id and group id numbers on hard disk. > diffoscope still tells me some differences in the ISOs. Is there a chance to quote some of the found differences here ? If it's about meta data of the ISO then i would be the person to decode them and to make first theories from where the deviations might come. Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Am Donnerstag, dem 10.02.2022 um 16:24 -0800 schrieb Chris Lamb: > Hi Thomas, > > > > Have you had a look at the diffoscope output after this patch is > > > applied? > > > > Where can a curious bystander get that look ? > > I was referring to a hypothetical diff that you might generate > locally; updating the result on tests.reproducible-builds.org would > require a new upload. :) > > > i see in the isoinfo output only differences of user id, group id, > > and timestamps of /boot and /boot/floppy.img. > [...] > > https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1005197;filename=pcmemtest.diff.txt;msg=5 > > should fix both. > > It perhaps should, but alas it does not. When I apply my patch, the > user ID, group ID and timestamp differences do disappear for me too, > but that is not the only difference I see. I mean, hence why I said > that my patch "does not make the ISOs entirely reproducible; there is > another part or parts of the image that elude me right now". :) > > I wouldn't jump to the conclusion that it is a bug in xorriso just > yet. It might be reproducible for you locally, but it is likely I am > varying more things between my two builds. And even more on > tests.reproducible-builds.org. > > The next step might be to upload with this patch so we can both work > with the same output on tests.reproducible-builds.org. > > > --set_all_file_dates "=$SOURCE_DATE_EPOCH" > > Ah, neat. :) Yes, of course, feel free to use this over touch(1). > > Hi Thomas and Chris, thanks very much for your efforts. I modifed the patch now to use --set_all_file_dates. diffoscope still tells me some differences in the ISOs. I built it multiple times with cowbuilder. As soon as it migrated to bookworm I'll do an upload with the patch. Regards, Felix
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi Thomas, >> Have you had a look at the diffoscope output after this patch is >> applied? > > Where can a curious bystander get that look ? I was referring to a hypothetical diff that you might generate locally; updating the result on tests.reproducible-builds.org would require a new upload. :) > i see in the isoinfo output only differences of user id, group id, > and timestamps of /boot and /boot/floppy.img. [...] > https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1005197;filename=pcmemtest.diff.txt;msg=5 > should fix both. It perhaps should, but alas it does not. When I apply my patch, the user ID, group ID and timestamp differences do disappear for me too, but that is not the only difference I see. I mean, hence why I said that my patch "does not make the ISOs entirely reproducible; there is another part or parts of the image that elude me right now". :) I wouldn't jump to the conclusion that it is a bug in xorriso just yet. It might be reproducible for you locally, but it is likely I am varying more things between my two builds. And even more on tests.reproducible-builds.org. The next step might be to upload with this patch so we can both work with the same output on tests.reproducible-builds.org. > --set_all_file_dates "=$SOURCE_DATE_EPOCH" Ah, neat. :) Yes, of course, feel free to use this over touch(1). Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org 🍥 chris-lamb.co.uk `-
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, Chris Lamb wrote: > Have you had a look at the diffoscope output after this > patch is applied? Where can a curious bystander get that look ? I only found a diff from february 8: https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/pcmemtest.html where i see in the isoinfo output only differences of user id, group id, and timestamps of /boot and /boot/floppy.img. The patch in the original message of this bug report https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1005197;filename=pcmemtest.diff.txt;msg=5 should fix both. > Perhaps looking through > the xorriso manual page would be helpful, as there are a number of header > checksums in such files. If SOURCE_DATE_EPOCH is set then all automaticly generated ids in the ISO meta data should be reproducibly derived from that time value. If all input is the same, then all checksums should be the same. If not, it would be a bug in xorriso. (I just tested a mock-up of the xorrisofs run in pcmemtest.diff.txt. Both resulting ISOs were identical.) -- Side note: The gesture find iso -print0 | xargs -0r touch --date="@$(SOURCE_DATE_EPOCH)" could be replaced by xorrisofs option --set_all_file_dates "=$SOURCE_DATE_EPOCH" which acts on the files in the emerging ISO filesystem rather than on the input files on hard disk. (The '=' is xorriso's equivalent of date's '@'. See man xorriso, command -alter_date .) Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi Felix, > thanks for your patch. Just committed it. > Do you or anyone else have a hint how to make the ISOs fully > reproducible? How to do this is totally new to me. Good question. Have you had a look at the diffoscope output after this patch is applied? From memory, it looked like some kind of header, and often these are checksums or timestamps. Perhaps looking through the xorriso manual page would be helpful, as there are a number of header checksums in such files. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org 🍥 chris-lamb.co.uk `-
Bug#1005197: pcmemtest: please make the build partly reproducible
Am Dienstag, dem 08.02.2022 um 12:06 -0800 schrieb Chris Lamb: > Hi, > > Whilst working on the Reproducible Builds effort [0] we noticed that > pcmemtest could not be built reproducibly. > > This is because the generated ISOs inherit the uid/gid of the build > process, as well as the modification times of (at least) floppy.img > > Patch attached that modifies the two Makefiles involved, although > this > does not make the ISOs entirely reproducible; there is another part > or parts of the image that elude me right now. > > [0] https://reproducible-builds.org/ > > > Regards, > Hi Chris, thanks for your patch. Just committed it. Do you or anyone else have a hint how to make the ISOs fully reproducible? How to do this is totally new to me. Regards, Felix
Bug#1005197: pcmemtest: please make the build partly reproducible
Source: pcmemtest Version: 1.5-2 Severity: wishlist Tags: patch User: reproducible-bui...@lists.alioth.debian.org Usertags: timestamps X-Debbugs-Cc: reproducible-b...@lists.alioth.debian.org Hi, Whilst working on the Reproducible Builds effort [0] we noticed that pcmemtest could not be built reproducibly. This is because the generated ISOs inherit the uid/gid of the build process, as well as the modification times of (at least) floppy.img Patch attached that modifies the two Makefiles involved, although this does not make the ISOs entirely reproducible; there is another part or parts of the image that elude me right now. [0] https://reproducible-builds.org/ Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- --- a/debian/patches/reproducible-builds.patch 1969-12-31 16:00:00.0 -0800 --- b/debian/patches/reproducible-builds.patch 2022-02-08 12:03:57.320136181 -0800 @@ -0,0 +1,28 @@ +Description: Make the build reproducible +Author: Chris Lamb +Last-Update: 2022-02-08 + +--- pcmemtest-1.5.orig/build32/Makefile pcmemtest-1.5/build32/Makefile +@@ -125,7 +125,8 @@ esp.img: memtest.efi + iso: memtest.mbr floppy.img esp.img + @mkdir -p iso/boot + cp floppy.img iso/boot/floppy.img +- xorrisofs -pad -R -J -volid PCMemTest32 -graft-points -hide-rr-moved --grub2-mbr memtest.mbr \ ++ find iso -print0 | xargs -0r touch --date="@$(SOURCE_DATE_EPOCH)" ++ xorrisofs -uid 1000 -gid 1000 -pad -R -J -volid PCMemTest32 -graft-points -hide-rr-moved --grub2-mbr memtest.mbr \ + -b /boot/floppy.img --efi-boot --interval:appended_partition_2:all:: \ + -part_like_isohybrid -iso_mbr_part_type 0x00 -append_partition 2 0xef ./esp.img \ + -o ./memtest.iso /boot=./iso/boot +--- pcmemtest-1.5.orig/build64/Makefile pcmemtest-1.5/build64/Makefile +@@ -124,7 +124,8 @@ esp.img: memtest.efi + memtest.iso: memtest.mbr floppy.img esp.img + @mkdir -p iso/boot + cp floppy.img iso/boot/floppy.img +- xorrisofs -pad -R -J -volid PCMemTest64 -graft-points -hide-rr-moved --grub2-mbr memtest.mbr \ ++ find iso -print0 | xargs -0r touch --date="@$(SOURCE_DATE_EPOCH)" ++ xorrisofs -uid 1000 -gid 1000 -pad -R -J -volid PCMemTest64 -graft-points -hide-rr-moved --grub2-mbr memtest.mbr \ + -b /boot/floppy.img --efi-boot --interval:appended_partition_2:all:: \ + -part_like_isohybrid -iso_mbr_part_type 0x00 -append_partition 2 0xef ./esp.img \ + -o ./memtest.iso /boot=./iso/boot --- a/debian/patches/series 1969-12-31 16:00:00.0 -0800 --- b/debian/patches/series 2022-02-08 11:50:45.010064249 -0800 @@ -0,0 +1 @@ +reproducible-builds.patch
