After rebasing ElectroBSD from FreeBSD stable/11 to stable/12 I recently noticed that the "memstick" images are unfortunately still not 100% reproducible.
I already tried to fix this myself a couple of years ago but got distracted and unfortunately nobody else seems to have fixed the problem in the mean time, either. IIRC, last time I checked the differences "only" occurred in the UFS data which was created with makefs (so mkimg was blameless). I'd like to look into this again and as a first step it would be helpful to understand which parts of the UFS file system actually vary. Is anyone aware of a tool that can parse images created with mkimg that contain an UFS file system created with makefs, or a tool that can't deal with the whole image but can work with an UFS file system image? As a work around ElectroBSD contains image-checksum.sh since 2015: <https://www.fabiankeil.de/sourcecode/electrobsd/ElectroBSD-20211130-817fe7d130ce/image-checksum.sh> While image-checksum.img can interpret the UFS filesystem from the "outside" by mounting it first, the relevant differences seem to be below the "POSIX layer" and as far as image-checksum.sh is concerned the images are "reproducible enough" to get the same "partial image checksums". [fk@elektrobier3 ~/scratch/ElectroBSD-20211130-817fe7d130ce-2021-12-01]$ sudo /usr/src/release/scripts/image-checksum.sh -v /dpool/images/elektrobier3/ElectroBSD-20211130-817fe7d130ce-j2/ElectroBSD-20211130-817fe7d130ce.img /dpool/images/test-vm/ElectroBSD-20211130-817fe7d130ce-2021-12-01-18\:47/ElectroBSD-20211130-817fe7d130ce.img [...] gpart checksum: 63089c3730bb9a55342f1a71132bc2fad29ab6b0b5746e3e13ea0ee2208fd9e4 Boot code checksum: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 mtree checksum: 67ca25a06160e79acb8160f74c49b33c40b7a31f2abcfe4c801a7cf235594007 ls checksum: f9b944005ac17ecea86f354a19e7f41e7037bb72d0cbae1eb46148c2f98c4350 Partial image checksum for /dpool/images/elektrobier3/ElectroBSD-20211130-817fe7d130ce-j2/ElectroBSD-20211130-817fe7d130ce.img: a9e1d8158b47f0a5c5636cb3aadcad916c13e002f04d7c3d634a7884f501d169 [...] gpart checksum: 63089c3730bb9a55342f1a71132bc2fad29ab6b0b5746e3e13ea0ee2208fd9e4 Boot code checksum: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 mtree checksum: 67ca25a06160e79acb8160f74c49b33c40b7a31f2abcfe4c801a7cf235594007 ls checksum: f9b944005ac17ecea86f354a19e7f41e7037bb72d0cbae1eb46148c2f98c4350 Partial image checksum for /dpool/images/test-vm/ElectroBSD-20211130-817fe7d130ce-2021-12-01-18:47/ElectroBSD-20211130-817fe7d130ce.img: a9e1d8158b47f0a5c5636cb3aadcad916c13e002f04d7c3d634a7884f501d169 While I can diff the hd/hexdump(1) output manually, this requires lots of cpu time and RAM, is therefore slow, and the output isn't particularly helpful. Quoting from top while diff was running: PID JID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND [...] 96115 0 fk 1 102 0 2740M 924M CPU1 1 102:43 98.23% diff -u ElectroBSD-20211130-817fe7d130ce.img.elektrobier3.txt ElectroBSD-20211130-817fe7d130ce.img.test-vm.txt And the complete output as example: [fk@elektrobier3 ~/scratch/ElectroBSD-20211130-817fe7d130ce-2021-12-01]$ time diff -u ElectroBSD-20211130-817fe7d130ce.img.elektrobier3.txt ElectroBSD-20211130-817fe7d130ce.img.test-vm.txt > hd.diff real 124m35.611s user 117m35.689s sys 0m26.180s [fk@elektrobier3 ~/scratch/ElectroBSD-20211130-817fe7d130ce-2021-12-01]$ cat hd.diff --- ElectroBSD-20211130-817fe7d130ce.img.elektrobier3.txt 2021-12-02 13:56:56.879948000 +0100 +++ ElectroBSD-20211130-817fe7d130ce.img.test-vm.txt 2021-12-02 14:22:08.070199000 +0100 @@ -5212,7 +5212,7 @@ 000da4c0 6d 64 36 34 00 00 00 00 00 00 00 00 00 00 00 00 |md64............| 000da4d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * -000da540 00 c0 73 01 08 00 00 00 00 d0 73 01 08 00 00 00 |..s.......s.....| +000da540 00 d0 73 01 08 00 00 00 00 e0 73 01 08 00 00 00 |..s.......s.....| 000da550 00 00 00 00 00 00 00 00 00 00 00 00 00 80 00 00 |................| 000da560 00 00 00 00 00 00 00 00 48 9d 03 00 00 00 00 00 |........H.......| 000da570 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| @@ -5257,7 +5257,7 @@ 000e24c0 6d 64 36 34 00 00 00 00 00 00 00 00 00 00 00 00 |md64............| 000e24d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * -000e2540 00 c0 73 01 08 00 00 00 00 d0 73 01 08 00 00 00 |..s.......s.....| +000e2540 00 d0 73 01 08 00 00 00 00 e0 73 01 08 00 00 00 |..s.......s.....| 000e2550 00 00 00 00 00 00 00 00 00 00 00 00 00 80 00 00 |................| 000e2560 00 00 00 00 00 00 00 00 48 9d 03 00 00 00 00 00 |........H.......| 000e2570 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| @@ -47913116,7 +47913116,7 @@ 364fa4c0 6d 64 36 34 00 00 00 00 00 00 00 00 00 00 00 00 |md64............| 364fa4d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * -364fa540 00 c0 73 01 08 00 00 00 00 d0 73 01 08 00 00 00 |..s.......s.....| +364fa540 00 d0 73 01 08 00 00 00 00 e0 73 01 08 00 00 00 |..s.......s.....| 364fa550 00 00 00 00 00 00 00 00 00 00 00 00 00 80 00 00 |................| 364fa560 00 00 00 00 00 00 00 00 48 9d 03 00 00 00 00 00 |........H.......| 364fa570 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| So it looks like only a couple of bytes vary but it's not obvious (to me at least) what those varying bytes mean. BTW, the diffoscope output isn't particularly helpful either: --- /dpool/images/elektrobier3/ElectroBSD-20211130-817fe7d130ce-j2/ElectroBSD-20211130-817fe7d130ce.img +++ /dpool/images/test-vm/ElectroBSD-20211130-817fe7d130ce-2021-12-01-18:47/ElectroBSD-20211130-817fe7d130ce.img │┄ xxd not available in path. Falling back to Python hexlify. @@ -27940,15 +27940,15 @@ 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000456c656374726f4253442d31322e332d535441424c452d61 6d64363400000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 -00c073010800000000d073010800000000000000000000000000000000800000 +00d073010800000000e073010800000000000000000000000000000000800000 0000000000000000489d03000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 00000000000000000000010000000000c703000000000000bc07000000000000 ff27000000000000050000000000000000000000000000000000000000000000 000000000000000000000000000000007814a56100000000489d030000000000 @@ -28964,15 +28964,15 @@ 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000456c656374726f4253442d31322e332d535441424c452d61 6d64363400000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 -00c073010800000000d073010800000000000000000000000000000000800000 +00d073010800000000e073010800000000000000000000000000000000800000 0000000000000000489d03000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 00000000000000000000010000000000c703000000000000bc07000000000000 ff27000000000000050000000000000000000000000000000000000000000000 000000000000000000000000000000007814a56100000000489d030000000000 @@ -28474660,15 +28474660,15 @@ 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000456c656374726f4253442d31322e332d535441424c452d61 6d64363400000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 -00c073010800000000d073010800000000000000000000000000000000800000 +00d073010800000000e073010800000000000000000000000000000000800000 0000000000000000489d03000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 00000000000000000000010000000000c703000000000000bc07000000000000 ff27000000000000050000000000000000000000000000000000000000000000 000000000000000000000000000000007814a56100000000489d030000000000 Fabian
pgppBAFmseQgK.pgp
Description: OpenPGP digital signature