Re: should bsdinstall work?

2021-05-04 Thread tech-lists

On Tue, May 04, 2021 at 07:33:40PM -0700, Chris wrote:

Doesn't bsdinstall allow choosing different means of fetching the
dist files? 


iirc it gives an ftp list. I tried with ftp.uk and ftp.de.freebsd.org.


Last I remember using it, I was able to choose ftp http(s),
etc... Making, or using an already matched MANIFEST is fairly trivial.


i'll try downloading a matched manifest by hand, next. Thanks for the
tip.
--
J.


signature.asc
Description: PGP signature


Re: FYI: WITH_REPRODUCIBLE_BUILD= problem for some files? [aarch64 test did not reproduce the issue]

2021-05-04 Thread Mark Millard via freebsd-current


On 2021-May-4, at 13:38, Mark Millard  wrote:

> [The first buidlworld is still in process. So while waiting . . .]
> 
> On 2021-May-4, at 10:31, Mark Millard  wrote:
> 
>> I probably know why the huge count of differences this time
>> unlike the original report . . .
>> 
>> Previously I built based on a checked-in branch as part of
>> my experimenting. This time it was in a -dirty form (not
>> checked in), again as part of my experimental exploration.
>> 
>> WITH_REPRODUCIBLE_BUILD= makes a distinction between these
>> if I remember right: (partially?) disabling itself for
>> -dirty style.
>> 
>> To reproduce the original style of test I need to create
>> a branch with my few patches checked in and do the
>> buildworlds from that branch.
>> 
>> This will, of course, take a while.
>> 
>> Sorry for the noise.
>> 
> 
> I've confirmed some of the details of the large number of
> files with difference while waiting for the 1st buildworld :
> 
> The 4 bytes at the end of the .gnu_debuglink section
> that are ending up different are the checksum for the
> .debug file. The .debug files have differences such as:
> 
> │ -<1a>   DW_AT_comp_dir: (indirect string) 
> /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/usr/13_0R-src/arm64.aarch64/lib/csu/aarch64
> │ +<1a>   DW_AT_comp_dir: (indirect string) 
> /usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/lib/csu/aarch64
> 
> So I need to build, snapshot (in case need
> to reference), install, clean-out, build,
> install elsewhere, compare. (Or analogous
> that uses the same build base-path for both
> installs despite separate buildworld's.)
> This is separate from any potential -dirty
> vs. checked-in handling variation by
> WITH_REPRODUCIBLE_BUILD= .
> 
> My process that produced the original armv7
> report happened to do that before I accidentally
> discovered the presence of the few files with
> differences. My new experiments were different
> and I'd not though of needing to vary the
> procedure to get you the right evidence.
> 

The two aarch64 test installs did not show any
differences in a "diff -rq" . Ignoring *.meta
files generated during the builds, the build
directory tree snapshots showed just the
differences:

# diff -rq 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr | 
grep -v '\.meta' | more
Files 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c
 and 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c
 differ
Files 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk
 and 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk
 differ

# diff -u 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c
 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c
--- 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c
 2021-05-04 13:45:14.463351000 -0700
+++ 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c
 2021-05-04 19:04:32.338203000 -0700
@@ -4,7 +4,7 @@
 ** Words from CORE set written in FICL
 ** Author: John Sadler (john_sad...@alum.mit.edu)
 ** Created: 27 December 1997
-** Last update: Tue May  4 13:45:14 PDT 2021
+** Last update: Tue May  4 19:04:32 PDT 2021
 ***/
 /*
 ** DO NOT EDIT THIS FILE -- it is generated by softwords/softcore.awk

# diff -u 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk
 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk
--- 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk
 2021-05-04 10:55:26.030179000 -0700
+++ 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk
 2021-05-04 16:14:24.513346000 -0700
@@ -1,4 +1,4 @@
-.info Using cached toolchain metadata from build at CA72_4c8G_ZFS on Tue May  
4 10:55:26 PDT 2021
+.info Using cached toolchain metadata from build at CA72_4c8G_ZFS on Tue May  
4 16:14:24 PDT 2021
 _LOADED_TOOLCHAIN_METADATA=t
 COMPILER_VERSION=110001
 X_COMPILER_VERSION=110001



I may run a 'target cortex-a7 (so: armv7) from aarch64' test
next. That was the context for the original discovery and
report.



===
Mark Millard
marklmi at yahoo.com
( ds

Re: should bsdinstall work?

2021-05-04 Thread Chris

On 2021-05-04 18:58, tech-lists wrote:

Hi,

On Tue, May 04, 2021 at 04:35:17PM -0400, Nathan Whitehorn wrote:


Where are you trying to install
to? Usually the assumption is that the microsd images *are* the
installed system rather than a tool you use to install a system.


I'm trying to install -current (or stable/13 or releng/13.0) to a
bootable usb3-connected external hd on raspberry pi 4 hardware. The goal
is to have this pi booting without microsd to a root-on-zfs system.

I have used raspios 64-bit to update its firmware and configured it so
it tries to boot the microsd card and if this fails, boots to usb3. Took
out that card, wrote
FreeBSD-14.0-CURRENT-arm64-aarch64-RPI-20210415-14d0cd7225e-246078.img
to another and booted it, then ran bsdinstall and selected the external
hd. The install fails to progress beyond the point I mentioned.

Maybe another wau of doing it would be to just write
FreeBSD-14.0-CURRENT-arm64-aarch64-RPI-20210415-14d0cd7225e-246078.img
to the hd from my freebsd desktop. Would that method be expected to
work? The thing is, I want root-on-zfs *and* booting with no microsd.
Just writing the .img to the hd would mean i'd have to abandon zfs.

Doesn't bsdinstall allow choosing different means of fetching the
dist files? Last I remember using it, I was able to choose ftp http(s),
etc... Making, or using an already matched MANIFEST is fairly trivial.
You could obtain your desired base files and create or use the one
with it. Then point bsdinstall to somewhere to get them. If you need
a remote. I could give you one to use.

HTH

--Chris
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: should bsdinstall work?

2021-05-04 Thread tech-lists

Hi,

On Tue, May 04, 2021 at 04:35:17PM -0400, Nathan Whitehorn wrote:


Where are you trying to install
to? Usually the assumption is that the microsd images *are* the
installed system rather than a tool you use to install a system.


I'm trying to install -current (or stable/13 or releng/13.0) to a
bootable usb3-connected external hd on raspberry pi 4 hardware. The goal
is to have this pi booting without microsd to a root-on-zfs system.

I have used raspios 64-bit to update its firmware and configured it so
it tries to boot the microsd card and if this fails, boots to usb3. Took
out that card, wrote
FreeBSD-14.0-CURRENT-arm64-aarch64-RPI-20210415-14d0cd7225e-246078.img
to another and booted it, then ran bsdinstall and selected the external
hd. The install fails to progress beyond the point I mentioned.

Maybe another wau of doing it would be to just write
FreeBSD-14.0-CURRENT-arm64-aarch64-RPI-20210415-14d0cd7225e-246078.img
to the hd from my freebsd desktop. Would that method be expected to
work? The thing is, I want root-on-zfs *and* booting with no microsd.
Just writing the .img to the hd would mean i'd have to abandon zfs.

--
J.


signature.asc
Description: PGP signature


ZFS rename with associated snapshot present: odd error message

2021-05-04 Thread Mark Millard via freebsd-current
I had a:

# zfs list -tall
NAME   USED  AVAIL REFER  MOUNTPOINT
. . .
zroot/DESTDIRs/13_0R-CA72-instwrld-norm  1.44G   117G   96K  
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm
zroot/DESTDIRs/13_0R-CA72-instwrld-norm@dirty-style  1.44G  - 1.44G  -. 
. .
. . .

(copied/pasted from somewhat earlier) and then attempted:

# zfs rename zroot/DESTDIRs/13_0R-CA72-instwrld-norm 
zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0
cannot open 'zroot/DESTDIRs/13_0R-CA72-instwrld-norm@dirty-style': snapshot 
delimiter '@' is not expected here

Despite the "cannot open" message, the result looks like:

# zfs list -tall
NAME   USED  AVAIL 
REFER  MOUNTPOINT
. . .
zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0  1.44G   114G   
96K  /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt-0
zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0@dirty-style  1.44G  - 
1.44G  -
. . .

Still, it leaves me wondering if everything is okay
given that internal attempt to use the old name with
@dirty-style when it was apparently no longer
available under that naming.

For reference:

# uname -apKU
FreeBSD CA72_4c8G_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 
releng/13.0-n244733-ea31abc261ff-dirty: Thu Apr 29 21:53:20 PDT 2021 
root@CA72_4c8G_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/sys/GENERIC-NODBG-CA72
  arm64 aarch64 1300139 1300139


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: should bsdinstall work?

2021-05-04 Thread Mark Millard via freebsd-current
tech-lists tech-lists at zyxst.net wrote on
Tue May 4 19:22:39 UTC 2021 :

> I've been trying to set up a boot-from-usb3 raspberry pi 4. I thought
> i'd be able to do this by booting into latest current snapshot image for
> arm64 rpi (written to microsd), and then running bsdinstall as root.

I've done something similar, deliberately making the root
file system ZFS based to experiment with bectl usage and
to have GPT instead of MBR, for example.

Even when other things work, do not expect bsdinstall to
deal with the RPi firmware or u-boot in the msdos file
system partition. I copied those over separately.

> I can select the auto-zfs option then select the usb3 disk as
> installation destination. The bsdinstall selects the sets and downloads
> them, but then get the error:
> 
> "error while fetching 
> file://usr/freebsd-dist/MANIFEST
>  : No such file or
> directory"

As I remember, I had to use bsdinstall's DISTRIBUTIONS
environment variable to pick what to download in order
to get MANIFEST to download. Otherwise I got just the 2
files (base.txz and kernel.txz) and the complaint that
you report.

Ultimately I picked to get more than just base.txz and
kernel.txz as well.

I'll note that:

http://ftp3.freebsd.org/pub/FreeBSD/releases/arm64/13.0-RELEASE/

has the MANIFIEST file available. As does:

http://ftp3.freebsd.org/pub/FreeBSD/snapshots/arm64/14.0-CURRENT/

Unfortrunately, there has not been a successful snapshot
build for arm64/aarch64 or arm/armv7 recently. So there
are no .../13.0-STABLE/ examples yet.

Another place to stable-13 materials is paths matching:

https://artifact.ci.freebsd.org/snapshot/stable-13/*/arm64/aarch64/

that have MANIFEST and *.txz files. For example,

https://artifact.ci.freebsd.org/snapshot/stable-13/d6d039ea74a26357173d1263682d4f5119037434/arm64/aarch64/

has a snapshot as of one of today's commits. When looking
in such folders, be sure to check the dates on the files:
they might be older than the parent directory dates
suggest. Sometimes one finds the directory is empty
despite the arm64/aarch64/ level being created.)

There is also:

https://artifact.ci.freebsd.org/snapshot/stable-11/
https://artifact.ci.freebsd.org/snapshot/stable-12/
https://artifact.ci.freebsd.org/snapshot/main/

and aliases based on names like 13.0-STABLE and head .

> also tried via release/13 image, same error. Looks like it's downloading
> the files but then the installer can't see them, something like that.
> 
> the downloaded files are there:
> 
> 
> root at generic
> :~ # ls -lah /mnt/usr/freebsd-dist/
> total 187454
> drwxr-xr-x  2 root  wheel 4B Apr  9 06:51 .
> drwxr-xr-x  6 root  wheel 6B Apr  9 06:50 ..
> -rw-r--r--  1 root  wheel   158M Apr  9 06:51 base.txz
> -rw-r--r--  1 root  wheel25M Apr  9 06:51 kernel.txz

Yea, without DISTRIBUTIONS it seemed to grab just the
2 files without also getting MANIFEST. Or at least that
is my memory of how it went. I used DISTRIBUTIONS to
get more files and so also got MANIFEST.

You can also put the files in place separately and
tell bsdinstall to look in a file:///path that you
type as I remember. file:///usr/freebsd-dist could
be an example.

> root at generic
> :~ # 
> 
> worth submitting a PR? or is bsdinstall legacy and I need to use some
> other method. I've not tried releng/12.2 yet.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FYI: WITH_REPRODUCIBLE_BUILD= problem for some files? [Ignore recent test: -dirty vs. checked-in usage difference]

2021-05-04 Thread Mark Millard via freebsd-current
[The first buidlworld is still in process. So while waiting . . .]

On 2021-May-4, at 10:31, Mark Millard  wrote:

> I probably know why the huge count of differences this time
> unlike the original report . . .
> 
> Previously I built based on a checked-in branch as part of
> my experimenting. This time it was in a -dirty form (not
> checked in), again as part of my experimental exploration.
> 
> WITH_REPRODUCIBLE_BUILD= makes a distinction between these
> if I remember right: (partially?) disabling itself for
> -dirty style.
> 
> To reproduce the original style of test I need to create
> a branch with my few patches checked in and do the
> buildworlds from that branch.
> 
> This will, of course, take a while.
> 
> Sorry for the noise.
> 

I've confirmed some of the details of the large number of
files with difference while waiting for the 1st buildworld :

The 4 bytes at the end of the .gnu_debuglink section
that are ending up different are the checksum for the
.debug file. The .debug files have differences such as:

│ -<1a>   DW_AT_comp_dir: (indirect string) 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/usr/13_0R-src/arm64.aarch64/lib/csu/aarch64
│ +<1a>   DW_AT_comp_dir: (indirect string) 
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/lib/csu/aarch64

So I need to build, snapshot (in case need
to reference), install, clean-out, build,
install elsewhere, compare. (Or analogous
that uses the same build base-path for both
installs despite separate buildworld's.)
This is separate from any potential -dirty
vs. checked-in handling variation by
WITH_REPRODUCIBLE_BUILD= .

My process that produced the original armv7
report happened to do that before I accidentally
discovered the presence of the few files with
differences. My new experiments were different
and I'd not though of needing to vary the
procedure to get you the right evidence.


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: should bsdinstall work?

2021-05-04 Thread Nathan Whitehorn
Yes, it should work just fine; however, we don't provision the microsd 
images for the installer, especially for -CURRENT. The actual OS is 
fetched in plaintext to allow caching and the MANIFEST files are what 
provides authentication -- they provide the checksums of the files that 
get fetched so that they can be verified against corruption and 
tampering. For snapshots, the current version changes all the time and 
that doesn't work; it also means that network-install media have to be 
set up with those checksums in advance. Where are you trying to install 
to? Usually the assumption is that the microsd images *are* the 
installed system rather than a tool you use to install a system.

-Nathan

On 5/4/21 3:22 PM, tech-lists wrote:

Hi,

I've been trying to set up a boot-from-usb3 raspberry pi 4. I thought
i'd be able to do this by booting into latest current snapshot image for
arm64 rpi (written to microsd), and then running bsdinstall as root.

I can select the auto-zfs option then select the usb3 disk as
installation destination. The bsdinstall selects the sets and downloads
them, but then get the error:

"error while fetching file://usr/freebsd-dist/MANIFEST : No such file or
directory"

also tried via release/13 image, same error. Looks like it's downloading
the files but then the installer can't see them, something like that.

the downloaded files are there:

root@generic:~ # ls -lah /mnt/usr/freebsd-dist/
total 187454
drwxr-xr-x  2 root  wheel 4B Apr  9 06:51 .
drwxr-xr-x  6 root  wheel 6B Apr  9 06:50 ..
-rw-r--r--  1 root  wheel   158M Apr  9 06:51 base.txz
-rw-r--r--  1 root  wheel    25M Apr  9 06:51 kernel.txz
root@generic:~ #
worth submitting a PR? or is bsdinstall legacy and I need to use some
other method. I've not tried releng/12.2 yet.

thanks,


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


should bsdinstall work?

2021-05-04 Thread tech-lists

Hi,

I've been trying to set up a boot-from-usb3 raspberry pi 4. I thought
i'd be able to do this by booting into latest current snapshot image for
arm64 rpi (written to microsd), and then running bsdinstall as root.

I can select the auto-zfs option then select the usb3 disk as
installation destination. The bsdinstall selects the sets and downloads
them, but then get the error:

"error while fetching file://usr/freebsd-dist/MANIFEST : No such file or
directory"

also tried via release/13 image, same error. Looks like it's downloading
the files but then the installer can't see them, something like that.

the downloaded files are there:

root@generic:~ # ls -lah /mnt/usr/freebsd-dist/
total 187454
drwxr-xr-x  2 root  wheel 4B Apr  9 06:51 .
drwxr-xr-x  6 root  wheel 6B Apr  9 06:50 ..
-rw-r--r--  1 root  wheel   158M Apr  9 06:51 base.txz
-rw-r--r--  1 root  wheel25M Apr  9 06:51 kernel.txz
root@generic:~ # 


worth submitting a PR? or is bsdinstall legacy and I need to use some
other method. I've not tried releng/12.2 yet.

thanks,
--
J.


signature.asc
Description: PGP signature


diffoscope's odd UnicodeDecodeError error message: reason found

2021-05-04 Thread Mark Millard via freebsd-current
I had reported in the reproducable build list messages:

> # diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
> [...]
> $<3/>2021-05-04 08:49:21 W: diffoscope.main: Fuzzy-matching is currently 
> disabled as the "tlsh" module is unavailable.
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: 
> invalid start byte

Well, it turns out that the file name pattern was
incorrect and only matched one file.

By contrast:

# diffoscope /.zfs/snapshot/2021-04-*/bin/sh
$<3/>2021-05-04 11:05:25 W: diffoscope.main: Fuzzy-matching is currently 
disabled as the "tlsh" module is unavailable.

worked fine.

And making the "one file" status obvious:

# diffoscope c_tests/a.out
$<3/>2021-05-04 11:11:45 W: diffoscope.main: Fuzzy-matching is currently 
disabled as the "tlsh" module is unavailable.
$<3/>Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 745, 
in main
sys.exit(run_diffoscope(parsed_args))
  File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 677, 
in run_diffoscope
difference = load_diff_from_path(path1)
  File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", 
line 31, in load_diff_from_path
return load_diff(codecs.getreader("utf-8")(fp), path)
  File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", 
line 35, in load_diff
return JSONReaderV1().load(fp, path)
  File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/json.py", 
line 33, in load
raw = json.load(fp)
  File "/usr/local/lib/python3.7/json/__init__.py", line 293, in load
return loads(fp.read(),
  File "/usr/local/lib/python3.7/codecs.py", line 504, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: 
invalid start byte

Not exactly an obvious error message for the issue.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FYI: WITH_REPRODUCIBLE_BUILD= problem for some files? [Ignore recent test: -dirty vs. checked-in usage difference]

2021-05-04 Thread Mark Millard via freebsd-current
I probably know why the huge count of differences this time
unlike the original report . . .

Previously I built based on a checked-in branch as part of
my experimenting. This time it was in a -dirty form (not
checked in), again as part of my experimental exploration.

WITH_REPRODUCIBLE_BUILD= makes a distinction between these
if I remember right: (partially?) disabling itself for
-dirty style.

To reproduce the original style of test I need to create
a branch with my few patches checked in and do the
buildworlds from that branch.

This will, of course, take a while.

Sorry for the noise.



===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FYI: WITH_REPRODUCIBLE_BUILD= problem for some files?

2021-05-04 Thread Mark Millard via freebsd-current
[Just adding readelf -S info since it seems to show more.]

On 2021-May-4, at 10:01, Mark Millard  wrote:


> On 2021-May-4, at 08:51, Mark Millard  wrote:
> 
>> On 2021-May-4, at 06:01, Ed Maste  wrote:
>> 
>>> On Mon, 3 May 2021 at 22:26, Mark Millard  wrote:
 
 But I'll note that I've built and stalled py37-diffoscope
 (new to me). A basic quick test showed that it reports:
 
 W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" 
 module is unavailable.
>>> 
>>> I just looked up tlsh - its "A Locality Sensitive Hash"; I presume
>>> diffoscope uses it to infer file renames. I believe the warning
>>> emitted here should have no impact on the output we're looking for.
>> 
>> Okay.
>> 
>>> As far as the utf-8 issues go, diffoscope requires a utf-8 locale and
>>> I suspect that is the issue. If you don't have LANG set already, try
>>> setting LANG=C.UTF-8 in your environment.
>> 
>> That is not the issue for the UnicodeDecodeError:
>> 
>> # echo $LANG
>> C.UTF-8
>> 
>> # diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
>> $<3/>2021-05-04 08:49:21 W: diffoscope.main: Fuzzy-matching is currently 
>> disabled as the "tlsh" module is unavailable.
>> $<3/>Traceback (most recent call last):
>> File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 745, 
>> in main
>>   sys.exit(run_diffoscope(parsed_args))
>> File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 677, 
>> in run_diffoscope
>>   difference = load_diff_from_path(path1)
>> File 
>> "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", 
>> line 31, in load_diff_from_path
>>   return load_diff(codecs.getreader("utf-8")(fp), path)
>> File 
>> "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", 
>> line 35, in load_diff
>>   return JSONReaderV1().load(fp, path)
>> File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/json.py", 
>> line 33, in load
>>   raw = json.load(fp)
>> File "/usr/local/lib/python3.7/json/__init__.py", line 293, in load
>>   return loads(fp.read(),
>> File "/usr/local/lib/python3.7/codecs.py", line 504, in read
>>   newchars, decodedbytes = self.decode(data, self.errors)
>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: 
>> invalid start byte
>> 
> 
> Well, the list of differing files is huge. But this seems to
> be .gnu_debuglink content for the area it is in.

Specifically: the last 4 bytes of the .gnu_debuglink section.

> I'll note
> that I did installworld but not the likes of distrib-dirs
> or distribution this time.
> 
> This test did buildworld to two distinct directories:
> 
> zroot/BUILDs/13_0R-CA72-nodbg-clang   5.13G   118G 5.13G  
> /usr/obj/BUILDs/13_0R-CA72-nodbg-clang
> zroot/BUILDs/13_0R-CA72-nodbg-clang-alt   4.28G   118G 4.28G  
> /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt
> 
> and installworld to 2 distinct directories:
> 
> zroot/DESTDIRs/13_0R-CA72-instwrld-alt1.44G   118G 1.44G  
> /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt
> zroot/DESTDIRs/13_0R-CA72-instwrld-norm   1.44G   118G 1.44G  
> /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm
> 
> Previously (armv7 target) I had built, installed, rebuilt
> to same directory (after clean-out) and installed to an
> alternate directory. That had gotten only a few files
> different but I do not know (yet) if it was the procedural
> difference that made the difference.
> 
> Prefix of the list of different files this time:
> 
> # diff -rq /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/ 
> /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/ | more
> Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/[ and 
> /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/[ differ
> Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat and 
> /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat differ
> Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags and 
> /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags differ
> Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio and 
> /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio differ
> . . .
> 
> Looking, aarch64 seems to typically get a back-to-back
> sequence of 4 bytes different in native programs in my
> builds:
> 
> # cmp -x /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat 
> /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat
> 3bd4 1d 65
> 3bd5 eb a3
> 3bd6 bb ca
> 3bd7 8e 1a
> 
> # ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat 
> /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat
> -r-xr-xr-x  1 root  wheel  18448 May  4 08:55:01 2021 
> /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat
> -r-xr-xr-x  1 root  wheel  18448 May  3 23:16:36 2021 
> /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat
> 
> Sections:
> Idx Name  Size  VMA   LMA   File off  Algn
> . . .
> 25 .gnu_debuglink 0010      3bc8  2**0
>  CONTENTS, READONLY

Section Header

Re: FYI: WITH_REPRODUCIBLE_BUILD= problem for some files?

2021-05-04 Thread Mark Millard via freebsd-current


On 2021-May-4, at 08:51, Mark Millard  wrote:

> On 2021-May-4, at 06:01, Ed Maste  wrote:
> 
>> On Mon, 3 May 2021 at 22:26, Mark Millard  wrote:
>>> 
>>> But I'll note that I've built and stalled py37-diffoscope
>>> (new to me). A basic quick test showed that it reports:
>>> 
>>> W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" 
>>> module is unavailable.
>> 
>> I just looked up tlsh - its "A Locality Sensitive Hash"; I presume
>> diffoscope uses it to infer file renames. I believe the warning
>> emitted here should have no impact on the output we're looking for.
> 
> Okay.
> 
>> As far as the utf-8 issues go, diffoscope requires a utf-8 locale and
>> I suspect that is the issue. If you don't have LANG set already, try
>> setting LANG=C.UTF-8 in your environment.
> 
> That is not the issue for the UnicodeDecodeError:
> 
> # echo $LANG
> C.UTF-8
> 
> # diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
> $<3/>2021-05-04 08:49:21 W: diffoscope.main: Fuzzy-matching is currently 
> disabled as the "tlsh" module is unavailable.
> $<3/>Traceback (most recent call last):
>  File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 745, 
> in main
>sys.exit(run_diffoscope(parsed_args))
>  File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 677, 
> in run_diffoscope
>difference = load_diff_from_path(path1)
>  File 
> "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", line 
> 31, in load_diff_from_path
>return load_diff(codecs.getreader("utf-8")(fp), path)
>  File 
> "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", line 
> 35, in load_diff
>return JSONReaderV1().load(fp, path)
>  File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/json.py", 
> line 33, in load
>raw = json.load(fp)
>  File "/usr/local/lib/python3.7/json/__init__.py", line 293, in load
>return loads(fp.read(),
>  File "/usr/local/lib/python3.7/codecs.py", line 504, in read
>newchars, decodedbytes = self.decode(data, self.errors)
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: 
> invalid start byte
> 

Well, the list of differing files is huge. But this seems to
be .gnu_debuglink content for the area it is in. I'll note
that I did installworld but not the likes of distrib-dirs
or distribution this time.

This test did buildworld to two distinct directories:

zroot/BUILDs/13_0R-CA72-nodbg-clang   5.13G   118G 5.13G  
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang
zroot/BUILDs/13_0R-CA72-nodbg-clang-alt   4.28G   118G 4.28G  
/usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt

and installworld to 2 distinct directories:

zroot/DESTDIRs/13_0R-CA72-instwrld-alt1.44G   118G 1.44G  
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt
zroot/DESTDIRs/13_0R-CA72-instwrld-norm   1.44G   118G 1.44G  
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm

Previously (armv7 target) I had built, installed, rebuilt
to same directory (after clean-out) and installed to an
alternate directory. That had gotten only a few files
different but I do not know (yet) if it was the procedural
difference that made the difference.

Prefix of the list of different files this time:

# diff -rq /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/ 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/ | more
Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/[ and 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/[ differ
Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat and 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat differ
Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags and 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags differ
Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio and 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio differ
. . .

Looking, aarch64 seems to typically get a back-to-back
sequence of 4 bytes different in native programs in my
builds:

# cmp -x /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat
3bd4 1d 65
3bd5 eb a3
3bd6 bb ca
3bd7 8e 1a

# ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat
-r-xr-xr-x  1 root  wheel  18448 May  4 08:55:01 2021 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat
-r-xr-xr-x  1 root  wheel  18448 May  3 23:16:36 2021 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat

Sections:
Idx Name  Size  VMA   LMA   File off  Algn
. . .
 25 .gnu_debuglink 0010      3bc8  2**0
  CONTENTS, READONLY

3bd4-3bc8 == 0xC

# cmp -x /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags
2208 88 a1
2209 e6 40
220a 60 94
220b bf ce

# ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags
-r-xr-xr-x  1 root  wheel  

Re: FYI: WITH_REPRODUCIBLE_BUILD= problem for some files?

2021-05-04 Thread Ed Maste
On Tue, 4 May 2021 at 11:52, Mark Millard  wrote:
>
> > As far as the utf-8 issues go, diffoscope requires a utf-8 locale and
> > I suspect that is the issue. If you don't have LANG set already, try
> > setting LANG=C.UTF-8 in your environment.
>
> That is not the issue for the UnicodeDecodeError:
>
> # echo $LANG
> C.UTF-8
>
> # diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
> [...]
> $<3/>2021-05-04 08:49:21 W: diffoscope.main: Fuzzy-matching is currently 
> disabled as the "tlsh" module is unavailable.
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: 
> invalid start byte

Hmm, interesting - if you don't mind sharing I'd be interested in a
copy of /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh, in order to track
down what appears to be a diffoscope issue.

To investigate the non-reproducibility though we can just manually run
through the same sort of process that Diffoscope uses. I would suggest
cmp -x   to find the offsets of the difference(s), then
use readelf -S  to determine which section(s) have differences.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FYI: WITH_REPRODUCIBLE_BUILD= problem for some files?

2021-05-04 Thread Mark Millard via freebsd-current



On 2021-May-4, at 06:01, Ed Maste  wrote:

> On Mon, 3 May 2021 at 22:26, Mark Millard  wrote:
>> 
>> But I'll note that I've built and stalled py37-diffoscope
>> (new to me). A basic quick test showed that it reports:
>> 
>> W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" 
>> module is unavailable.
> 
> I just looked up tlsh - its "A Locality Sensitive Hash"; I presume
> diffoscope uses it to infer file renames. I believe the warning
> emitted here should have no impact on the output we're looking for.

Okay.

> As far as the utf-8 issues go, diffoscope requires a utf-8 locale and
> I suspect that is the issue. If you don't have LANG set already, try
> setting LANG=C.UTF-8 in your environment.

That is not the issue for the UnicodeDecodeError:

# echo $LANG
C.UTF-8

# diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
$<3/>2021-05-04 08:49:21 W: diffoscope.main: Fuzzy-matching is currently 
disabled as the "tlsh" module is unavailable.
$<3/>Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 745, 
in main
sys.exit(run_diffoscope(parsed_args))
  File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 677, 
in run_diffoscope
difference = load_diff_from_path(path1)
  File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", 
line 31, in load_diff_from_path
return load_diff(codecs.getreader("utf-8")(fp), path)
  File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", 
line 35, in load_diff
return JSONReaderV1().load(fp, path)
  File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/json.py", 
line 33, in load
raw = json.load(fp)
  File "/usr/local/lib/python3.7/json/__init__.py", line 293, in load
return loads(fp.read(),
  File "/usr/local/lib/python3.7/codecs.py", line 504, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: 
invalid start byte

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FYI: WITH_REPRODUCIBLE_BUILD= problem for some files?

2021-05-04 Thread Ed Maste
On Mon, 3 May 2021 at 22:26, Mark Millard  wrote:
>
> But I'll note that I've built and stalled py37-diffoscope
> (new to me). A basic quick test showed that it reports:
>
> W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module 
> is unavailable.

I just looked up tlsh - its "A Locality Sensitive Hash"; I presume
diffoscope uses it to infer file renames. I believe the warning
emitted here should have no impact on the output we're looking for.

As far as the utf-8 issues go, diffoscope requires a utf-8 locale and
I suspect that is the issue. If you don't have LANG set already, try
setting LANG=C.UTF-8 in your environment.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Problem compiling harfbuzz

2021-05-04 Thread Herbert J. Skuhra
On Tue, 04 May 2021 10:20:24 +0200, Filippo Moretti via freebsd-current wrote:
> 
> Good morning,    harfbuzz did not compile with he 
> following error:  File 
> "/usr/local/lib/python3.8/site-packages/mesonbuild/modules/gnome.py", line 
> 449, in _get_dependencies_flags
>     if use_gir_args and self._gir_has_option('--extra-library'):
>   File "/usr/local/lib/python3.8/site-packages/mesonbuild/modules/gnome.py", 
> line 498, in _gir_has_option
>     p, o, e = Popen_safe(exe.get_command() + ['--help'], 
> stderr=subprocess.STDOUT)
>   File 
> "/usr/local/lib/python3.8/site-packages/mesonbuild/mesonlib/universal.py", 
> line 1320, in Popen_safe
>     p = subprocess.Popen(args, universal_newlines=True, close_fds=False,
>   File "/usr/local/lib/python3.8/subprocess.py", line 858, in __init__
>     self._execute_child(args, executable, preexec_fn, close_fds,
>   File "/usr/local/lib/python3.8/subprocess.py", line 1704, in _execute_child
>     raise child_exception_type(errno_num, err_msg, err_filename)
> FileNotFoundError: [Errno 2] No such file or directory: 
> '/usr/local/bin/g-ir-scanner'
> ===>  Script "configure" failed unexpectedly.
> Please report the problem to desk...@freebsd.org [maintainer] and attach the
> "/usr/ports/print/harfbuzz/work/harfbuzz-2.8.1/_build/meson-logs/meson-log.txt"
> including the output of the failure of your make command. Also, it might be
> a good idea to provide an overview of all packages installed

Have you tried to (re)build devel/gobject-introspection? I think
ports list is more appropriate.

--
Herbert
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Problem compiling harfbuzz

2021-05-04 Thread Filippo Moretti via freebsd-current
Good morning,    harfbuzz did not compile with he following 
error:  File 
"/usr/local/lib/python3.8/site-packages/mesonbuild/modules/gnome.py", line 449, 
in _get_dependencies_flags
    if use_gir_args and self._gir_has_option('--extra-library'):
  File "/usr/local/lib/python3.8/site-packages/mesonbuild/modules/gnome.py", 
line 498, in _gir_has_option
    p, o, e = Popen_safe(exe.get_command() + ['--help'], 
stderr=subprocess.STDOUT)
  File 
"/usr/local/lib/python3.8/site-packages/mesonbuild/mesonlib/universal.py", line 
1320, in Popen_safe
    p = subprocess.Popen(args, universal_newlines=True, close_fds=False,
  File "/usr/local/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 
'/usr/local/bin/g-ir-scanner'
===>  Script "configure" failed unexpectedly.
Please report the problem to desk...@freebsd.org [maintainer] and attach the
"/usr/ports/print/harfbuzz/work/harfbuzz-2.8.1/_build/meson-logs/meson-log.txt"
including the output of the failure of your make command. Also, it might be
a good idea to provide an overview of all packages installed 

My setup:FreeBSD STING FreeBSD STING 14.0-CURRENT FreeBSD 14.0-CURRENT #24 
main-n246214-78ffcb86d98: Tue Apr 20 17:51:50 CEST 2021 
root@STING:/usr/obj/usr/src/amd64.amd64/sys/STING  amd64

regardsFilippo


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FYI: WITH_REPRODUCIBLE_BUILD= problem for some files?

2021-05-04 Thread Mark Millard via freebsd-current



On 2021-May-3, at 21:27, Mark Millard  wrote:

> On 2021-May-3, at 19:26, Mark Millard  wrote:
> 
>> On 2021-May-3, at 10:51, Mark Millard  wrote:
>> 
>>> On 2021-May-3, at 07:47, Ed Maste  wrote:
>>> 
 On Thu, 29 Apr 2021 at 02:50, Mark Millard via freebsd-current
  wrote:
> 
> Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/sbin/ping and 
> /usr/obj/DESTDIRs/13_0R-CA7-poud/sbin/ping differ
> Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/sbin/ping6 and 
> /usr/obj/DESTDIRs/13_0R-CA7-poud/sbin/ping6 differ
> Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/bin/ntpq and 
> /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/bin/ntpq differ...
 
 This is unexpected. Unfortunately I haven't looked at reproducibility
 in a while, and my work was all on x86. This could be a regression or
 a longstanding issue with arm64.
 
 If you install the diffoscope package (py37-diffoscope) and run it on
 the two directories / files it should give a more convenient view of
 the differences. (Or, if you can make a tarball of the differing files
 I can take a look.)
>>> 
>>> I no longer have the same content in those directory
>>> trees: newer rebuild and the same buildworld used to
>>> installworld to both places, instead of 2 different
>>> buildworld's. I'm also unsure how reproducible getting
>>> differences was.
>>> 
>>> I can eventually do experiments to test multiple separate
>>> buildworld's and installworld's, but the machine is busy
>>> building ports and the llvm builds involved means it
>>> will be some time before I'd switch activities. And the
>>> buildworld's involve llvm builds as well and take notable
>>> time themselves. So my next comparison will not be any
>>> time soon.
>>> 
>>> I'll let you know if I manage to generate another example,
>>> this time being sure to keep the data. If I try multiple
>>> times without finding any differences, I'll eventually
>>> decide "enough is enough" and let you know.
>> 
>> I've still got a long ways to go to do the first
>> actual comparison of builds.
>> 
>> But I'll note that I've built and stalled py37-diffoscope
>> (new to me). A basic quick test showed that it reports:
>> 
>> W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" 
>> module is unavailable.
>> 
>> As I'm not familiar with the tool, you might need to send
>> notes about how you want me to use the tool to get the
>> output that you would want. (And, so, I get to learn . . .)
> 
> I've tried another experiment (* in the path matches "28" and "30"):
> 
> # diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
> $<3/>2021-05-03 21:08:48 W: diffoscope.main: Fuzzy-matching is currently 
> disabled as the "tlsh" module is unavailable.
> $<3/>Traceback (most recent call last):
>  File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 745, 
> in main
>sys.exit(run_diffoscope(parsed_args))
>  File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 677, 
> in run_diffoscope
>difference = load_diff_from_path(path1)
>  File 
> "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", line 
> 31, in load_diff_from_path
>return load_diff(codecs.getreader("utf-8")(fp), path)
>  File 
> "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", line 
> 35, in load_diff
>return JSONReaderV1().load(fp, path)
>  File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/json.py", 
> line 33, in load
>raw = json.load(fp)
>  File "/usr/local/lib/python3.7/json/__init__.py", line 293, in load
>return loads(fp.read(),
>  File "/usr/local/lib/python3.7/codecs.py", line 504, in read
>newchars, decodedbytes = self.decode(data, self.errors)
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: 
> invalid start byte
> 
> The two older snapshots of a Boot Environment have
> bin/sh files that compare equal. But every program I
> tried the above sort of thing against on got the same
> UnicodeDecodeError result from diffoscope, byte value
> and position matching.
> 
> These snapshots have more than an installworld in them
> and so are messy to compare overall. But the
> installworld (and installkernel) content show similar
> differences to what I reported before as far as
> example files with differences go. But this is aarch64,
> not armv7.
> 
> It will still be notable time before I have simple
> installworld tree's to compare.

While waiting for the 2nd buildworld to complete, I used
the 1st buildworld's materials to installworld twice (to
different, empty directory trees) and then did a diff.
The diff reported:

# diff -rq /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/ 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm2/ | more
diff: /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/local: No such file 
or directory
diff: 
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/sys/pjdfstest/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests