> On Aug 9, 2025, at 12:55 AM, Florian Smeets <[email protected]> wrote:
>
>
> FWIW, I revived the system by deleting all newer packages rm
> /var/cache/pkg/*snap20250808* and just untaring the stuff in /var/cache/pkg
> to /
>
> cd / ; for i in `/rescue/ls -1 /var/cache/pkg/FreeBSD-* ` ; do /rescue/tar
> xvzf $i ; done
>
> After that, I resorted to building from source and installing to get the
> system into a half way consistent state with a chance of surviving a reboot.
So, looking at Florian's solution, I came across two issues.
1) After you build from src, pkg thinks you have one thing installed, and
buildworld has installed something different. I cannot figure out how to tell
pkg "delete anything you know from the pkgdb about these pkgs, and just let me
re-run pkgbasify". There's a -O option that exists in netbsd's pkg, and I'm
sure there's some absurd invocation that involves manipulating the raw pkgdb,
but nothing "easy"/
2) Pkgbase got me into this problem, and I wanted it to get me out. I had all
the pkgs sitting there in /var/cache/pkg, ready to go, I just needed to get
them in there. Doing a pkg-info yielded a scant 20 packages or so.
Worse still: /rescue/ls /var/cache/pkg/FreeBSD-* gave me "argument list too
long" and with no grep, I basicaly had to rely on scrollback to get what I was
looking for.
Ergo, my solution was more like:
cd /var/cache
/rescue/cp -pR pkg pkg-backup
mkdir ../pkg-snap20250808125411
cp FreeBSD*snap20250808125411* ../pkg-snap20250808125411
(From there, I used rm cautiously to remove all the -dbg packages -- I can put
them back later, but for right now I needed them to work.
Let's just try one package...
# ABI=FreeBSD:15:amd64 /usr/local/sbin/pkg-static add -M
./FreeBSD-libarchive-15.snap20250808125411.pkg
pkg-static: Setting ABI requires setting OSVERSION, guessing the OSVERSION as:
1500000
Installing FreeBSD-libarchive-15.snap20250808125411...
Newer FreeBSD version for package FreeBSD-libarchive:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1500058
- running userland: 1500000
Ignore the mismatch and continue? [y/N]: y
`-- Installing FreeBSD-runtime-15.snap20250808125411...
| `-- Installing FreeBSD-libarchive-15.snap20250808125411...
| | `-- Installing FreeBSD-runtime-15.snap20250808125411...
| | `-- Installing FreeBSD-libarchive-15.snap20250808125411...
| | | `-- Installing FreeBSD-runtime-15.snap20250808125411...
| | | `-- Installing FreeBSD-libarchive-15.snap20250808125411...
| | | | `-- Installing FreeBSD-runtime-15.snap20250808125411...
| | | | `-- Installing FreeBSD-libarchive-15.snap20250808125411...
^C
(Okay, so there's no "y" option to pkg add... and these things will forever sit
and try to install circular dependencies. Dumb).
New solution: What if we're in a new directory that only has packages in it
that have already been installed?
mkdir /tmp/foo
cd /var/cache/pkg-snap20250808125411/
cp FreeBSD-libarchive-15.snap20250808125411.pkg /tmp/foo
cd /tmp/foo
ABI=FreeBSD:15:amd64 /usr/local/sbin/pkg-static add -M
./FreeBSD-libarchive-15.snap20250808125411.pkg
pkg-static: Setting ABI requires setting OSVERSION, guessing the OSVERSION as:
1500000
Installing FreeBSD-libarchive-15.snap20250808125411...
Newer FreeBSD version for package FreeBSD-libarchive:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1500058
- running userland: 1500000
Ignore the mismatch and continue? [y/N]: y
pkg-static: Missing shlib dependency: libbsdxml.so.4
pkg-static: Missing shlib dependency: libbz2.so.4
pkg-static: Missing shlib dependency: libcrypto.so.30
pkg-static: Missing shlib dependency: liblzma.so.5
pkg-static: Missing shlib dependency: libprivatezstd.so.5
pkg-static: Missing shlib dependency: libz.so.6
Extracting FreeBSD-libarchive-15.snap20250808125411: 100%
Okay, but this will still be exhausting if I need to keep hitting "yes"...
for i in `ls -1 /var/cache/pkg-snap20250808125411/`
> do
> cp /var/cache/pkg-snap20250808125411/$i /tmp/foo
> cd /tmp/foo
> OSVERSION=1500058 ABI=FreeBSD:15:amd64 /usr/local/sbin/pkg-static add -M ./$i
> done
Installing FreeBSD-acct-15.snap20250808125411...
pkg-static: Missing shlib dependency: libxo.so.0
Extracting FreeBSD-acct-15.snap20250808125411: 100%
Installing FreeBSD-acct-man-15.snap20250808125411...
Extracting FreeBSD-acct-man-15.snap20250808125411: 100%
Installing FreeBSD-acpi-15.snap20250808125411...
[...]
Installing FreeBSD-zfs-15.snap20250808125411...
Extracting FreeBSD-zfs-15.snap20250808125411: 100%
Installing FreeBSD-zfs-dev-15.snap20250808125411...
Extracting FreeBSD-zfs-dev-15.snap20250808125411: 100%
Installing FreeBSD-zfs-dev-lib32-15.snap20250808125411...
pkg-static: Missing shlib dependency: libicp.so.3:32
pkg-static: Missing shlib dependency: libzfsbootenv.so.1:32
Extracting FreeBSD-zfs-dev-lib32-15.snap20250808125411: 100%
Installing FreeBSD-zfs-lib32-15.snap20250808125411...
Extracting FreeBSD-zfs-lib32-15.snap20250808125411: 100%
Installing FreeBSD-zfs-man-15.snap20250808125411...
Extracting FreeBSD-zfs-man-15.snap20250808125411: 100%
Installing FreeBSD-zoneinfo-15.snap20250808125411...
...I started a normal shell (/bin/csh) and tried a service command, and sshd
restarted.
But pkg is still being persnickety, and I don't know what the mechanism is to
bootstrap it is:
# ldd /usr/local/sbin/pkg
/usr/local/sbin/pkg:
libelf.so.2 => /lib/libelf.so.2 (0x2a08ac8d1000)
libjail.so.1 => /lib/libjail.so.1 (0x2a08ad283000)
libm.so.5 => /lib/libm.so.5 (0x2a08adcbd000)
libarchive.so.7 => /usr/lib/libarchive.so.7 (0x2a08ae37a000)
libz.so.6 => /lib/libz.so.6 (0x2a08abe01000)
libutil.so.9 => not found (0)
libssl.so.30 => /usr/lib/libssl.so.30 (0x2a08af3bb000)
libcrypto.so.30 => /lib/libcrypto.so.30 (0x2a08af8a2000)
libthr.so.3 => /lib/libthr.so.3 (0x2a08b0575000)
libc.so.7 => /lib/libc.so.7 (0x2a08b19a8000)
libbz2.so.4 => /usr/lib/libbz2.so.4 (0x2a08b257a000)
liblzma.so.5 => /usr/lib/liblzma.so.5 (0x2a08b2d8e000)
libbsdxml.so.4 => /lib/libbsdxml.so.4 (0x2a08b3302000)
libprivatezstd.so.5 => /usr/lib/libprivatezstd.so.5 (0x2a08b3d33000)
libsys.so.7 => /lib/libsys.so.7 (0x2a08b4a90000)
libmd.so.7 => /lib/libmd.so.7 (0x2a08b4f76000)
[vdso] (0x2a08ab287000)
I've got a libutil.so.10, but not 9. Doing a pkg-static install -f pkg doesn't
update it to use .so.10 either. Where does one get this file?
====
I don't have another system with pkgbase on it, so I can't ask what pkg
provides that file. I don't know if, given a list of .pkg files, there's a way
to poll them all and ask. (pkg grep?)
While this is still a prerelease thing -- these recipes should be in the
pkgbase wiki page. Knowing how to recover from a pkgbase failure -- or how to
rectify a world and pkgbase that are out of step with each other, are going to
be things we're going to need to do occasionally.
>
> When starting my upgrade I saw that it wanted to remove a lot of non pkgbase
> packages (how are we doing to differentiate pkgbase packages and "ports"
> packages in the future?). I thought this might be related to the krb5 thing,
> so I created an up to date poudriere jail via pkgbase method and rebuilt all
> my pkgs, but even then I saw the same thing as Den, that pkg wanted to remove
> a lot of ports pkgs, as this system is not important I thought I can resolve
> that after the pkgbase upgrade and started the upgrade.
>
> I didn't save scroll back. In my case I saw at the top the first ~100 pkg
> transactions were uninstalling pkgbase pkgs, then it upgraded some, then pkg
> exited with a segfault.
>
> Leaving me with ELF interpreter /libexec/ld-elf.so.1 not found, error 2
>
> One thing I checked was /libexec/ was completely empty.
>
> Florian
>
>> root@poudriere:/home/dmahoney # pkg-static install -f pkg
>> pkg-static: Unable to determine the ABI, none of the ABI_FILEs can be read.
>> pkg-static: Cannot parse configuration file!
>> root@poudriere:/home/dmahoney # /rescue/sh
>> Cannot read termcap database;
>> using dumb terminal settings.
>> # pkg-static
>> pkg: not enough arguments
>> Usage: pkg [-v] [-d] [-l] [-N] [-j <jail name or id>|-c <chroot path>|-r
>> <rootdir>] [-C <configuration file>] [-R <repo config dir>] [-o var=value]
>> [-4|-6] <command> [<args>]
>> For more information on available commands and options see 'pkg help'.
>> # pkg-static install -f pkg
>> pkg-static: Unable to determine the ABI, none of the ABI_FILEs can be read.
>> Now, if this were 14.x, I'd fix this by untarring a distfile right over / to
>> get back up and running. Not quite an option in 15, is it? (I would
>> encourage the people making this stuff to please consider keeping those
>> built, even if bsdinstall uses pkgbase).
>> This is dayjob's poudriere system -- the build process for it is well
>> documented and the parts of it that aren't managed by puppet are easily
>> managed because I captured every command used to create every jail and ports
>> tree (which was slow because many of them came from freebsd-archive; both
>> old jails and old copies of ports trees last known to work with given
>> versions of FreeBSD (kind of required when you needed to build packages in
>> 2020 because remote hands were Not An Option). Like all our systems, the
>> bits we care about (homedirs, /usr/local/etc) are in backups as well. We
>> kind of need it to work.
>> So this isn't an emergency. This system is really there so that me, as a
>> port maintainer, can build a debug build of something, but this machine
>> isn't in our critical path. ...but What If It Was? We run Critical Stuff,
>> out there on lone servers in faraway places (on bare metal)
>> But it's *really* not instilling me with a lot of confidence in the
>> readiness of this pkgbase idea. (For the record, I've also had
>> freebsd-update leave me dead on the table in similar ways in the past. I
>> literally called them out during a BSDcan talk without trying to bash Colin
>> too hard).
>> I can capture more scrollback if people want, but I wasn't doing any of the
>> crazy -f commands people are talking about. This was literally a "pkg
>> upgrade".
>> Full command output is over at
>> https://users.isc.org/~dmahoney/failedupgrade.txt if devs want to have a
>> look and try to black-box it. I'll keep the VM running (and logged in, in a
>> screen session) if there are things people want me to try.
>> This begs the questions:
>> * Is there CI for pkgbase that tries to upgrade from whatever version is
>> immediately previous to it, before publishing it? (I know that's what
>> version I was running, it's the version everyone's been stuck at for weeks!)
>> * For all the debate about "pkgbase and pkg should be exactly the same",
>> perhaps pkgbase could have an auto-bectl in it?
>> * Is there support somewhere for having a lockstep "set" of packages that
>> one knows are in /var/db/pkg and constitute a relatively concurrent install?
>> (Or a list of files that I could tell pkg-static to install with a glob,
>> out of /var/cache/pkg)
>> * With something like -CURRENT, is there any support for saying "Okay not
>> the current-current tree, but current-minus-one" (I guess that would be if
>> you're working with the weekly builds, but that's not quite the same. Your
>> only option is "base system of the now" but it happens less frequently).
>> * And for running -CURRENT where this kind of breakage can happen, could we
>> get a statically linked version of pkg?
>> Any questions, let me know.
>> -Dan
>> Sent from my iPhone
>
> <OpenPGP_0xEF5BA4DCD5A9F3C0.asc>
