Re: KLD zfs.ko: depends on kernel - not available or version mismatch
> On 9 Dec 2020, at 3:48, John Kennedy wrote: > >> I had to copy over several files from /etc and /usr/local/etc and >> re-installed the most important packages. This was admittedly a bit messy, >> it is possible that I forgot to copy something over. >> (Originally my intention was to dd the contents of the spinning disk over, >> but apparently that disk has a few wonky sectors, dd failed after a few >> device timeouts) > > ... so, no guarantee that things are totally sane. > > The "sane" we're looking for is how you can presumably be booting a kernel > located at /boot/kernel/kernel and not have it match the kernel modules > found under /boot/kernel. (...) > What I have built in my source tree is the kernel/zfs module I'd expect: > > # md5 -r /usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel > /usr/obj/usr/src/amd64.amd64/sys/GENERIC/modules/usr/src/sys/modules/zfs/zfs.ko > /boot/kernel/kernel /boot/kernel/zfs.ko | sort > 941ab52d075e444da6eea7fb56213e10 /boot/kernel/kernel > 941ab52d075e444da6eea7fb56213e10 > /usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel > 97d4e0c8ffed1f75e924bf8768a95ff1 /boot/kernel/zfs.ko > 97d4e0c8ffed1f75e924bf8768a95ff1 > /usr/obj/usr/src/amd64.amd64/sys/GENERIC/modules/usr/src/sys/modules/zfs/zfs.ko > > What are you seeing after your installkernel equivalent? It turns out that I was being fooled by the BIOS. Even though I selected the device that this kernel and modules were on as the device to boot from, the actual kernel was still coming from the old spinning disk! It would probably have taken me significantly longer to figure that out without your hints, as I was trying to solve the wrong problem. So thanks a lot for that. Having things to verify was a tremendous help. I used the opportunity to switch to EFI booting, which took me the better part of the evening, but that's working now and booting the correct kernel. It’s even booting into a 1280x720 resolution with the help of the drm-devel-kmod. The next challenge is getting Xorg to run on this Navi-10 GPU; so far I get stuck with "[KMS] drm report modesetting isn't supported”. > Your hashes won't match mine due to non-reproducible build. > > I'd make sure you don't have anything in /boot/modules or otherwise load any > extra modules until sanity is restored (just to reduce random variables). Ah yes, I wasn’t aware of /boot/modules. Last time I used CURRENT, modules were still in the kernel directory. Hence, that was also where I pointed kldload to to test my modules, which explains part of the confusion (and there’s no modules.old…). Alban Hertroys -- There is always an exception to always. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: KLD zfs.ko: depends on kernel - not available or version mismatch
On Tue, Dec 08, 2020 at 07:10:26PM +0100, Alban Hertroys wrote: > > You didn't say that you've installed the new kernel, which at least starts > > you down the road towards a driver/kernel mismatch. You presumably have a > > non-ZFS boot+root. > > I???m fairly sure I did, actually. > > Last time I checked, "make buildworld buildkernel" was equivalent to "make > buildworld && make buildkernel", while "make kernel??? is a shorthand for > ???make buildkernel && make installkernel??? > > So, unless I???m mistaken, ???make buildworld kernel??? should be equivalent > to your first two lines. > > Nevertheless, I retried without these assumptions, the result was the same. I > forgot to ???make delete-old??? though, I rarely remember to do that??? Ah, the dangers of command syntax being close to human syntax. You're trying to do the right thing, so maybe we can sanity check that. > I had to copy over several files from /etc and /usr/local/etc and > re-installed the most important packages. This was admittedly a bit messy, it > is possible that I forgot to copy something over. > (Originally my intention was to dd the contents of the spinning disk over, > but apparently that disk has a few wonky sectors, dd failed after a few > device timeouts) ... so, no guarantee that things are totally sane. The "sane" we're looking for is how you can presumably be booting a kernel located at /boot/kernel/kernel and not have it match the kernel modules found under /boot/kernel. The fact that it is happy with the old kernel modules (presumably under found in /boot/kernel.old) may be a red herring if they're just compatible enough. I can see what I'm expecting to boot here: # grep -E 'boot\/kernel|f7b0aedd1c50' /var/log/messages | tail -2 Dec 6 08:59:04 ouroboros syslogd: kernel boot file is /boot/kernel/kernel Dec 6 08:59:04 ouroboros kernel: FreeBSD 13.0-CURRENT #237 r368388+f7b0aedd1c50-c273383(master): Sun Dec 6 08:27:47 PST 2020 So, I build my system with WITHOUT_REPRODUCIBLE_BUILD=YES in /etc/src.conf, so I can easily see my build version with uname -v: FreeBSD 13.0-CURRENT #237 r368388+f7b0aedd1c50-c273383(master): ... That matches my source tree: # git log -n1 /usr/src | grep revision svn path=/head/; revision=368388 (I've always used git for my sources, but I'm sure there is a svn equivalent.) The version I'm running is what and where I'd expect it to be: # strings -a < /boot/kernel/kernel | grep 'FreeBSD 13' | tail -1 FreeBSD 13.0-CURRENT #237 r368388+f7b0aedd1c50-c273383(master): Sun Dec 6 08:27:47 PST 2020 It certainly isn't the previous kernel: # strings -a < /boot/kernel.old/kernel | grep 'FreeBSD 13' | tail -1 FreeBSD 13.0-CURRENT #236 r368353+0252bfaea893-c273359(master): Fri Dec 4 16:55:41 PST 2020 Not sure what that'll look like with reproducible builds. The hash-check below is a decent stamp, in case the timestamps in /boot/kernel are misleading. What I have built in my source tree is the kernel/zfs module I'd expect: # md5 -r /usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel /usr/obj/usr/src/amd64.amd64/sys/GENERIC/modules/usr/src/sys/modules/zfs/zfs.ko /boot/kernel/kernel /boot/kernel/zfs.ko | sort 941ab52d075e444da6eea7fb56213e10 /boot/kernel/kernel 941ab52d075e444da6eea7fb56213e10 /usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel 97d4e0c8ffed1f75e924bf8768a95ff1 /boot/kernel/zfs.ko 97d4e0c8ffed1f75e924bf8768a95ff1 /usr/obj/usr/src/amd64.amd64/sys/GENERIC/modules/usr/src/sys/modules/zfs/zfs.ko What are you seeing after your installkernel equivalent? Your hashes won't match mine due to non-reproducible build. I'd make sure you don't have anything in /boot/modules or otherwise load any extra modules until sanity is restored (just to reduce random variables). ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: KLD zfs.ko: depends on kernel - not available or version mismatch
On Dec 8, 2020, at 10:10 AM, Alban Hertroys wrote: > > So I tried again to move to HEAD: > > cd /usr/src > svn up > make buildworld -j12 > make buildkernel -j12 > make installkernel > shutdown -r now > > mount -u / > zpool import -Nf system (my /usr FS) > > KLD zfs.ko: depends on kernel - not available or version mismatch > linker_load_file: /boot/kernel/zfs.ko - unsupported file type > cd /usr/obj/usr/src/amd64.amd64/sys/GENERIC ls -l kernel /boot/kernel/kernel ls -l modules/usr/src/sys/modules/zfs/zfs.ko /boot/kernel/zfs.ko cmp modules/usr/src/sys/modules/zfs/zfs.ko /boot/kernel/zfs.ko The kernel should be *older* than zfs.ko and both instances of zfs.ko should be identical. One other thought is to manually do kldload opensolaris.ko zfs.ko In single user mode before using zpool. I have opensolaris_load="YES" zfs_load="YES" in /boot/loader.conf but you may not want to add those until you know zfs works. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: KLD zfs.ko: depends on kernel - not available or version mismatch
On Tue, 8 Dec 2020 19:10:26 +0100 Alban Hertroys wrote: > > On 8 Dec 2020, at 16:40, John Kennedy wrote: > > > > On Tue, Dec 08, 2020 at 08:56:25AM +0100, Alban Hertroys wrote: > >> This seems to have gotten lost in the moderate queue, but after a > >> week I am no closer to a solution, so here???s a resend: > >> > >> I???ve been trying to get a fresh world running (for the eventual > >> purpose of running amdgpu against my recent graphics adapter), but > >> I run into trouble with core loadable kernel modules, such as > >> zfs.ko from the subject. It also happens with other modules that I > >> tried randomly, for example, geom_mirror.ko. > >> > >> I updated to the latest current using svn up in /usr/src, then: > >>make clean > >>make buildworld kernel -j12 > >>shutdown -r now > >> > >> boot to single user mode > >> > >>kldload zfs > > > > I'm not sure you've provided enough information for a one-shot > > armchair diagnosis, but some things seem factually wrong. For > > example, my normal rebuild procedure is: > > > > cd /usr/src && make buildworld && make buildkernel > > make installkernel > > shutdown -r now > > > > cd /usr/src && mergemaster -pi > > make installworld > > mergemaster -Fi > > make -DBATCH_DELETE_OLD_FILES delete-old > > Aha! So that’s how to prevent having to press ‘y’ for every > deprecated file! > > > shutdown -r now > > > > cd /usr/src && make -DBATCH_DELETE_OLD_FILES delete-old-libs > > > > (I'm on a desktop system here. You haven't described your setup.) > > This is also a desktop system. > > > You didn't say that you've installed the new kernel, which at least > > starts you down the road towards a driver/kernel mismatch. You > > presumably have a non-ZFS boot+root. > > I’m fairly sure I did, actually. > > Last time I checked, "make buildworld buildkernel" was equivalent to > "make buildworld && make buildkernel", while "make kernel” is a > shorthand for “make buildkernel && make installkernel” > > So, unless I’m mistaken, “make buildworld kernel” should be > equivalent to your first two lines. > > Nevertheless, I retried without these assumptions, the result was the > same. I forgot to “make delete-old” though, I rarely remember to do > that… > > > Did you mess around with the ZFS from ports (ZoL -> ZoF) > > at some point so you're not using the kernel's ZFS drivers? What > > ZFS entries do you have in /etc/loader.conf, /etc/rc.conf, and some > > of the varients that may get dragged in? (see rc.conf(5) for > > possibilities) > > Nope, stock modules here. > > > At the bottom of your email, you say / is UFS and /usr is ZFS, but > > I guess we have the extra fun of wondering what is under /usr on > > your /? If you have a pre-ZFS /usr that is populated by something > > now presumably very old (because all the new, current stuff went > > onto ZFS /usr, now unavailable). > > There is no populated directory /usr on the UFS file-system. This > install was created on a fresh NVME disk based on an existing install > on a spinning platter. The install happened with /usr mounted at the > ZFS file-system. > > I had to copy over several files from /etc and /usr/local/etc and > re-installed the most important packages. This was admittedly a bit > messy, it is possible that I forgot to copy something over. > (Originally my intention was to dd the contents of the spinning disk > over, but apparently that disk has a few wonky sectors, dd failed > after a few device timeouts) > > > I did sort-of manage to fix things, but recent kernels keep causing > the same issue: > > I noticed that uname -a said I was at revision 366335, while I had > the source tree up-to-date. For a test, I reverted back to that > revision and went through: make buildworld make buildkernel > > Which broke on /usr/local/sys/drm-current-kmod, which I turned out to > have installed through pkg. There have been changes to the linux_kpi > shortly after above revision - probably what broke compatibility > between HEAD and r366335. > > After removing that pkg, the kernel built and installed, world > installed fine too and I have a working system again, with kernel and > world in sync. > > So I tried again to move to HEAD: > > cd /usr/src > svn up > make buildworld -j12
Re: KLD zfs.ko: depends on kernel - not available or version mismatch
> On 8 Dec 2020, at 16:40, John Kennedy wrote:
>
> On Tue, Dec 08, 2020 at 08:56:25AM +0100, Alban Hertroys wrote:
>> This seems to have gotten lost in the moderate queue, but after a week I am
>> no closer to a solution, so here???s a resend:
>>
>> I???ve been trying to get a fresh world running (for the eventual purpose of
>> running amdgpu against my recent graphics adapter), but I run into trouble
>> with core loadable kernel modules, such as zfs.ko from the subject. It also
>> happens with other modules that I tried randomly, for example,
>> geom_mirror.ko.
>>
>> I updated to the latest current using svn up in /usr/src, then:
>> make clean
>> make buildworld kernel -j12
>> shutdown -r now
>>
>> boot to single user mode
>>
>> kldload zfs
>
> I'm not sure you've provided enough information for a one-shot armchair
> diagnosis, but some things seem factually wrong. For example, my normal
> rebuild procedure is:
>
> cd /usr/src && make buildworld && make buildkernel
> make installkernel
> shutdown -r now
>
> cd /usr/src && mergemaster -pi
> make installworld
> mergemaster -Fi
> make -DBATCH_DELETE_OLD_FILES delete-old
Aha! So that’s how to prevent having to press ‘y’ for every deprecated file!
> shutdown -r now
>
> cd /usr/src && make -DBATCH_DELETE_OLD_FILES delete-old-libs
>
> (I'm on a desktop system here. You haven't described your setup.)
This is also a desktop system.
> You didn't say that you've installed the new kernel, which at least starts
> you down the road towards a driver/kernel mismatch. You presumably have a
> non-ZFS boot+root.
I’m fairly sure I did, actually.
Last time I checked, "make buildworld buildkernel" was equivalent to "make
buildworld && make buildkernel", while "make kernel” is a shorthand for “make
buildkernel && make installkernel”
So, unless I’m mistaken, “make buildworld kernel” should be equivalent to your
first two lines.
Nevertheless, I retried without these assumptions, the result was the same. I
forgot to “make delete-old” though, I rarely remember to do that…
> Did you mess around with the ZFS from ports (ZoL -> ZoF)
> at some point so you're not using the kernel's ZFS drivers? What ZFS
> entries do you have in /etc/loader.conf, /etc/rc.conf, and some of the
> varients that may get dragged in? (see rc.conf(5) for possibilities)
Nope, stock modules here.
> At the bottom of your email, you say / is UFS and /usr is ZFS, but I guess we
> have the extra fun of wondering what is under /usr on your /? If you have a
> pre-ZFS /usr that is populated by something now presumably very old (because
> all the new, current stuff went onto ZFS /usr, now unavailable).
There is no populated directory /usr on the UFS file-system. This install was
created on a fresh NVME disk based on an existing install on a spinning
platter. The install happened with /usr mounted at the ZFS file-system.
I had to copy over several files from /etc and /usr/local/etc and re-installed
the most important packages. This was admittedly a bit messy, it is possible
that I forgot to copy something over.
(Originally my intention was to dd the contents of the spinning disk over, but
apparently that disk has a few wonky sectors, dd failed after a few device
timeouts)
I did sort-of manage to fix things, but recent kernels keep causing the same
issue:
I noticed that uname -a said I was at revision 366335, while I had the source
tree up-to-date. For a test, I reverted back to that revision and went through:
make buildworld
make buildkernel
Which broke on /usr/local/sys/drm-current-kmod, which I turned out to have
installed through pkg. There have been changes to the linux_kpi shortly after
above revision - probably what broke compatibility between HEAD and r366335.
After removing that pkg, the kernel built and installed, world installed fine
too and I have a working system again, with kernel and world in sync.
So I tried again to move to HEAD:
cd /usr/src
svn up
make buildworld -j12
make buildkernel -j12
make installkernel
shutdown -r now
mount -u /
zpool import -Nf system (my /usr FS)
KLD zfs.ko: depends on kernel - not available or version mismatch
linker_load_file: /boot/kernel/zfs.ko - unsupported file type
>> Which results in dmesg messages:
>>
>> KLD zfs.ko: depends on kernel - not available or version mismatch
>> linker_load_file: /boot/kernel/zfs.ko - unsupported file type
>> KLD zfs.ko: depends on kernel - not available or version mismatch
>> l
Re: KLD zfs.ko: depends on kernel - not available or version mismatch
On Tue, Dec 08, 2020 at 08:56:25AM +0100, Alban Hertroys wrote: > This seems to have gotten lost in the moderate queue, but after a week I am > no closer to a solution, so here???s a resend: > > I???ve been trying to get a fresh world running (for the eventual purpose of > running amdgpu against my recent graphics adapter), but I run into trouble > with core loadable kernel modules, such as zfs.ko from the subject. It also > happens with other modules that I tried randomly, for example, geom_mirror.ko. > > I updated to the latest current using svn up in /usr/src, then: > make clean > make buildworld kernel -j12 > shutdown -r now > > boot to single user mode > > kldload zfs I'm not sure you've provided enough information for a one-shot armchair diagnosis, but some things seem factually wrong. For example, my normal rebuild procedure is: cd /usr/src && make buildworld && make buildkernel make installkernel shutdown -r now cd /usr/src && mergemaster -pi make installworld mergemaster -Fi make -DBATCH_DELETE_OLD_FILES delete-old shutdown -r now cd /usr/src && make -DBATCH_DELETE_OLD_FILES delete-old-libs (I'm on a desktop system here. You haven't described your setup.) You didn't say that you've installed the new kernel, which at least starts you down the road towards a driver/kernel mismatch. You presumably have a non-ZFS boot+root. Did you mess around with the ZFS from ports (ZoL -> ZoF) at some point so you're not using the kernel's ZFS drivers? What ZFS entries do you have in /etc/loader.conf, /etc/rc.conf, and some of the varients that may get dragged in? (see rc.conf(5) for possibilities) At the bottom of your email, you say / is UFS and /usr is ZFS, but I guess we have the extra fun of wondering what is under /usr on your /? If you have a pre-ZFS /usr that is populated by something now presumably very old (because all the new, current stuff went onto ZFS /usr, now unavailable). > Which results in dmesg messages: > > KLD zfs.ko: depends on kernel - not available or version mismatch > linker_load_file: /boot/kernel/zfs.ko - unsupported file type > KLD zfs.ko: depends on kernel - not available or version mismatch > linker_load_file: /boot/kernel/zfs.ko - unsupported file type > KLD zfs.ko: depends on kernel - not available or version mismatch > linker_load_file: /boot/kernel/zfs.ko - unsupported file type > KLD zfs.ko: depends on kernel - not available or version mismatch > linker_load_file: /boot/kernel/zfs.ko - unsupported file type Be sure to check out /var/log/messages for extra issues. For example, with the bug I mentioned below, I couldn't load my nvidia driver and that manifested as one driver having issues because it depended on another, which had the root of the problem. > I can load the zfs kernel module from kernel.old just fine: > > ZFS filesystem version: 5 > ZFS storage pool version: features support (5000) I kicked my more bleeding-edge system over from 12.2-rel (r366954) up into 13.0-current (r367044, 1300123) on 2020/10/26. OpenZFS kicked in 2020/8/24? I think the CFT was ~2018/8/21, not sure when we had the OpenZFS ports. Current bumps the ABI version pretty frequently so I'd think you'd have tripped across versioning issues a long time ago if you had some drivers not being rebuilt. > This happens with any kernel module I???ve tried, such as geom_mirror and > amdgpu (from ports/graphics/drm-current-kmod - the latter causes a kernel > panic with kernel.old BTW). > > I???ve gone back as far as Oct 7 (before changes to kern/elf_load_obj.c off > the top of my head), looked at mailing list archives and forums etc, all to > no avail. > > I have / on UFS+J and /usr on ZFS and nothing in /etc/src.conf. I had > /etc/malloc.conf with the recommended symlink from UPDATING, but the same > happens with that moved out of the way. Nothing seems to help. > > Do I need to go back further to get into a usable state or is there something > else I should be doing? With very few exceptions (bug 250897, 2020/11/6), I've found 13-current bootable since 10/26 (up through my current system, 13.0 r368388 (2020/12/6). You obviously need to make sure that an extra drivers you add in are compiled against the kernel, but ZFS is typically one of those. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
KLD zfs.ko: depends on kernel - not available or version mismatch
This seems to have gotten lost in the moderate queue, but after a week I am no closer to a solution, so here’s a resend: I’ve been trying to get a fresh world running (for the eventual purpose of running amdgpu against my recent graphics adapter), but I run into trouble with core loadable kernel modules, such as zfs.ko from the subject. It also happens with other modules that I tried randomly, for example, geom_mirror.ko. I updated to the latest current using svn up in /usr/src, then: make clean make buildworld kernel -j12 shutdown -r now boot to single user mode kldload zfs Which results in dmesg messages: KLD zfs.ko: depends on kernel - not available or version mismatch linker_load_file: /boot/kernel/zfs.ko - unsupported file type KLD zfs.ko: depends on kernel - not available or version mismatch linker_load_file: /boot/kernel/zfs.ko - unsupported file type KLD zfs.ko: depends on kernel - not available or version mismatch linker_load_file: /boot/kernel/zfs.ko - unsupported file type KLD zfs.ko: depends on kernel - not available or version mismatch linker_load_file: /boot/kernel/zfs.ko - unsupported file type I can load the zfs kernel module from kernel.old just fine: ZFS filesystem version: 5 ZFS storage pool version: features support (5000) This happens with any kernel module I’ve tried, such as geom_mirror and amdgpu (from ports/graphics/drm-current-kmod - the latter causes a kernel panic with kernel.old BTW). I’ve gone back as far as Oct 7 (before changes to kern/elf_load_obj.c off the top of my head), looked at mailing list archives and forums etc, all to no avail. I have / on UFS+J and /usr on ZFS and nothing in /etc/src.conf. I had /etc/malloc.conf with the recommended symlink from UPDATING, but the same happens with that moved out of the way. Nothing seems to help. Do I need to go back further to get into a usable state or is there something else I should be doing? Regards, Alban Hertroys -- There is always an exception to always. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
KLD zfs.ko: depends on kernel - not available or version mismatch
I’ve been trying to get a fresh world running (for the eventual purpose of running amdgpu against my recent graphics adapter), but I run into trouble with core loadable kernel modules, such as zfs.ko from the subject. It also happens with other modules that I tried randomly, for example, geom_mirror.ko. I updated to the latest current using svn up in /usr/src, then: make clean make buildworld kernel -j12 shutdown -r now boot to single user mode kldload zfs Which results in dmesg messages: KLD zfs.ko: depends on kernel - not available or version mismatch linker_load_file: /boot/kernel/zfs.ko - unsupported file type KLD zfs.ko: depends on kernel - not available or version mismatch linker_load_file: /boot/kernel/zfs.ko - unsupported file type KLD zfs.ko: depends on kernel - not available or version mismatch linker_load_file: /boot/kernel/zfs.ko - unsupported file type KLD zfs.ko: depends on kernel - not available or version mismatch linker_load_file: /boot/kernel/zfs.ko - unsupported file type I can load the zfs kernel module from kernel.old just fine: ZFS filesystem version: 5 ZFS storage pool version: features support (5000) This happens with any kernel module I’ve tried, such as geom_mirror and amdgpu (from ports/graphics/drm-current-kmod - the latter causes a kernel panic with kernel.old BTW). I’ve gone back as far as Oct 7 (before changes to kern/elf_load_obj.c off the top of my head), looked at mailing list archives and forums etc, all to no avail. I have / on UFS+J and /usr on ZFS and nothing in /etc/src.conf. I had /etc/malloc.conf with the recommended symlink from UPDATING, but the same happens with that moved out of the way. Nothing seems to help. Do I need to go back further to get into a usable state or is there something else I should be doing? Regards, Alban Hertroys -- There is always an exception to always. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
