Re: Booting in bhyve always sets currdev to ZFS
On Tue, Jul 2, 2024 at 7:30 AM Rodney W. Grimes wrote: > > > On Sun, Jun 30, 2024 at 8:23?AM Rodney W. Grimes > > wrote: > > > > > > > On Fri, Jun 28, 2024 at 6:26?PM Rick Macklem > > > > wrote: > > > > > > > > > > Hi, > > > > > > > > > > I've installed FreeBSD current in a bhvye instance. Everything > > > > > went ok, with UFS as a root partition. > > > > > Then I created a zpool in another partition... > > > > > - Now, every time I boot it I have to > > > > > OK set currdev=disk0s1a: > > > > > to get it to boot. > > > > > > > > > > What is the trick to keep ZFS from messing up the boot variables? > > > > I've been poking around, but haven't learned much. > > > > I think it is userboot (although there are so many boot programs in > > > > /boot, > > > > I am not 100% sure?) that sets currdev=zfs:example: since it sees there > > > > is ZFS > > > > in a partition on the drive. It is not the boot partition and doesn't > > > > have > > > > any boot stuff in it. > > > > > > > > When I look at userboot, it appears that it always sets > > > > userboot_zfs_found > > > > to 1 whenever userboot_zfs_probe() is called, given that there is a ZFS > > > > partition with a pool on it. This makes extract_currdev get set to the > > > > ZFS stuff, > > > > assuming I am reading the code correctly. > > > > What I do not understand is why I have not seen this before? > > > > (Was the a change to building it with USERBOOT_ZFS_SUPPORT done?) > > > > > > > > It seems that userboot_zfs_probe() should check for boot files on the > > > > volume > > > > and not just that a pool exists on the partition, maybe? > > > > > > > > Anyhow, manually setting currdev=disk0s1a: gets around the problem. > > > > > > You should be able to set that in your ufs partition /boot/loader.conf > > > file and at least then your not having to drop to the OK prompt and > > > manually entering this. > > Doesn't help, because it is not reading files from the boot ufs file system. > > (I could copy all the boot stuff over into the ZFS file system, but that > > kinda > > defeats the purpose of having UFS as the boot fs.) > > It had to of loaded userboot from the ufs, right? I assume it is reading some boot file from UFS. Way back when I thought a primary boot was read from 8K at the begining of the partition, but that's how out of date I on this. -> Put another way, I don't know if "userboot" is the one I am seeing try to use the ZFS partition. > > What type of disk is this? gpt or mbr? It's old MBR. Although I'll assume bhyve knows other stuff, most of my old hardware only does MBR and not EFI, so I'll still in the MBR habit. > > Can you show the output of gpart show and gpart list? root@nfsv4-foo:~ # gpart show => 63 67108801 vtbd0 MBR (32G) 63 1 - free - (512B) 64 67108800 1 freebsd [active] (32G) => 0 67108800 vtbd0s1 BSD (32G) 0 335544321 freebsd-ufs (16G) 33554432 125829122 freebsd-swap (6.0G) 46137344 209714564 freebsd-zfs (10G) root@nfsv4-foo:~ # gpart list Geom name: vtbd0 modified: false state: OK fwheads: 255 fwsectors: 63 last: 67108863 first: 63 entries: 4 scheme: MBR Providers: 1. Name: vtbd0s1 Mediasize: 34359705600 (32G) Sectorsize: 512 Stripesize: 32768 Stripeoffset: 0 Mode: r3w3e5 efimedia: HD(1,MBR,0x90909090,0x40,0x3c0) attrib: active rawtype: 165 length: 34359705600 offset: 32768 type: freebsd index: 1 end: 67108863 start: 64 Consumers: 1. Name: vtbd0 Mediasize: 34359738368 (32G) Sectorsize: 512 Stripesize: 32768 Stripeoffset: 0 Mode: r3w3e8 Geom name: vtbd0s1 modified: false state: OK fwheads: 255 fwsectors: 63 last: 67108799 first: 0 entries: 8 scheme: BSD Providers: 1. Name: vtbd0s1a Mediasize: 17179869184 (16G) Sectorsize: 512 Stripesize: 32768 Stripeoffset: 0 Mode: r1w1e1 rawtype: 7 length: 17179869184 offset: 0 type: freebsd-ufs index: 1 end: 33554431 start: 0 2. Name: vtbd0s1b Mediasize: 6442450944 (6.0G) Sectorsize: 512 Stripesize: 32768 Stripeoffset: 0 Mode: r1w1e0 rawtype: 1 length: 6442450944 offset: 17179869184 type: freebsd-swap index: 2 end: 46137343 start: 33554432 3. Name: vtbd0s1d Mediasize: 10737385472 (10G) Sectorsize: 512 Stripesize: 32768 Stripeoffset: 0 Mode: r1w1e1 rawtype: 27 length: 10737385472 offset: 23622320128 type: freebsd-zfs index: 4 end: 67108799 start: 46137344 Consumers: 1. Name: vtbd0s1 Mediasize: 34359705600 (32G) Sectorsize: 512 Stripesize: 32768 Stripeoffset: 0 Mode: r3w3e5 > > > > > For example, it always fails with: > > ERROR: cannot open /boot/lua/loader.lua: no such file or directory. > > > > When I look in stand/efi/loader/main.c, there is a function that checks > > for boot files called sanity_check_currdev(). However, userboot does not > > have a similar function and just seems to assume
Re: Booting in bhyve always sets currdev to ZFS
> On Sun, Jun 30, 2024 at 8:23?AM Rodney W. Grimes > wrote: > > > > > On Fri, Jun 28, 2024 at 6:26?PM Rick Macklem > > > wrote: > > > > > > > > Hi, > > > > > > > > I've installed FreeBSD current in a bhvye instance. Everything > > > > went ok, with UFS as a root partition. > > > > Then I created a zpool in another partition... > > > > - Now, every time I boot it I have to > > > > OK set currdev=disk0s1a: > > > > to get it to boot. > > > > > > > > What is the trick to keep ZFS from messing up the boot variables? > > > I've been poking around, but haven't learned much. > > > I think it is userboot (although there are so many boot programs in /boot, > > > I am not 100% sure?) that sets currdev=zfs:example: since it sees there > > > is ZFS > > > in a partition on the drive. It is not the boot partition and doesn't have > > > any boot stuff in it. > > > > > > When I look at userboot, it appears that it always sets userboot_zfs_found > > > to 1 whenever userboot_zfs_probe() is called, given that there is a ZFS > > > partition with a pool on it. This makes extract_currdev get set to the > > > ZFS stuff, > > > assuming I am reading the code correctly. > > > What I do not understand is why I have not seen this before? > > > (Was the a change to building it with USERBOOT_ZFS_SUPPORT done?) > > > > > > It seems that userboot_zfs_probe() should check for boot files on the > > > volume > > > and not just that a pool exists on the partition, maybe? > > > > > > Anyhow, manually setting currdev=disk0s1a: gets around the problem. > > > > You should be able to set that in your ufs partition /boot/loader.conf > > file and at least then your not having to drop to the OK prompt and > > manually entering this. > Doesn't help, because it is not reading files from the boot ufs file system. > (I could copy all the boot stuff over into the ZFS file system, but that kinda > defeats the purpose of having UFS as the boot fs.) It had to of loaded userboot from the ufs, right? What type of disk is this? gpt or mbr? Can you show the output of gpart show and gpart list? > > For example, it always fails with: > ERROR: cannot open /boot/lua/loader.lua: no such file or directory. > > When I look in stand/efi/loader/main.c, there is a function that checks > for boot files called sanity_check_currdev(). However, userboot does not > have a similar function and just seems to assume that the ZFS partition > in the bootable one, if it finds one. That sounds like a wrong assumption by the code. > --> I am wondering if sanity_check_currdev() shoud be added to userboot > to handle this case? Warner?? > rick > > > > > There is much magic in boot code that bites, and not just in FreeBSD. > > Often assumptions are made that "this" boot code, and "this" OS are > > the only things on the disk(s). > > > > The fact that Linux uses a "apple-zfs" uuid, and FreeBSD a "FreeBSD-zfs" > > drove me nuts for 2 days until I found this little tid bit. > > > > Zpool import doesnt care, but the boot code does! > > > > -- > > Rod Grimes > > rgri...@freebsd.org -- Rod Grimes rgri...@freebsd.org
Re: Booting in bhyve always sets currdev to ZFS
On 29. Jun 2024, at 03:26, Rick Macklem wrote: > > Hi, > > I've installed FreeBSD current in a bhvye instance. Everything > went ok, with UFS as a root partition. > Then I created a zpool in another partition... > - Now, every time I boot it I have to > OK set currdev=disk0s1a: > to get it to boot. > > What is the trick to keep ZFS from messing up the boot variables? I remember having this problem a few years ago. I see I have in an old startup file, ‘-e “rootdev=disk0”’ to set a loader environment variable. I also recall needing different Bhyve firmware, and I used gptboot. Some of these things might have been for different reasons. Other details are fuzzy. Regards, Jan M.
Re: Booting in bhyve always sets currdev to ZFS
On Sun, Jun 30, 2024 at 8:23 AM Rodney W. Grimes wrote: > > > On Fri, Jun 28, 2024 at 6:26?PM Rick Macklem wrote: > > > > > > Hi, > > > > > > I've installed FreeBSD current in a bhvye instance. Everything > > > went ok, with UFS as a root partition. > > > Then I created a zpool in another partition... > > > - Now, every time I boot it I have to > > > OK set currdev=disk0s1a: > > > to get it to boot. > > > > > > What is the trick to keep ZFS from messing up the boot variables? > > I've been poking around, but haven't learned much. > > I think it is userboot (although there are so many boot programs in /boot, > > I am not 100% sure?) that sets currdev=zfs:example: since it sees there is > > ZFS > > in a partition on the drive. It is not the boot partition and doesn't have > > any boot stuff in it. > > > > When I look at userboot, it appears that it always sets userboot_zfs_found > > to 1 whenever userboot_zfs_probe() is called, given that there is a ZFS > > partition with a pool on it. This makes extract_currdev get set to the > > ZFS stuff, > > assuming I am reading the code correctly. > > What I do not understand is why I have not seen this before? > > (Was the a change to building it with USERBOOT_ZFS_SUPPORT done?) > > > > It seems that userboot_zfs_probe() should check for boot files on the volume > > and not just that a pool exists on the partition, maybe? > > > > Anyhow, manually setting currdev=disk0s1a: gets around the problem. > > You should be able to set that in your ufs partition /boot/loader.conf > file and at least then your not having to drop to the OK prompt and > manually entering this. Doesn't help, because it is not reading files from the boot ufs file system. (I could copy all the boot stuff over into the ZFS file system, but that kinda defeats the purpose of having UFS as the boot fs.) For example, it always fails with: ERROR: cannot open /boot/lua/loader.lua: no such file or directory. When I look in stand/efi/loader/main.c, there is a function that checks for boot files called sanity_check_currdev(). However, userboot does not have a similar function and just seems to assume that the ZFS partition in the bootable one, if it finds one. --> I am wondering if sanity_check_currdev() shoud be added to userboot to handle this case? rick > > There is much magic in boot code that bites, and not just in FreeBSD. > Often assumptions are made that "this" boot code, and "this" OS are > the only things on the disk(s). > > The fact that Linux uses a "apple-zfs" uuid, and FreeBSD a "FreeBSD-zfs" > drove me nuts for 2 days until I found this little tid bit. > > Zpool import doesnt care, but the boot code does! > > > -- > Rod Grimes rgri...@freebsd.org
Re: Booting in bhyve always sets currdev to ZFS
> On Fri, Jun 28, 2024 at 6:26?PM Rick Macklem wrote: > > > > Hi, > > > > I've installed FreeBSD current in a bhvye instance. Everything > > went ok, with UFS as a root partition. > > Then I created a zpool in another partition... > > - Now, every time I boot it I have to > > OK set currdev=disk0s1a: > > to get it to boot. > > > > What is the trick to keep ZFS from messing up the boot variables? > I've been poking around, but haven't learned much. > I think it is userboot (although there are so many boot programs in /boot, > I am not 100% sure?) that sets currdev=zfs:example: since it sees there is ZFS > in a partition on the drive. It is not the boot partition and doesn't have > any boot stuff in it. > > When I look at userboot, it appears that it always sets userboot_zfs_found > to 1 whenever userboot_zfs_probe() is called, given that there is a ZFS > partition with a pool on it. This makes extract_currdev get set to the > ZFS stuff, > assuming I am reading the code correctly. > What I do not understand is why I have not seen this before? > (Was the a change to building it with USERBOOT_ZFS_SUPPORT done?) > > It seems that userboot_zfs_probe() should check for boot files on the volume > and not just that a pool exists on the partition, maybe? > > Anyhow, manually setting currdev=disk0s1a: gets around the problem. You should be able to set that in your ufs partition /boot/loader.conf file and at least then your not having to drop to the OK prompt and manually entering this. There is much magic in boot code that bites, and not just in FreeBSD. Often assumptions are made that "this" boot code, and "this" OS are the only things on the disk(s). The fact that Linux uses a "apple-zfs" uuid, and FreeBSD a "FreeBSD-zfs" drove me nuts for 2 days until I found this little tid bit. Zpool import doesnt care, but the boot code does! -- Rod Grimes rgri...@freebsd.org
Re: Booting in bhyve always sets currdev to ZFS
On Fri, Jun 28, 2024 at 6:26 PM Rick Macklem wrote: > > Hi, > > I've installed FreeBSD current in a bhvye instance. Everything > went ok, with UFS as a root partition. > Then I created a zpool in another partition... > - Now, every time I boot it I have to > OK set currdev=disk0s1a: > to get it to boot. > > What is the trick to keep ZFS from messing up the boot variables? I've been poking around, but haven't learned much. I think it is userboot (although there are so many boot programs in /boot, I am not 100% sure?) that sets currdev=zfs:example: since it sees there is ZFS in a partition on the drive. It is not the boot partition and doesn't have any boot stuff in it. When I look at userboot, it appears that it always sets userboot_zfs_found to 1 whenever userboot_zfs_probe() is called, given that there is a ZFS partition with a pool on it. This makes extract_currdev get set to the ZFS stuff, assuming I am reading the code correctly. What I do not understand is why I have not seen this before? (Was the a change to building it with USERBOOT_ZFS_SUPPORT done?) It seems that userboot_zfs_probe() should check for boot files on the volume and not just that a pool exists on the partition, maybe? Anyhow, manually setting currdev=disk0s1a: gets around the problem. rick > > Thanks, rick
