Re: Re: [current-users] Re: Best practice for setting up disks for ZFS on NetBSD
m...@mjch.net ("Malcolm Herbert") writes:
>As far as I understand it, ZFS vdevs have their own ID, so they can be laid
>out correctly no matter the OS device each are discovered on ... wouldn't that
>make a raidframe wrapper redundant? it would also mean the zpool vdevs
>couldn't be used on other systems that understand ZFS because they're unlikely
>to understand raidframe ...
The device paths are cached, thus the need to re-import. Ideally the cache
would be invalidated automatically, but I was told is not in our code base.
--
--
Michael van Elst
Internet: mlel...@serpens.de
"A potential Snark may lurk in every tree."
Re: Best practice for setting up disks for ZFS on NetBSD
a...@absd.org (David Brownlee) writes: >Aha - *this* is nice. I recall seeing devpubd mentioned a while back >but never took a look. More fool me! >One small issue - rcorder shows rc.d/devpubd running quite late in the >boot process - much later than rc.d/zfs.I wonder if it should be >adjusted to run directly after rc.d/root? That is the plan. The current hooks are shell scripts that require /usr and need to rewritten (probably as tiny C programs). -- -- Michael van Elst Internet: mlel...@serpens.de "A potential Snark may lurk in every tree."
Re: Best practice for setting up disks for ZFS on NetBSD
On Thu, 3 Dec 2020 at 08:31, Tobias Nygren wrote:
>
> On Thu, 3 Dec 2020 00:30:17 + David Brownlee wrote:
> >
> > In the event of disk renumbering both are thrown out, needing a "zfs
> > export foo;zfs import foo" to recover. Is there some way to avoid that?
>
> You can use named gpt wedges and symlink them to a stable path in /dev.
> I did this manually for my storage box setup but was later informed we
> have devpubd(8) which is capable of doing this.
Aha - *this* is nice. I recall seeing devpubd mentioned a while back
but never took a look. More fool me!
One small issue - rcorder shows rc.d/devpubd running quite late in the
boot process - much later than rc.d/zfs.I wonder if it should be
adjusted to run directly after rc.d/root?
On Thu, 3 Dec 2020 at 08:45, Hauke Fath wrote:
>
> On Thu, 3 Dec 2020 00:30:17 +, David Brownlee wrote:
> > - Wedges, setup as a single large gpt partition of type zfs (eg /dev/dk7)
> > - Entire disk (eg: /dev/wd0 or /dev/sd4)
>
> "Traditional" (solarish) zfs lore recommends giving zfs the entire
> disk, unpartitioned, since it can make more efficient use of it then.
> Zfs will generally be able to re-assemble a volume after renumbering
> components - I have seen it do that on OmniOS after swapping an Areca
> RAID controller in JBOD mode out for a plain SAS controller.
That was one of my motives behind preferring the whole disk approach
(in addition to not cluttering up iostats with nearly a dozen
otherwise unwanted dk entries). However, if on NetBSD disk
renumbering can be handled by a wedge setup but not whole disks, then
I'll be switching to wedges...
On Thu, 3 Dec 2020 at 02:47, HIROSE yuuji wrote:
>
> >> On Thu, 3 Dec 2020 00:30:17 + a...@absd.org (David Brownlee) said:
> >
> > What would be the best practice for setting up disks to use under ZFS
> > on NetBSD, with particular reference to handling renumbered devices?
>
> Creating raidframe for thoset wedges or disks and "raidframe -A yes"
> would be helpful to create stable device-name for zpool.
>
> I prefer to use dummy raidframe even if the host has only single device
> to make HDD/SSDs bootable when they attached to USB-SATA adapters.
Ahh, thats cute - I don't know why that didn't occur to me :) (Used a
similar setup for PCengines ALIX/APU USB images for older NetBSD
releases - for small enough disks on more recent NetBSD releases we
switched to plain ROOT{a,b,e} in fstab)
On Thu, 3 Dec 2020 at 07:18, Malcolm Herbert wrote:
>
> As far as I understand it, ZFS vdevs have their own ID, so they can be laid
> out correctly no matter the OS device each are discovered on ... wouldn't
> that make a raidframe wrapper redundant? it would also mean the zpool vdevs
> couldn't be used on other systems that understand ZFS because they're
> unlikely to understand raidframe ...
Thats what _should_ happen, but I believe ZFS relies on a persistent
device identifier such as /dev/rdsk/c0t5d0s2 or
/dev/disk/by-uuid/48e3b830-ff84-4434-ac74-b57b2ca59842 but NetBSD
doesn't directly provide one. However... Tobias pointed out devpubd
which apart from the rc.d order is close to perfect with this and
wedges.
On Thu, 3 Dec 2020 at 00:38, Brian Buhrow wrote:
>
> hello David. In the absence of other variables, I'd suggest using
> wedges. That gives you the ability to replace disks that go bad with
> differently sized disks in the future, while still retaining your zfs vdev
> sizes, something zfs likes a lot.
> Also, I'm pretty sure zfs recovers fine from wedge renumbering, at
> least it does under FreeBSD, much like raidframe does when it's
> autoconfiguring.
> I should say that while I have a lot of experience with zfs under
> FreeBSD, I've not used it much under NetBSD, mostly due to its instability,
> which is apparently now becoming much less of a problem -- something I'm
> very happy about.
I've had one box which managed to trip over a bunch of issues creating
and mounting zfs filesystems (now all resolved by others and pulled up
into -9), but I've been using it for a while now on a few other
machines, both mirrored and simple multi disk pools and have been very
happy overall.
Thanks for all the replies :)
David
Re: [current-users] Re: Best practice for setting up disks for ZFS on NetBSD
>> On Thu, 03 Dec 2020 18:19:05 +1100
>> m...@mjch.net ("Malcolm Herbert") said:
>
> As far as I understand it, ZFS vdevs have their own ID, so they can be laid
> out correctly no matter the OS device each are discovered on ... wouldn't
> that make a raidframe wrapper redundant? it would also mean the zpool vdevs
> couldn't be used on other systems that understand ZFS because they're
> unlikely to understand raidframe ...
My knowledge about ZFS is the same.
I never care about device numbers for zpool disks on FreeBSD.
But I found that NetBSD-9.1 STABLE(23 Oct) couldn't find zpool after
reconnecting external SSD as sd1, which was connected as sd0.
In that case, "zpool import" never find last zpool elements anymore.
I tried:
# sed 's,dev/sd0,dev/sd1,g' zpool.cache > zpool.cache2
# zpool import -c zpool.cache2 -a
These operation were needed every time when sdX number moved.
I've not tested on newer kernel.
Recent kernel detect pool IDs smoothly??
Anyway I needed to get the host stable state at that time,
I constructed redundant raidframe and found working fine for me.
This should be a dirty workaround for a short term...
--yuuji
>
> Regards,
> Malcolm
>
> On Thu, 3 Dec 2020, at 12:00, HIROSE yuuji wrote:
> > >> On Thu, 3 Dec 2020 00:30:17 +
> > >> a...@absd.org (David Brownlee) said:
> > >
> > > What would be the best practice for setting up disks to use under ZFS
> > > on NetBSD, with particular reference to handling renumbered devices?
> > >
> > > The two obvious options seem to be:
> > >
> > > - Wedges, setup as a single large gpt partition of type zfs (eg /dev/dk7)
> > > - Entire disk (eg: /dev/wd0 or /dev/sd4)
> > >
> >
> > Creating raidframe for thoset wedges or disks and "raidframe -A yes"
> > would be helpful to create stable device-name for zpool.
> >
> > I prefer to use dummy raidframe even if the host has only single device
> > to make HDD/SSDs bootable when they attached to USB-SATA adapters.
> >
> > --yuuji
> >
>
> --
> Malcolm Herbert
> m...@mjch.net
Re: Best practice for setting up disks for ZFS on NetBSD
On Thu, 3 Dec 2020 00:30:17 +, David Brownlee wrote: > - Wedges, setup as a single large gpt partition of type zfs (eg /dev/dk7) > - Entire disk (eg: /dev/wd0 or /dev/sd4) "Traditional" (solarish) zfs lore recommends giving zfs the entire disk, unpartitioned, since it can make more efficient use of it then. Zfs will generally be able to re-assemble a volume after renumbering components - I have seen it do that on OmniOS after swapping an Areca RAID controller in JBOD mode out for a plain SAS controller. Cheerio, Hauke -- Hauke Fath Linnéweg 7 64342 Seeheim-Jugenheim Germany
Re: Best practice for setting up disks for ZFS on NetBSD
On Thu, 3 Dec 2020 00:30:17 + David Brownlee wrote: > In the event of disk renumbering both are thrown out, needing a "zfs > export foo;zfs import foo" to recover. Is there some way to avoid > that? You can use named gpt wedges and symlink them to a stable path in /dev. I did this manually for my storage box setup but was later informed we have devpubd(8) which is capable of doing this.
Re: [current-users] Re: Best practice for setting up disks for ZFS on NetBSD
As far as I understand it, ZFS vdevs have their own ID, so they can be laid out correctly no matter the OS device each are discovered on ... wouldn't that make a raidframe wrapper redundant? it would also mean the zpool vdevs couldn't be used on other systems that understand ZFS because they're unlikely to understand raidframe ... Regards, Malcolm On Thu, 3 Dec 2020, at 12:00, HIROSE yuuji wrote: > >> On Thu, 3 Dec 2020 00:30:17 + > >> a...@absd.org (David Brownlee) said: > > > > What would be the best practice for setting up disks to use under ZFS > > on NetBSD, with particular reference to handling renumbered devices? > > > > The two obvious options seem to be: > > > > - Wedges, setup as a single large gpt partition of type zfs (eg /dev/dk7) > > - Entire disk (eg: /dev/wd0 or /dev/sd4) > > > > Creating raidframe for thoset wedges or disks and "raidframe -A yes" > would be helpful to create stable device-name for zpool. > > I prefer to use dummy raidframe even if the host has only single device > to make HDD/SSDs bootable when they attached to USB-SATA adapters. > > --yuuji > -- Malcolm Herbert m...@mjch.net
Re: Best practice for setting up disks for ZFS on NetBSD
>> On Thu, 3 Dec 2020 00:30:17 + >> a...@absd.org (David Brownlee) said: > > What would be the best practice for setting up disks to use under ZFS > on NetBSD, with particular reference to handling renumbered devices? > > The two obvious options seem to be: > > - Wedges, setup as a single large gpt partition of type zfs (eg /dev/dk7) > - Entire disk (eg: /dev/wd0 or /dev/sd4) > Creating raidframe for thoset wedges or disks and "raidframe -A yes" would be helpful to create stable device-name for zpool. I prefer to use dummy raidframe even if the host has only single device to make HDD/SSDs bootable when they attached to USB-SATA adapters. --yuuji
Re: Best practice for setting up disks for ZFS on NetBSD
hello David. In the absence of other variables, I'd suggest using wedges. That gives you the ability to replace disks that go bad with differently sized disks in the future, while still retaining your zfs vdev sizes, something zfs likes a lot. Also, I'm pretty sure zfs recovers fine from wedge renumbering, at least it does under FreeBSD, much like raidframe does when it's autoconfiguring. I should say that while I have a lot of experience with zfs under FreeBSD, I've not used it much under NetBSD, mostly due to its instability, which is apparently now becoming much less of a problem -- something I'm very happy about. Anyway, hope that helps. -Brian On Dec 3, 12:30am, David Brownlee wrote: } Subject: Best practice for setting up disks for ZFS on NetBSD } What would be the best practice for setting up disks to use under ZFS } on NetBSD, with particular reference to handling renumbered devices? } } The two obvious options seem to be: } } - Wedges, setup as a single large gpt partition of type zfs (eg /dev/dk7) } - Entire disk (eg: /dev/wd0 or /dev/sd4) } } (I'm going to skip disklabel partitions as they are size limited and } also encounter other issues with zpool import) } } Creating disks with single zfs wedges has the advantage of marking } each disk as "hey, zfs in use here", so it should be less likely to } accidentally overwrite it with something else, the wedge layer is } light enough to not add any measurable overhead, and providing the zfs } partition is aligned correctly zfs should be getting close enough to } "the real disk" } } Using the entire disk seems simpler, and the system (including tools } like iostat) are not suddenly cluttered by (in this case) a set of } unnecessary dk entries. } } In the event of disk renumbering both are thrown out, needing a "zfs } export foo;zfs import foo" to recover. Is there some way to avoid } that? } } David >-- End of excerpt from David Brownlee
Best practice for setting up disks for ZFS on NetBSD
What would be the best practice for setting up disks to use under ZFS on NetBSD, with particular reference to handling renumbered devices? The two obvious options seem to be: - Wedges, setup as a single large gpt partition of type zfs (eg /dev/dk7) - Entire disk (eg: /dev/wd0 or /dev/sd4) (I'm going to skip disklabel partitions as they are size limited and also encounter other issues with zpool import) Creating disks with single zfs wedges has the advantage of marking each disk as "hey, zfs in use here", so it should be less likely to accidentally overwrite it with something else, the wedge layer is light enough to not add any measurable overhead, and providing the zfs partition is aligned correctly zfs should be getting close enough to "the real disk" Using the entire disk seems simpler, and the system (including tools like iostat) are not suddenly cluttered by (in this case) a set of unnecessary dk entries. In the event of disk renumbering both are thrown out, needing a "zfs export foo;zfs import foo" to recover. Is there some way to avoid that? David
