Re: booting off a ZFS pool consisting of multiple striped mirror vdevs
On Thu, Feb 18, 2010 at 4:36 PM, Dan Naumov wrote: > > A stripe of 3-way mirrors, whoa. Out of curiosity, what is the system > used for? I am not doubting that there exist some uses/workloads for a > system that uses 6 disks with 2 disks worth of usable space, but > that's a bit of an unusual configuration. What are your system/disc > specs and what kind of performance are you seeing from the pool? > It's for a reasonably busy webserver hosting a few hundred domains, which tends to be somewhat seek-intensive. For this pool we had two main criteria: speed and double-disk redundancy. A stripe of three two-way mirrors would only give us single-disk redundancy in the worst case (i.e. losing both disks in one of the mirrors), so we went with two three-way mirrors instead. Even though we're only getting the capacity of two disks worth of space, we'll still have 6x the capacity of the array it's replacing. A simple-minded dd test gives me ~180MB/s writing a single long file, and 400-500MB/s reading. Matt ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: booting off a ZFS pool consisting of multiple striped mirror vdevs
On Fri, Feb 19, 2010 at 1:03 AM, Matt Reimer wrote: > On Thu, Feb 18, 2010 at 10:57 AM, Matt Reimer wrote: >> >> On Tue, Feb 16, 2010 at 12:38 AM, Dan Naumov wrote: >>> >>> > I don't know, but I plan to test that scenario in a few days. >>> > >>> > Matt >>> >>> Please share the results when you're done, I am really curious :) >> >> Booting from a stripe of two raidz vdevs works: >> FreeBSD/i386 boot >> Default: doom:/boot/zfsloader >> boot: status >> pool: doom >> config: >> NAME STATE >> doom ONLINE >> raidz1 ONLINE >> label/doom-0 ONLINE >> label/doom-1 ONLINE >> label/doom-2 ONLINE >> raidz1 ONLINE >> label/doom-3 ONLINE >> label/doom-4 ONLINE >> label/doom-5 ONLINE >> I'd guess a stripe of mirrors would work fine too. If I get a chance I'll >> test that combo. > > A stripe of three-way mirrors works: > FreeBSD/i386 boot > Default: mithril:/boot/zfsloader > boot: status > pool: mithril > config: > NAME STATE > mithril ONLINE > mirror ONLINE > label/mithril-0 ONLINE > label/mithril-1 ONLINE > label/mithril-2 ONLINE > mirror ONLINE > label/mithril-3 ONLINE > label/mithril-4 ONLINE > label/mithril-5 ONLINE > Matt A stripe of 3-way mirrors, whoa. Out of curiosity, what is the system used for? I am not doubting that there exist some uses/workloads for a system that uses 6 disks with 2 disks worth of usable space, but that's a bit of an unusual configuration. What are your system/disc specs and what kind of performance are you seeing from the pool? - Sincerely, Dan Naumov ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: booting off a ZFS pool consisting of multiple striped mirror vdevs
On Thu, Feb 18, 2010 at 10:57 AM, Matt Reimer wrote: > On Tue, Feb 16, 2010 at 12:38 AM, Dan Naumov wrote: > >> > I don't know, but I plan to test that scenario in a few days. >> > >> > Matt >> >> Please share the results when you're done, I am really curious :) >> > > Booting from a stripe of two raidz vdevs works: > > FreeBSD/i386 boot > Default: doom:/boot/zfsloader > boot: status > pool: doom > config: > > NAME STATE > doom ONLINE > raidz1 ONLINE > label/doom-0 ONLINE > label/doom-1 ONLINE > label/doom-2 ONLINE > raidz1 ONLINE > label/doom-3 ONLINE > label/doom-4 ONLINE > label/doom-5 ONLINE > > I'd guess a stripe of mirrors would work fine too. If I get a chance I'll > test that combo. > A stripe of three-way mirrors works: FreeBSD/i386 boot Default: mithril:/boot/zfsloader boot: status pool: mithril config: NAME STATE mithril ONLINE mirror ONLINE label/mithril-0 ONLINE label/mithril-1 ONLINE label/mithril-2 ONLINE mirror ONLINE label/mithril-3 ONLINE label/mithril-4 ONLINE label/mithril-5 ONLINE Matt ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: booting off a ZFS pool consisting of multiple striped mirror vdevs
On Tue, Feb 16, 2010 at 12:38 AM, Dan Naumov wrote: > > I don't know, but I plan to test that scenario in a few days. > > > > Matt > > Please share the results when you're done, I am really curious :) > Booting from a stripe of two raidz vdevs works: FreeBSD/i386 boot Default: doom:/boot/zfsloader boot: status pool: doom config: NAME STATE doom ONLINE raidz1 ONLINE label/doom-0 ONLINE label/doom-1 ONLINE label/doom-2 ONLINE raidz1 ONLINE label/doom-3 ONLINE label/doom-4 ONLINE label/doom-5 ONLINE I'd guess a stripe of mirrors would work fine too. If I get a chance I'll test that combo. > If booting of a stripe of 3 mirrors should work assuming no BIOS bugs, > can you explain why is booting off simple stripes (of any number of > disks) currently unsupported? I haven't tested that myself, but > everywhere I look seems to indicate that booting off a simple stripe > doesn't work or is that "everywhere" also out of date after your > changes? :) It's probably unsupported in Solaris/OpenSolaris because of their bootloader. Our bootloader is completely different from theirs and so is not subject to those restrictions in the ZFS docs. The bottom line is that I think FreeBSD can boot from pretty much any configuration, except possibly from systems with huge numbers of disks. Matt ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
RE: booting off a ZFS pool consisting of multiple striped mirror vdevs
On Tue, 2010-02-16 at 10:38 +0200, Dan Naumov wrote: > > I don't know, but I plan to test that scenario in a few days. > > > > Matt > > Please share the results when you're done, I am really curious :) > > > > It *should* work... I made changes a while back that allow for multiple > > vdevs to attach to the root. In this case you should have 3 mirror > > vdevs attached to the root, so as long as the BIOS can enumerate all of > > the drives, we should find all of the vdevs and build the tree > > correctly. It should be simple enough to test in qemu, except that the > > BIOS in qemu is a little broken and might not id all of the drives. > > > > robert. > > If booting of a stripe of 3 mirrors should work assuming no BIOS bugs, > can you explain why is booting off simple stripes (of any number of > disks) currently unsupported? I haven't tested that myself, but > everywhere I look seems to indicate that booting off a simple stripe > doesn't work or is that "everywhere" also out of date after your > changes? :) I would need to setup a qemu test for it... But I can't think of a reason that it shouldn't work. I currently boot from two physical drives attached to the root. In the boot code we enumerate all the physical vdevs, so that we can correctly identify the correct disk and offset from the block pointer. The mirror driver is really pretty simple. robert. > > - Sincerely, > Dan Naumov -- Robert Noland FreeBSD ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
RE: booting off a ZFS pool consisting of multiple striped mirror vdevs
> I don't know, but I plan to test that scenario in a few days. > > Matt Please share the results when you're done, I am really curious :) > It *should* work... I made changes a while back that allow for multiple > vdevs to attach to the root. In this case you should have 3 mirror > vdevs attached to the root, so as long as the BIOS can enumerate all of > the drives, we should find all of the vdevs and build the tree > correctly. It should be simple enough to test in qemu, except that the > BIOS in qemu is a little broken and might not id all of the drives. > > robert. If booting of a stripe of 3 mirrors should work assuming no BIOS bugs, can you explain why is booting off simple stripes (of any number of disks) currently unsupported? I haven't tested that myself, but everywhere I look seems to indicate that booting off a simple stripe doesn't work or is that "everywhere" also out of date after your changes? :) - Sincerely, Dan Naumov ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: booting off a ZFS pool consisting of multiple striped mirror vdevs
On Mon, 2010-02-15 at 10:07 -0800, Matt Reimer wrote: > On Sat, Feb 13, 2010 at 12:04 PM, Dan Naumov wrote: > > > Hello > > > > I have succesfully tested and used a "full ZFS install" of FreeBSD 8.0 > > on both single disk and mirror disk configurations using both MBR and > > GPT partitioning. AFAIK, with the more recent -CURRENT and -STABLE it > > is also possible to boot off a root filesystem located on raidz/raidz2 > > pools. But what about booting off pools consisting of multiple striped > > mirror or raidz vdevs? Like this: > > > > Assume each disk looks like a half of a traditional ZFS mirror root > > configuration using GPT > > > > 1: freebsd-boot > > 2: freebsd-swap > > 3: freebsd-zfs > > > > |disk1+disk2| + |disk3+disk4| + |disk5+disk6| > > > > My logic tells me that while booting off any of the 6 disks, boot0 and > > boot1 stage should obviously work fine, but what about the boot2 > > stage? Can it properly handle booting off a root filesystem thats > > striped across 3 mirror vdevs or is booting off a single mirror vdev > > the best that one can do right now? > > > > > I don't know, but I plan to test that scenario in a few days. It *should* work... I made changes a while back that allow for multiple vdevs to attach to the root. In this case you should have 3 mirror vdevs attached to the root, so as long as the BIOS can enumerate all of the drives, we should find all of the vdevs and build the tree correctly. It should be simple enough to test in qemu, except that the BIOS in qemu is a little broken and might not id all of the drives. robert. > Matt > ___ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" -- Robert Noland FreeBSD ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: booting off a ZFS pool consisting of multiple striped mirror vdevs
On Sat, Feb 13, 2010 at 12:04 PM, Dan Naumov wrote: > Hello > > I have succesfully tested and used a "full ZFS install" of FreeBSD 8.0 > on both single disk and mirror disk configurations using both MBR and > GPT partitioning. AFAIK, with the more recent -CURRENT and -STABLE it > is also possible to boot off a root filesystem located on raidz/raidz2 > pools. But what about booting off pools consisting of multiple striped > mirror or raidz vdevs? Like this: > > Assume each disk looks like a half of a traditional ZFS mirror root > configuration using GPT > > 1: freebsd-boot > 2: freebsd-swap > 3: freebsd-zfs > > |disk1+disk2| + |disk3+disk4| + |disk5+disk6| > > My logic tells me that while booting off any of the 6 disks, boot0 and > boot1 stage should obviously work fine, but what about the boot2 > stage? Can it properly handle booting off a root filesystem thats > striped across 3 mirror vdevs or is booting off a single mirror vdev > the best that one can do right now? > > I don't know, but I plan to test that scenario in a few days. Matt ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
