Re: [zfs-discuss] [illumos-Developer] ZFS spare disk usage issue
- Original Message - Hi all I just did a small test on RAIDz2 to check whether my suspicion was right about ZFS not treating spares as replicas/copies of drives, and I think I've found it true. The short story: If two spares replaces two drives in raidz2, losing a third drive, even with the spares active, makes the pool unavailable. See full report on Update 2010-03-04 14:15 CET I just tested on another system. This one, not in production yet, has a mirrored rpool and a 14-drive RAID10 pool named tos-data. I started a copy from a Windows machine into this CIFS share just to generate some traffic. Then I did a zfs detach of one side of each of the mirrors for tos-data and created a new 5-drive raidz2 pool name jalla with two dedicated spares. I started a dd to fill it up and plugged one drive, waited for it to resilver and plugged another, again waited for the resilver to finish and plugged the third. The server now hangs on all pools. I've also tested removing drives from mirrors and waiting for them to resilver to spares. This seems to work as expected, although I doubt booting from one will work without grub being installed. ODT: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.odt PDF: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.pdf These are mow updated as well Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [illumos-Developer] ZFS spare disk usage issue
I understand that some of it may be a simple bug, but should it hang _all_ the pools? That's what happens when the third drive is removed... roy - Original Message - This looks like a pretty simple bug. The issue is that the state of the SPARE vdev is being reported as REMOVED instead of DEGRADED. If it were the latter (as it should be), then everything would work just fine. Please file a bug at bugs.illumos.org . On a side note, this continues to expose the overly simplistic vdev state model used by ZFS (one which I can take a bulk of the responsibility for). Back before the days of ditto blocks and SPA3.0, it was sufficient to model state as a fairly binary proposition. But this now has ramifications that don't necessarily make sense. For example, one may be able open a pool even if a toplevel vdev is faulted. And even when a spare has finished resilvering, it's left in the DEGRADED state, which has implications for allocation policies (though I remember discussions around changing this). But the pool state is derived directly from the toplevel vdev state, so if you switch spares to be ONLINE, then 'zpool status' would think your pool is perfectly healthy. In this case it's true from a data protection standpoint, but not necessarily from a all is well in the world standpoint, as you are down one spare, and that spare may not have the same RAS properties as other devices in your RAID-Z stripe (it may put 3 disks on the same controller in one stripe, for example). - Eric On Fri, Mar 4, 2011 at 7:06 AM, Roy Sigurd Karlsbakk r...@karlsbakk.net wrote: Hi all I just did a small test on RAIDz2 to check whether my suspicion was right about ZFS not treating spares as replicas/copies of drives, and I think I've found it true. The short story: If two spares replaces two drives in raidz2, losing a third drive, even with the spares active, makes the pool unavailable. See full report on ODT: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.odt PDF: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.pdf Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. ___ Developer mailing list develo...@lists.illumos.org http://lists.illumos.org/m/listinfo/developer -- Eric Schrock Delphix 275 Middlefield Road, Suite 50 Menlo Park, CA 94025 http://www.delphix.com -- Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [illumos-Developer] ZFS spare disk usage issue
This looks like a pretty simple bug. The issue is that the state of the SPARE vdev is being reported as REMOVED instead of DEGRADED. If it were the latter (as it should be), then everything would work just fine. Please file a bug at bugs.illumos.org. On a side note, this continues to expose the overly simplistic vdev state model used by ZFS (one which I can take a bulk of the responsibility for). Back before the days of ditto blocks and SPA3.0, it was sufficient to model state as a fairly binary proposition. But this now has ramifications that don't necessarily make sense. For example, one may be able open a pool even if a toplevel vdev is faulted. And even when a spare has finished resilvering, it's left in the DEGRADED state, which has implications for allocation policies (though I remember discussions around changing this). But the pool state is derived directly from the toplevel vdev state, so if you switch spares to be ONLINE, then 'zpool status' would think your pool is perfectly healthy. In this case it's true from a data protection standpoint, but not necessarily from a all is well in the world standpoint, as you are down one spare, and that spare may not have the same RAS properties as other devices in your RAID-Z stripe (it may put 3 disks on the same controller in one stripe, for example). - Eric On Fri, Mar 4, 2011 at 7:06 AM, Roy Sigurd Karlsbakk r...@karlsbakk.netwrote: Hi all I just did a small test on RAIDz2 to check whether my suspicion was right about ZFS not treating spares as replicas/copies of drives, and I think I've found it true. The short story: If two spares replaces two drives in raidz2, losing a third drive, even with the spares active, makes the pool unavailable. See full report on ODT: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.odt PDF: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.pdf Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. ___ Developer mailing list develo...@lists.illumos.org http://lists.illumos.org/m/listinfo/developer -- Eric Schrock Delphix 275 Middlefield Road, Suite 50 Menlo Park, CA 94025 http://www.delphix.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [illumos-Developer] ZFS spare disk usage issue
So should I post a bug, or is there one there already? Btw, I can't reach http://bugs.illumos.org/ - it times out roy - Original Message - We've talked about this, and I will be putting together a fix for this incorrect state handling. :-) - Garrett On Fri, 2011-03-04 at 11:50 -0500, Eric Schrock wrote: This looks like a pretty simple bug. The issue is that the state of the SPARE vdev is being reported as REMOVED instead of DEGRADED. If it were the latter (as it should be), then everything would work just fine. Please file a bug at bugs.illumos.org. On a side note, this continues to expose the overly simplistic vdev state model used by ZFS (one which I can take a bulk of the responsibility for). Back before the days of ditto blocks and SPA3.0, it was sufficient to model state as a fairly binary proposition. But this now has ramifications that don't necessarily make sense. For example, one may be able open a pool even if a toplevel vdev is faulted. And even when a spare has finished resilvering, it's left in the DEGRADED state, which has implications for allocation policies (though I remember discussions around changing this). But the pool state is derived directly from the toplevel vdev state, so if you switch spares to be ONLINE, then 'zpool status' would think your pool is perfectly healthy. In this case it's true from a data protection standpoint, but not necessarily from a all is well in the world standpoint, as you are down one spare, and that spare may not have the same RAS properties as other devices in your RAID-Z stripe (it may put 3 disks on the same controller in one stripe, for example). - Eric On Fri, Mar 4, 2011 at 7:06 AM, Roy Sigurd Karlsbakk r...@karlsbakk.net wrote: Hi all I just did a small test on RAIDz2 to check whether my suspicion was right about ZFS not treating spares as replicas/copies of drives, and I think I've found it true. The short story: If two spares replaces two drives in raidz2, losing a third drive, even with the spares active, makes the pool unavailable. See full report on ODT: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.odt PDF: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.pdf Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. ___ Developer mailing list develo...@lists.illumos.org http://lists.illumos.org/m/listinfo/developer -- Eric Schrock Delphix 275 Middlefield Road, Suite 50 Menlo Park, CA 94025 http://www.delphix.com ___ Developer mailing list develo...@lists.illumos.org http://lists.illumos.org/m/listinfo/developer -- Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [illumos-Developer] ZFS spare disk usage issue
On Fri, 2011-03-04 at 18:03 +0100, Roy Sigurd Karlsbakk wrote: So should I post a bug, or is there one there already? Btw, I can't reach http://bugs.illumos.org/ - it times out Try again in a few minutes... the server just got rebooted. - Garrett roy - Original Message - We've talked about this, and I will be putting together a fix for this incorrect state handling. :-) - Garrett On Fri, 2011-03-04 at 11:50 -0500, Eric Schrock wrote: This looks like a pretty simple bug. The issue is that the state of the SPARE vdev is being reported as REMOVED instead of DEGRADED. If it were the latter (as it should be), then everything would work just fine. Please file a bug at bugs.illumos.org. On a side note, this continues to expose the overly simplistic vdev state model used by ZFS (one which I can take a bulk of the responsibility for). Back before the days of ditto blocks and SPA3.0, it was sufficient to model state as a fairly binary proposition. But this now has ramifications that don't necessarily make sense. For example, one may be able open a pool even if a toplevel vdev is faulted. And even when a spare has finished resilvering, it's left in the DEGRADED state, which has implications for allocation policies (though I remember discussions around changing this). But the pool state is derived directly from the toplevel vdev state, so if you switch spares to be ONLINE, then 'zpool status' would think your pool is perfectly healthy. In this case it's true from a data protection standpoint, but not necessarily from a all is well in the world standpoint, as you are down one spare, and that spare may not have the same RAS properties as other devices in your RAID-Z stripe (it may put 3 disks on the same controller in one stripe, for example). - Eric On Fri, Mar 4, 2011 at 7:06 AM, Roy Sigurd Karlsbakk r...@karlsbakk.net wrote: Hi all I just did a small test on RAIDz2 to check whether my suspicion was right about ZFS not treating spares as replicas/copies of drives, and I think I've found it true. The short story: If two spares replaces two drives in raidz2, losing a third drive, even with the spares active, makes the pool unavailable. See full report on ODT: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.odt PDF: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.pdf Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. ___ Developer mailing list develo...@lists.illumos.org http://lists.illumos.org/m/listinfo/developer -- Eric Schrock Delphix 275 Middlefield Road, Suite 50 Menlo Park, CA 94025 http://www.delphix.com ___ Developer mailing list develo...@lists.illumos.org http://lists.illumos.org/m/listinfo/developer ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
