On Wed, 2006-06-28 at 22:13 +0100, Peter Tribble wrote: > On Wed, 2006-06-28 at 17:32, Erik Trimble wrote: > > Given a reasonable number of hot-spares, I simply can't see the (very) > > marginal increase in safety give by using HW RAID5 as out balancing the > > considerable speed hit using RAID5 takes. > > That's not quite right. There's no significant difference in > performance, > and the question is whether you're prepared to give up a small amount of > space for orders of magnitude increase in safety.
As indicated by previous posts, even with HW assist, RAID5/6 on N disks will be noticably slower than a stripe of N disks. Theoretical read performance for RAID5 is at best (in a limited number of cases) equal to striping, and for the general read case, is (N-1)/N slower. Even assuming no performance hit at all for the parity calculation on writes, writes to a RAID5 are at best equal to a stripe (assuming a full stripe has to be written), and usually at least (N-1)/N slower, as the parity must be written in addition to the normal data (i.e. N/(N-1) data must be written, rather than N). > Each extra disk you can survive the failure of leads to a massive > increase in safety. By something like (just considering isolated > random disk failures) the ratio of the MTBF of a disk to the time > it takes to get a spare back in and repair the LUN. That's something > like 100,000 - which isn't a marginal increase in safety! > > Yes, I know it's not that simple. The point to take from this is > simply that being able to survive 2 failures instead of 1 doesn't > double your safety, it increases it by a very large number. And > by a very large number again for the next failure you can survive. > > In the stripe case, a single disk loss (pretty common) loses all your > redundancy straight off. A second disk failure (not particularly rare) > and all your data's toast. Hot spares don't really help in this case. > > At this point having HW raid-5 underneath means you're still humming > along safely. > Agreed. However, part of the issue that comes up is that you take into account the possibility of a drive failing before the hotspare can be re-silvered when a drive. In general, this is why I don't see stripes being more than 6-8 drives wide. It takes somewhere between 30 minutes and 2 hours to resilver a drive in a mirrored stripe of that size (depending on capacity and activity). So, the tradeoff has to be made. And, note that you'll reduce your performance with the RAID5 resyncing for considerably longer than it takes to re-silver the stripe. Put in another way (assuming a hotspare is automatically put in place after a drive failure): With mirrored stripes, I'm vulnerable to complete data loss while the 2nd stripe resyncs. Say 2 hours or so, worst case. By percentages, a 2nd drive loss in a mirrored stripe has a 50% chance of causing data loss. With mirrored RAID5, I'm invulnerable to a second drive loss. However, my resync time is vastly greater than for a stripe, increasing my window for more drive failures by at least double, if not more. It's all a tradeoff. However, in general, I haven't see 2nd drives fail within a stripe resync time, UNLESS I saw many drives fail (that is, 30-40% of a group fail close together), in which case, this overwhelms any RAID ability to compensate. It's certainly possible. (which is why striped mirrors are preferred to mirrored stripes). In the end, it comes down to the local needs. Speaking in generalizations, my opinion is that a mirror of RAID5 doesn't significantly increase your safety enough to warrant a 15-20% reduction in space, and at least that in performance, vs a mirror of stripes. And, of course, backups help. :-) -- Erik Trimble Java System Support Mailstop: usca14-102 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800) _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
