Re: [zfs-discuss] HW raid vs ZFS
On Mon, 11 Jan 2010, Anil wrote: What is the recommended way to make use of a Hardware RAID controller/HBA along with ZFS? Does it make sense to do RAID5 on the HW and then RAIDZ on the software? OR just stick to ZFS RAIDZ and connect the drives to the controller, w/o any HW RAID (to benefit from the batteries). What will give the most performance benefit? ZFS will definitely benefit from battery backed RAM on the controller as long as the controller immediately acknowledges cache flushes (rather than waiting for battery-protected data to flush to the disks). There will be benefit as long as the size of the write data backlog does not exceed controller RAM size. The notion of performance is highly arbitrary since there are different types of performance. What sort of performance do you need to optimize for? I think that the best performance for most cases is to use mirroring. Use two controllers with battery-backed RAM and split the mirrors across the controllers so that a write to a mirror pair results in a write to each controller. Unfortunately, this is not nearly as space efficient as RAID5 or raidz. Many people will recommend against using RAID5 in hardware since then zfs is not as capable of repairing errors, and because most RAID5 controller cards use a particular format on the drives so that the drives become tied to the controller brand/model and it is not possible to move the pool to a different system without using an identical controller. If the controller fails and is no longer available for purchase, or the controller is found to have a design defect, then the pool may be toast. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] HW raid vs ZFS
ZFS will definitely benefit from battery backed RAM on the controller as long as the controller immediately acknowledges cache flushes (rather than waiting for battery-protected data to flush to the I am little confused with this. Do we not want the controller to ignore these cache flushes (since the cache is battery protected)? What did you mean by acknowledge? If it acknowledges and flushes the cache, then what is the benefit of the cache at all (if ZFS keeps telling to flush ever few seconds)? The notion of performance is highly arbitrary since there are different types of performance. What sort of performance do you need to optimize for? It won't be a big deal. Just general web/small databases. Just wondering how much of magnitude in performance difference there is. Many people will recommend against using RAID5 in hardware since then zfs is not as capable of repairing errors, and because most RAID5 controller cards use a particular format on the drives so that the drives become tied to the controller brand/model and it is not possible to move the pool to a different system without using an identical controller. If the controller fails and is no longer available for purchase, or the controller is found to have a design defect, then the pool may be toast. I use mostly DAS for my servers. This is a x4170 with 8 drive bays. So, what's the final recommendation? Should I just RAID 1 on the hardware and put ZFS on top of it? Got three options: Get the Storagetek Internal HBA (with batteries) and...: 7 disks 600gb usable === no hardware raid 2 rpool 3 raidz 1 hot spare 1 ssd 8 disks 600gb usable === 2 rpool 2 mirror + 2 mirror (the mirrors would be in hardware with zfs doing striping across the mirrors) 1 hot spare 1 ssd 8 disks 1200gb usable === no hardware raid 2 rpool 3 raidz + 3 raidz The hot spare and the SSD are optional and I can add them at a later point. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] HW raid vs ZFS
On 11-Jan-10, at 1:12 PM, Bob Friesenhahn wrote: On Mon, 11 Jan 2010, Anil wrote: What is the recommended way to make use of a Hardware RAID controller/HBA along with ZFS? ... Many people will recommend against using RAID5 in hardware since then zfs is not as capable of repairing errors, and because most RAID5 controller cards use a particular format on the drives so that the drives become tied to the controller brand/model and it is not possible to move the pool to a different system without using an identical controller. If the controller fails and is no longer available for purchase, or the controller is found to have a design defect, then the pool may be toast. +1 These drawbacks of proprietary RAID are frequently overlooked. Marty Scholes had a neat summary in a posting here, 21 October 2009: http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg30452.html Back when I did storage admin for a smaller company where availability was hyper-critical (but we couldn't afford EMC/ Veritas), we had a hardware RAID5 array. After a few years of service, we ran into some problems: * Need to restripe the array? Screwed. * Need to replace the array because current one is EOL? Screwed. * Array controller barfed for whatever reason? Screwed. * Need to flash the controller with latest firmware? Screwed. * Need to replace a component on the array, e.g. NIC, controller or power supply? Screwed. * Need to relocate the array? Screwed. If we could stomach downtime or short-lived storage solutions, none of this would have mattered. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] HW raid vs ZFS
On Mon, 11 Jan 2010, Anil wrote: ZFS will definitely benefit from battery backed RAM on the controller as long as the controller immediately acknowledges cache flushes (rather than waiting for battery-protected data to flush to the I am little confused with this. Do we not want the controller to ignore these cache flushes (since the cache is battery protected)? What did you mean by acknowledge? If it acknowledges and flushes the cache, then what is the benefit of the cache at all (if ZFS keeps telling to flush ever few seconds)? We want the controller to flush unwritten data to disk as quickly as it can regardless of whether it receives a cache flush request. If the data is safely stored in battery backed RAM, then we would like the controller to acknowledge the flush request immediately. The primary benefit of the battery-protected cache is to reduce latency for small writes. I use mostly DAS for my servers. This is a x4170 with 8 drive bays. So, what's the final recommendation? Should I just RAID 1 on the hardware and put ZFS on top of it? Unless you have a severe I/O bottleneck through your controller, you should do the mirroring in zfs rather than in the controller. The reason for this is that zfs mirrors are highly resilient, intelligent, and resilver time is reduced. Zfs will be able to detect and correct errors that the controller might not be aware of, or be unable to correct. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
