Re: [zfs-discuss] Painfully slow RAIDZ2 as fibre channel COMSTAR export
Hi Dave So which hard drives are connected to which controllers? And what device drivers are those controllers using? The output from 'format', 'cfgadm' and 'prtconf -D' may help us to understand. Strange that you say that there are two hard drives per controllers, but three drives are showing high %b. And strange that you have c7,c8,c9,c10,c11 which looks like FIVE controllers! Regards Nigel Smith -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Painfully slow RAIDZ2 as fibre channel COMSTAR export
So which hard drives are connected to which controllers? And what device drivers are those controllers using? 0. c7t0d0 DEFAULT cyl 7764 alt 2 hd 255 sec 63 /p...@0,0/pci8086,3...@3/pci1000,3...@0/s...@0,0 1. c7t1d0 ATA-Hitachi HDS72202-A20N-1.82TB /p...@0,0/pci8086,3...@3/pci1000,3...@0/s...@1,0 2. c8t0d0 ATA-Hitachi HDS72202-A20N-1.82TB /p...@0,0/pci8086,3...@7/pci1000,3...@0/s...@0,0 3. c8t1d0 ATA-Hitachi HDS72202-A20N-1.82TB /p...@0,0/pci8086,3...@7/pci1000,3...@0/s...@1,0 4. c9t0d0 ATA-Hitachi HDS72202-A20N-1.82TB /p...@0,0/pci8086,3...@9/pci1000,3...@0/s...@0,0 5. c9t1d0 ATA-Hitachi HDS72202-A20N-1.82TB /p...@0,0/pci8086,3...@9/pci1000,3...@0/s...@1,0 6. c10d0 DEFAULT cyl 7764 alt 2 hd 255 sec 63 /p...@0,0/pci-...@1f,2/i...@0/c...@0,0 7. c10d1 Hitachi- JK1131YAGP8N3-0001-1.82TB /p...@0,0/pci-...@1f,2/i...@0/c...@1,0 8. c11d0 Hitachi- JK1131YAGZE4Z-0001-1.82TB /p...@0,0/pci-...@1f,2/i...@1/c...@0,0 9. c11d1 Hitachi- JK1131YAGGMT9-0001-1.82TB /p...@0,0/pci-...@1f,2/i...@1/c...@1,0 Strange that you say that there are two hard drives per controllers, but three drives are showing high %b. And strange that you have c7,c8,c9,c10,c11 which looks like FIVE controllers! c7, c8 and c9 are LSI controllers using the MPT driver. The motherboard has 6 SATA ports which are presented as two controllers (presumably c10 and c11) one for ports 0-3 and one for ports 4 and 5; both currently use the PCI-IDE drivers. And as you say, it's odd that there are three drives on c10 and c11, since they should have only two of the raidz2 drives; I need to go double-check my cabling. The way it's *supposed* to be configured is: c7: two RAIDZ2 drives and one of the boot mirror drives c8: two RAIDZ2 drives c9: two RAIDZ2 drives c10: one RAIDZ2 drive and one of the boot mirror drives c11: one RAIDZ2 drive (The theory here is that since this server is going to spend its life being shipped places in the back of a truck I want to make sure that no single controller failure can either render it unbootable or destroy the RAIDZ2.) That said, I think that this is probably *a* tuning problem but not *the* tuning problem, since I was getting acceptable performance over CIFS and miserable performance over FC. Richard Elling suggested I try the latest dev release to see if I'm encountering a bug that forces synchronous writes, so I'm off to straighten out my controller distribution, check to see if I have write caching turned off on the motherboard ports, install the b132 build, and possibly grab some dinner while I'm about it. I'll report back to the list with any progress or lack thereof. -- Dave Pooser, ACSA Manager of Information Services Alford Media http://www.alfordmedia.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Painfully slow RAIDZ2 as fibre channel COMSTAR export
On Sun, 14 Feb 2010, Dave Pooser wrote: c7, c8 and c9 are LSI controllers using the MPT driver. The motherboard has 6 SATA ports which are presented as two controllers (presumably c10 and c11) one for ports 0-3 and one for ports 4 and 5; both currently use the PCI-IDE drivers. One should expect that the IDE interface will be less performant than the SATA interface. For example, it seems likely that IDE does not support NCQ, so only one write could be scheduled at a time while SATA can burst multiple writes into the drive cache at a time. This would explain if the IDE drives seem to be 100% busy while the SATA drives are almost idle. This would cause issues for synchronous writes. Absent careful engineering, raidz2 usually only has one bottleneck at a time. See http://en.wikipedia.org/wiki/Integrated_Drive_Electronics#IDE_and_ATA-1;. That said, I think that this is probably *a* tuning problem but not *the* tuning problem, since I was getting acceptable performance over CIFS and miserable performance over FC. Richard Elling suggested I try the latest dev release to see if I'm encountering a bug that forces synchronous writes, so A difference in the way synchronous writes are handled could certainly make a huge difference. It is useful to do asynchronous and synchronous write benchmarks on the local system before getting the higher level protocols involved. As far as the warning about different sized devices goes, I am wondering if there is a limit to the maximum size of an IDE-based device and so some devices are claimed larger than others. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Painfully slow RAIDZ2 as fibre channel COMSTAR export
c7, c8 and c9 are LSI controllers using the MPT driver. The motherboard has 6 SATA ports which are presented as two controllers (presumably c10 and c11) one for ports 0-3 and one for ports 4 and 5; both currently use the PCI-IDE drivers. on my motherboard, i can make the onboard sata ports show up as IDE or SATA, you may look into that. It would probably be something like AHCI mode. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Painfully slow RAIDZ2 as fibre channel COMSTAR export
On Sun, Feb 14, 2010 at 8:49 PM, Thomas Burgess wonsl...@gmail.com wrote: c7, c8 and c9 are LSI controllers using the MPT driver. The motherboard has 6 SATA ports which are presented as two controllers (presumably c10 and c11) one for ports 0-3 and one for ports 4 and 5; both currently use the PCI-IDE drivers. on my motherboard, i can make the onboard sata ports show up as IDE or SATA, you may look into that. It would probably be something like AHCI mode. Those are actual IDE ports. That's why they show up as the same controller, different disk. He might have a tough time turning IDE ports into SATA in the BIOS ;) --Tim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Painfully slow RAIDZ2 as fibre channel COMSTAR export
on my motherboard, i can make the onboard sata ports show up as IDE or SATA, you may look into that. It would probably be something like AHCI mode. Yeah, I changed the motherboard setting from enhanced to AHCI and now those ports show up as SATA. -- Dave Pooser, ACSA Manager of Information Services Alford Media http://www.alfordmedia.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Painfully slow RAIDZ2 as fibre channel COMSTAR export
oh, so i WAS right? awesome On Sun, Feb 14, 2010 at 10:45 PM, Dave Pooser dave@alfordmedia.comwrote: on my motherboard, i can make the onboard sata ports show up as IDE or SATA, you may look into that. It would probably be something like AHCI mode. Yeah, I changed the motherboard setting from enhanced to AHCI and now those ports show up as SATA. -- Dave Pooser, ACSA Manager of Information Services Alford Media http://www.alfordmedia.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Painfully slow RAIDZ2 as fibre channel COMSTAR export
I'm off to straighten out my controller distribution, check to see if I have write caching turned off on the motherboard ports, install the b132 build, and possibly grab some dinner while I'm about it. I'll report back to the list with any progress or lack thereof. OK, the issue seems to be resolved now-- I'm seeing write speeds in excess of 160MB/s. What I did to fix things: 1) Redistributed drives across controllers to match my actual configuration-- thanks to Nigel for pointing that one out 2) Set my motherboard controller to AHCI mode-- thanks to Richard and Thomas for suggesting that. Once I made that change I no longer saw the raidz contains devices of different sizes error, so it looks like Bob was right about the source of that error 3) Upgraded to OpenSolaris 2010.03 preview b132 which appears to correct a problem in 2009.06 where iSCSI (and apparently FC) forced all writes to be synchronous -- thanks to Richard for that pointer. Five hours from tearing my hair out to toasting a success-- this list is a great resource! -- Dave Pooser, ACSA Manager of Information Services Alford Media http://www.alfordmedia.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
