On Dec 30, 2012, at 9:02 AM, Eugen Leitl <eu...@leitl.org> wrote: > > Happy $holidays, > > I have a pool of 8x ST31000340AS on an LSI 8-port adapter as > a raidz3 (no compression nor dedup) with reasonable bonnie++ > 1.03 values, e.g. 145 MByte/s Seq-Write @ 48% CPU and 291 MByte/s > Seq-Read @ 53% CPU. It scrubs with 230+ MByte/s with reasonable > system load. No hybrid pools yet. This is latest beta napp-it > on OpenIndiana 151a5 server, living on a dedicated 64 GByte SSD. > > The system is a MSI E350DM-E33 with 8 GByte PC1333 DDR3 > memory, no ECC. All the systems have Intel NICs with mtu 9000 > enabled, including all switches in the path.
Does it work faster with the default MTU? Also check for retrans and errors, using the usual network performance debugging checks. > > My problem is pretty poor network throughput. An NFS > mount on 12.04 64 bit Ubuntu (mtu 9000) or CIFS are > read at about 23 MBytes/s. Windows 7 64 bit (also jumbo > frames) reads at about 65 MBytes/s. The highest transfer > speed on Windows just touches 90 MByte/s, before falling > back to the usual 60-70 MBytes/s. > > I kinda can live with above values, but I have a feeling > the setup should be able to saturate GBit Ethernet with > large file transfers, especially on Linux (20 MByte/s > is nothing to write home about). > > Does anyone have any suggestions on how to debug/optimize > throughput? > > Thanks, and happy 2013. > > P.S. Not sure whether this is pathological, but the system > does produce occasional soft errors like e.g. dmesg More likely these are due to SMART commands not being properly handled for SATA devices. They are harmless. -- richard > > Dec 30 17:45:00 oizfs scsi: [ID 107833 kern.notice] Requested Block: 0 > Error Block: 0 > Dec 30 17:45:00 oizfs scsi: [ID 107833 kern.notice] Vendor: ATA > Serial Number: > Dec 30 17:45:00 oizfs scsi: [ID 107833 kern.notice] Sense Key: Soft_Error > Dec 30 17:45:00 oizfs scsi: [ID 107833 kern.notice] ASC: 0x0 (<vendor > unique code 0x0>), ASCQ: 0x1d, FRU: 0x0 > Dec 30 17:45:01 oizfs scsi: [ID 107833 kern.warning] WARNING: > /scsi_vhci/disk@g5000c50009c72c48 (sd9): > Dec 30 17:45:01 oizfs Error for Command: <undecoded cmd 0xa1> Error > Level: Recovered > Dec 30 17:45:01 oizfs scsi: [ID 107833 kern.notice] Requested Block: 0 > Error Block: 0 > Dec 30 17:45:01 oizfs scsi: [ID 107833 kern.notice] Vendor: ATA > Serial Number: > Dec 30 17:45:01 oizfs scsi: [ID 107833 kern.notice] Sense Key: Soft_Error > Dec 30 17:45:01 oizfs scsi: [ID 107833 kern.notice] ASC: 0x0 (<vendor > unique code 0x0>), ASCQ: 0x1d, FRU: 0x0 > Dec 30 17:45:01 oizfs pcplusmp: [ID 805372 kern.info] pcplusmp: ide (ata) > instance 0 irq 0xe vector 0x45 ioapic 0x3 intin 0xe is bound to cpu 0 > Dec 30 17:45:01 oizfs pcplusmp: [ID 805372 kern.info] pcplusmp: ide (ata) > instance 0 irq 0xe vector 0x45 ioapic 0x3 intin 0xe is bound to cpu 1 > Dec 30 17:45:01 oizfs pcplusmp: [ID 805372 kern.info] pcplusmp: ide (ata) > instance 0 irq 0xe vector 0x45 ioapic 0x3 intin 0xe is bound to cpu 0 > Dec 30 17:45:01 oizfs scsi: [ID 107833 kern.warning] WARNING: > /scsi_vhci/disk@g5000c50009c73968 (sd4): > Dec 30 17:45:01 oizfs Error for Command: <undecoded cmd 0xa1> Error > Level: Recovered > Dec 30 17:45:01 oizfs scsi: [ID 107833 kern.notice] Requested Block: 0 > Error Block: 0 > Dec 30 17:45:01 oizfs scsi: [ID 107833 kern.notice] Vendor: ATA > Serial Number: > Dec 30 17:45:01 oizfs scsi: [ID 107833 kern.notice] Sense Key: Soft_Error > Dec 30 17:45:01 oizfs scsi: [ID 107833 kern.notice] ASC: 0x0 (<vendor > unique code 0x0>), ASCQ: 0x1d, FRU: 0x0 > Dec 30 17:45:03 oizfs scsi: [ID 107833 kern.warning] WARNING: > /scsi_vhci/disk@g5000c500098be9dd (sd10): > Dec 30 17:45:03 oizfs Error for Command: <undecoded cmd 0xa1> Error > Level: Recovered > Dec 30 17:45:03 oizfs scsi: [ID 107833 kern.notice] Requested Block: 0 > Error Block: 0 > Dec 30 17:45:03 oizfs scsi: [ID 107833 kern.notice] Vendor: ATA > Serial Number: > Dec 30 17:45:03 oizfs scsi: [ID 107833 kern.notice] Sense Key: Soft_Error > Dec 30 17:45:03 oizfs scsi: [ID 107833 kern.notice] ASC: 0x0 (<vendor > unique code 0x0>), ASCQ: 0x1d, FRU: 0x0 > Dec 30 17:45:04 oizfs scsi: [ID 107833 kern.warning] WARNING: > /pci@0,0/pci1462,7720@11/disk@3,0 (sd8): > Dec 30 17:45:04 oizfs Error for Command: <undecoded cmd 0xa1> Error > Level: Recovered > Dec 30 17:45:04 oizfs scsi: [ID 107833 kern.notice] Requested Block: 0 > Error Block: 0 > Dec 30 17:45:04 oizfs scsi: [ID 107833 kern.notice] Vendor: ATA > Serial Number: > Dec 30 17:45:04 oizfs scsi: [ID 107833 kern.notice] Sense Key: Soft_Error > Dec 30 17:45:04 oizfs scsi: [ID 107833 kern.notice] ASC: 0x0 (no > additional sense info), ASCQ: 0x0, FRU: 0x0 > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- richard.ell...@richardelling.com +1-760-896-4422
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
