Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> No, why are they to be expected with SATA2 mode? Is the defect > specific to the SATA2 circuitry? I guess it could be a temporary > workaround provided they would eventually fix the problem in > firmware, but I'm getting new drives, so I guess I can't complain :-) Probably your new disks do this too, I really don't know whats with flawkey sata2 but I'd be quite sure it would fix your issues. Performance drop is not even noticeable, so it's worth a try. Yours Markus Kovero ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Hi, do you have disks connected in sata1/2? With WD2003FYYS-01T8B0/WD20EADS-00S2B0/WD1001FALS-00J7B1/WD1002FBYS-01A6B0 these timeouts are to be expected if disk is in SATA2 mode, No, why are they to be expected with SATA2 mode? Is the defect specific to the SATA2 circuitry? I guess it could be a temporary workaround provided they would eventually fix the problem in firmware, but I'm getting new drives, so I guess I can't complain :-) -- Maurice Volaski, maurice.vola...@einstein.yu.edu Computing Support, Rose F. Kennedy Center Albert Einstein College of Medicine of Yeshiva University ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
... > > I have identified the culprit is the Western Digital drive WD2002FYPS-01U1B0. > It's not clear if they can fix it in firmware, but Western Digital is > replacing my drives. > Feb 17 04:45:10 thecratewall scsi_status=0x0, ioc_status=0x804b, > scsi_state=0xc > Feb 17 04:45:10 thecratewall scsi: [ID 365881 kern.info] > /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): > Feb 17 04:45:10 thecratewall Log info 0x31110630 received for target 13. > Feb 17 04:45:10 thecratewall scsi_status=0x0, ioc_status=0x804b, > scsi_state=0xc Hi, do you have disks connected in sata1/2? With WD2003FYYS-01T8B0/WD20EADS-00S2B0/WD1001FALS-00J7B1/WD1002FBYS-01A6B0 these timeouts are to be expected if disk is in SATA2 mode, we've get rid of these timeouts after forcing disks in SATA1-mode with jumpers, now they only appear when disk is having real issues and needs to be replaced. Yours Markus Kovero ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
At 11:19 AM +1000 2/19/10, James C. McPherson wrote: On 19/02/10 12:51 AM, Maurice Volaski wrote: For those who've been suffering this problem and who have non-Sun jbods, could you please let me know what model of jbod and cables (including length thereof) you have in your configuration. For those of you who have been running xVM without MSI support, could you please confirm whether the devices exhibiting the problem are internal to your host, or connected via jbod. And if via jbod, please confirm the model number and cables. For those who've been suffering this problem and who have non-Sun jbods, could you please let me know what model of jbod and cables >(including length thereof) you have in your configuration. I have a SuperMicro X8DTN motheboard and an LSI SAS3081E-R, which is firmware 1.28.02.00-IT. I have 24 drives attached to the backplane of the system with a single mini-SAS cable probably not even 18 inches long. All the drives are WD RE4-GP. OpenSolaris, snv_130, is running on VMWare, but I am using PCI passthrough for the LSI card. It turns out the the mpt_sas HBAs are affected the same way: Hi Maurice, this is very interesting to note. I'll pass the info along to the relevant team (they're in Beijing, so away for another few days due to Spring Festival). I have identified the culprit is the Western Digital drive WD2002FYPS-01U1B0. It's not clear if they can fix it in firmware, but Western Digital is replacing my drives. Feb 17 04:45:10 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:45:10 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:45:10 thecratewall Log info 0x31110630 received for target 13. Feb 17 04:45:10 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:45:10 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:45:10 thecratewall Log info 0x31110630 received for target 13. Feb 17 04:45:10 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:45:10 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:45:10 thecratewall Log info 0x31110630 received for target 13. Feb 17 04:45:10 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 243001 kern.warning] WARNING: /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110630 Feb 17 04:47:57 thecratewall scsi: [ID 243001 kern.warning] WARNING: /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110630 Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall Log info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall Log info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall Log info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall Log info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall Log info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc -- James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog -- Maurice Volaski, maurice.vola...@einstein.yu.edu Computing Support, Rose F. Kennedy Center Albert Einstein College of Medicine of Yeshiva University___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Hello. We are seeing the problem on both Sun and non-Sun hardware. On our Sun thumper x4540, we can reproduce it on all 3 devices. Our configuration is large stripes with only 2 vdevs. Doing a simple scrub will show the typical mpt timeout. We are running snv_131. Somebody observed similar problems with new LSI 2008 SAS2 6Gb HBAs? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
On 19/02/10 12:51 AM, Maurice Volaski wrote: For those who've been suffering this problem and who have non-Sun jbods, could you please let me know what model of jbod and cables (including length thereof) you have in your configuration. For those of you who have been running xVM without MSI support, could you please confirm whether the devices exhibiting the problem are internal to your host, or connected via jbod. And if via jbod, please confirm the model number and cables. >For those who've been suffering this problem and who have non-Sun jbods, could you please let me know what model of jbod and cables >(including length thereof) you have in your configuration. I have a SuperMicro X8DTN motheboard and an LSI SAS3081E-R, which is firmware 1.28.02.00-IT. I have 24 drives attached to the backplane of the system with a single mini-SAS cable probably not even 18 inches long. All the drives are WD RE4-GP. OpenSolaris, snv_130, is running on VMWare, but I am using PCI passthrough for the LSI card. It turns out the the mpt_sas HBAs are affected the same way: Hi Maurice, this is very interesting to note. I'll pass the info along to the relevant team (they're in Beijing, so away for another few days due to Spring Festival). Feb 17 04:45:10 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:45:10 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:45:10 thecratewall Log info 0x31110630 received for target 13. Feb 17 04:45:10 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:45:10 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:45:10 thecratewall Log info 0x31110630 received for target 13. Feb 17 04:45:10 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:45:10 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:45:10 thecratewall Log info 0x31110630 received for target 13. Feb 17 04:45:10 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 243001 kern.warning] WARNING: /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110630 Feb 17 04:47:57 thecratewall scsi: [ID 243001 kern.warning] WARNING: /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110630 Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall Log info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall Log info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall Log info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall Log info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewall Log info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewall scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc -- James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> >>For those who've been suffering this problem and > who have non-Sun > >>jbods, could you please let me know what model of > jbod and cables > >>(including length thereof) you have in your > configuration. > >> We are seeing the problem on both Sun and non-Sun hardware. On our Sun thumper x4540, we can reproduce it on all 3 devices. Our configuration is large stripes with only 2 vdevs. Doing a simple scrub will show the typical mpt timeout. We are running snv_131. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
For those who've been suffering this problem and who have non-Sun jbods, could you please let me know what model of jbod and cables (including length thereof) you have in your configuration. For those of you who have been running xVM without MSI support, could you please confirm whether the devices exhibiting the problem are internal to your host, or connected via jbod. And if via jbod, please confirm the model number and cables. >For those who've been suffering this problem and who have non-Sun jbods, could you please let me know what model of jbod and cables >(including length thereof) you have in your configuration. I have a SuperMicro X8DTN motheboard and an LSI SAS3081E-R, which is firmware 1.28.02.00-IT. I have 24 drives attached to the backplane of the system with a single mini-SAS cable probably not even 18 inches long. All the drives are WD RE4-GP. OpenSolaris, snv_130, is running on VMWare, but I am using PCI passthrough for the LSI card. It turns out the the mpt_sas HBAs are affected the same way: Feb 17 04:45:10 thecratewallscsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:45:10 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:45:10 thecratewallLog info 0x31110630 received for target 13. Feb 17 04:45:10 thecratewallscsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:45:10 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:45:10 thecratewallLog info 0x31110630 received for target 13. Feb 17 04:45:10 thecratewallscsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:45:10 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:45:10 thecratewallLog info 0x31110630 received for target 13. Feb 17 04:45:10 thecratewallscsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 243001 kern.warning] WARNING: /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewallmptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110630 Feb 17 04:47:57 thecratewall scsi: [ID 243001 kern.warning] WARNING: /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewallmptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110630 Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewallLog info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewallscsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewallLog info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewallscsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewallLog info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewallscsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewallLog info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewallscsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 17 04:47:57 thecratewall scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt_sas0): Feb 17 04:47:57 thecratewallLog info 0x31110630 received for target 33. Feb 17 04:47:57 thecratewallscsi_status=0x0, ioc_status=0x804b, scsi_state=0xc -- Maurice Volaski, maurice.vola...@einstein.yu.edu Computing Support, Rose F. Kennedy Center Albert Einstein College of Medicine of Yeshiva University ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
For those who've been suffering this problem and who have non-Sun
jbods, could you please let me know what model of jbod and cables
(including length thereof) you have in your configuration.
For those of you who have been running xVM without MSI support,
could you please confirm whether the devices exhibiting the problem
are internal to your host, or connected via jbod. And if via jbod,
please confirm the model number and cables.
>For those who've been suffering this problem and who have non-Sun
jbods, could you please let me know what model of jbod and cables
>(including length thereof) you have in your configuration.
I have a SuperMicro X8DTN motheboard and an LSI SAS3081E-R, which is
firmware 1.28.02.00-IT. I have 24 drives attached to the backplane
of the system with a single mini-SAS cable probably not even 18
inches long. All the drives are WD RE4-GP.
OpenSolaris, snv_130, is running on VMWare, but I am using PCI
passthrough for the LSI card.
I tried the card and drives under Linux. Interestingly, it appears to
be causing similar messages. On top of that, I upgraded the
BIOS/firmware on the card to 1.29 (phase 17) and it made no
difference. Under OpenSolaris, bits were flipping all over the place.
It's not clear if that is happening here, but Linux just doesn't
know. Either way, the card's not behaving under Linux either.
Feb 10 09:01:17 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
Feb 10 09:01:17 [kernel] mptbase: ioc1: LogInfo(0x31112000):
Originator={PL}, Code={Reset}, SubCode(0x2000)
Feb 10 09:01:18 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
- Last output repeated 10 times -
Feb 10 09:03:54 [kernel] mptbase: ioc1: LogInfo(0x31112000):
Originator={PL}, Code={Reset}, SubCode(0x2000)
Feb 10 09:03:55 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
- Last output repeated 12 times -
Feb 10 09:05:39 [kernel] mptbase: ioc1: LogInfo(0x31112000):
Originator={PL}, Code={Reset}, SubCode(0x2000)
Feb 10 09:05:40 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
- Last output repeated 25 times -
Feb 10 09:10:01 [cron] (root) CMD (test -x /usr/sbin/run-crons &&
/usr/sbin/run-crons )
Feb 10 09:11:15 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
Feb 10 09:11:15 [kernel] mptbase: ioc1: LogInfo(0x31112000):
Originator={PL}, Code={Reset}, SubCode(0x2000)
Feb 10 09:11:17 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
- Last output repeated 13 times -
Feb 10 09:13:09 [kernel] mptbase: ioc1: LogInfo(0x31112000):
Originator={PL}, Code={Reset}, SubCode(0x2000)
Feb 10 09:13:10 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
- Last output repeated 11 times -
Feb 10 09:20:01 [cron] (root) CMD (test -x /usr/sbin/run-crons &&
/usr/sbin/run-crons )
- Last output repeated 2 times -
Feb 10 09:40:06 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
Feb 10 09:40:06 [kernel] mptbase: ioc1: LogInfo(0x31112000):
Originator={PL}, Code={Reset}, SubCode(0x2000)
Feb 10 09:40:07 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
- Last output repeated 10 times -
Feb 10 09:45:52 [kernel] mptbase: ioc1: LogInfo(0x31112000):
Originator={PL}, Code={Reset}, SubCode(0x2000)
Feb 10 09:45:53 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
- Last output repeated 10 times -
Feb 10 09:50:01 [cron] (root) CMD (test -x /usr/sbin/run-crons &&
/usr/sbin/run-crons )
Feb 10 09:58:56 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
Feb 10 09:58:56 [kernel] mptbase: ioc1: LogInfo(0x31112000):
Originator={PL}, Code={Reset}, SubCode(0x2000)
Feb 10 09:58:57 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
- Last output repeated 10 times -
Feb 10 10:00:01 [cron] (root) CMD (rm -f /var/spool/cron/lastrun/cron.hourly)
Feb 10 10:00:01 [cron] (root) CMD (test -x /usr/sbin/run-crons &&
/usr/sbin/run-crons )
Feb 10 10:00:01 [run-crons] (root) CMD (/etc/cron.hourly/hourly_woodchuck)
Feb 10 10:00:02 [sSMTP] Sent mail for r...@theradargun (221 2.0.0
Bye) uid=0 username=root outbytes=877
Feb 10 10:10:01 [cron] (root) CMD (test -x /usr/sbin/run-crons &&
/usr/sbin/run-crons )
- Last output repeated 3 times -
Feb 10 10:40:15 [kernel] mptbase: ioc1: LogInfo(0x31123000):
Originator={PL}, Code={Abort}, SubCode(0x3000)
Feb 10 10:40:15 [kernel] mptbase: ioc1: LogInfo(0x31112000):
Originator={PL}, Code={Reset}, SubCode(0x2000)
Feb 10 10:40:
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
For those who've been suffering this problem and who have non-Sun jbods, could you please let me know what model of jbod and cables (including length thereof) you have in your configuration. For those of you who have been running xVM without MSI support, could you please confirm whether the devices exhibiting the problem are internal to your host, or connected via jbod. And if via jbod, please confirm the model number and cables. For those who've been suffering this problem and who have non-Sun jbods, could you please let me know what model of jbod and cables (including length thereof) you have in your configuration. I have a SuperMicro X8DTN motheboard and an LSI SAS3081E-R, which is firmware 1.28.02.00-IT. I have 24 drives attached to the backplane of the system with a single mini-SAS cable probably not even 18 inches long. All the drives are WD RE4-GP. OpenSolaris, snv_130, is running on VMWare, but I am using PCI passthrough for the LSI card. I have set mpt:mpt_enable_msi=0 set mptsas:mptsas_enable_msi=0 in /etc/system and have rebooted. Here is an excerpt of messages during a scrub eb 6 19:13:24 thecratedoor scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt1): Feb 6 19:13:24 thecratedoorLog info 0x31123000 received for target 22. Feb 6 19:13:24 thecratedoorscsi_status=0x0, ioc_status=0x804b, scsi_state=0xc Feb 6 19:13:29 thecratedoor scsi: [ID 243001 kern.warning] WARNING: /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt1): Feb 6 19:13:29 thecratedoormpt_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31123000 Feb 6 19:13:29 thecratedoor scsi: [ID 243001 kern.warning] WARNING: /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt1): Feb 6 19:13:29 thecratedoormpt_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31123000 Feb 6 19:13:29 thecratedoor scsi: [ID 243001 kern.warning] WARNING: /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt1): Feb 6 19:13:29 thecratedoormpt_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31123000 Feb 6 19:13:29 thecratedoor scsi: [ID 243001 kern.warning] WARNING: /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt1): Feb 6 19:13:29 thecratedoormpt_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31123000 Feb 6 19:13:30 thecratedoor scsi: [ID 365881 kern.info] /p...@0,0/pci15ad,7...@15/pci1000,3...@0 (mpt1): Feb 6 19:13:30 thecratedoorLog info 0x31123000 received for target 20. Feb 6 19:13:30 thecratedoorscsi_status=0x0, ioc_status=0x804b, scsi_state=0xc scrub: scrub in progress for 3h28m, 50.53% done, 3h23m to go config: NAME STATE READ WRITE CKSUM cratepooldoor ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 c4t1d0 ONLINE 0 0 0 c4t2d0 ONLINE 0 0 0 c4t3d0 ONLINE 0 0 0 c4t4d0 ONLINE 0 0 0 18K repaired c4t5d0 ONLINE 0 0 0 c4t6d0 ONLINE 0 0 0 512 repaired c4t7d0 ONLINE 0 0 0 c4t8d0 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 c4t9d0 ONLINE 0 0 0 c4t10d0 ONLINE 0 0 0 c4t11d0 ONLINE 0 0 0 c4t12d0 ONLINE 0 0 0 128K repaired c4t13d0 ONLINE 0 0 0 c4t14d0 ONLINE 0 0 0 c4t15d0 ONLINE 0 0 0 c4t16d0 ONLINE 0 0 0 raidz1-2 ONLINE 0 0 0 c4t17d0 ONLINE 0 0 0 18.5K repaired c4t18d0 ONLINE 0 0 0 18.5K repaired c4t19d0 ONLINE 0 0 0 c4t20d0 ONLINE 0 0 0 1K repaired c4t21d0 ONLINE 0 0 0 c4t22d0 ONLINE 0 0 0 256K repaired c4t23d0 ONLINE 0 0 0 c4t24d0 ONLINE 0 0 0 109K repaired I have another identical system and it's behaving the same way. -- Maurice Volaski, maurice.vola...@einstein.yu.edu Computing Support, Rose F. Kennedy Center Albert Einstein College of Medicine of Yeshiva University___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Hi Simon > I.e. you'll have to manually intervene > if a consumer drive causes the system to hang, and > replace it, whereas the RAID edition drives will > probably report the error quickly and then ZFS will > rewrite the data elsewhere, and thus maybe not kick > the drive. IMHO the relevant aspects are if ZFS is able to give accurate account on cache flush status and even realize if a drive is not responsive. That being said, I have no seen a specific report that ZFS would kick green drives at random or at pattern, like the poor SoHo storage enclosure users do all the time. > > So it sounds preferable to have TLER in operation, if > one can find a consumer-priced drive that allows it, > or just take the hit and go with whatever non-TLER > drive you choose and expect to have to manually > intervene if a drive plays up. OK for home user where > he is not too affected, but not good for businesses > which need to have something recovered quickly. One point about TLER is that two error correction schemes concur in the case you run a consumer drive on an active RAID controller that has its own mechanisms. When you run ZFS on a RAID controller in contrast to the best practise recommendations, an analogue question arises. On the other hand, if you run a green consumer drive on a dumb HBA , I wouldn't know what is wrong with it in the first place. As much as for manual interventions, the only one I am aware of would be to re-attach a single drive. Not an option if you are really affected like those miserable Thecus N7000 users that see the entire array of only a handful of drives drop out within hours - over and over again, or not even get to finish formatting the stripe set. The dire consequences of the gossiped TLER problems let me believe that there would be much more and quite specific reports in this place if this was a systematic issue with ZFS. Other than that, we are operating outside supported specs when running consumer level drives in large arrays. So far at least the perspective of Seagate and WD. > > > That all rather points to singular issues with > > firmware bugs or similar than to a systematic > issue, > > doesn't it? > > I'm not sure. Some people in the WDC threads seem to > report problems with pauses during media streaming > etc. This was again for SoHo storage enclosures - not for ZFS, right? > when the > 32MB+ cache is empty, then it loads another 32MB into > cache etc and so on? I am not sure if any current disk will have such a simplistic cache management that will draw upon completely cycling the buffer content, let alone for reads that belong to a single file (a disk basically is agnostic of files). Moreover, such a buffer management would be completely useless for a striped array. I don't know much better what a disk cache does either, but I am afraid that direction is probably not helpful to understanding certain phenomenons people have reported. I think that at this time we are seeing a quite large amount of evolutions going on in disk storage, whereas many established assumptions are being abandoned while backwards compatibility is not always taken care of. SAS 6G (will my controller really work in a PCIe 1.1 slot?) and 4k clusters are certainly only prominent examples. It's probably even more true than ever to fall back to established technologies in such times, including of biting the bullet of cost premium on occasion. Best regards Tonmaus -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Hi Tonmaus, > they are the new revision. OK. > I got the impression as well that the complaints you > reported were mainly related to embedded Linux > systems probably running LVM / mda. (thecus, Qnap, > ) Other reports I had seen related to typical HW > raids. I don't think the situation is comparable to > ZFS. That could be the case, but maybe I'll have to create a specific thread along the lines of "Anyone having success / problems with WD Green drives?" in order to know a bit more details. There were Mac users also complaining -- see the WDC links in the "Best 1.5TB drives" thread. > I have also followed some TLER related threads here. > I am not sure if there was ever a clear assertion if > consumer drive related Error correction will affect a > ZFS pool or not. Statistically we should have a lot > of "restrictive TLER settings helped me to solve my > ZFS pool issues" success reports here, if it were. IIRC I think Richard said that he thought that a troublesome non-RAID drive would affect MTTR and not reliability. I.e. you'll have to manually intervene if a consumer drive causes the system to hang, and replace it, whereas the RAID edition drives will probably report the error quickly and then ZFS will rewrite the data elsewhere, and thus maybe not kick the drive. So it sounds preferable to have TLER in operation, if one can find a consumer-priced drive that allows it, or just take the hit and go with whatever non-TLER drive you choose and expect to have to manually intervene if a drive plays up. OK for home user where he is not too affected, but not good for businesses which need to have something recovered quickly. > That all rather points to singular issues with > firmware bugs or similar than to a systematic issue, > doesn't it? I'm not sure. Cheers, Simon -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Hi Simon, they are the new revision. I got the impression as well that the complaints you reported were mainly related to embedded Linux systems probably running LVM / mda. (thecus, Qnap, ) Other reports I had seen related to typical HW raids. I don't think the situation is comparable to ZFS. I have also followed some TLER related threads here. I am not sure if there was ever a clear assertion if consumer drive related Error correction will affect a ZFS pool or not. Statistically we should have a lot of "restrictive TLER settings helped me to solve my ZFS pool issues" success reports here, if it were. That all rather points to singular issues with firmware bugs or similar than to a systematic issue, doesn't it? Cheers, Tonmaus -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Looks like I got the textbook response from Western Digital: --- Western Digital technical support only provides jumper configuration and physical installation support for hard drives used in systems running the Linux/Unix operating systems. For setup questions beyond physical installation of your Western Digital hard drive, please contact the vendor of your Linux/Unix operating system. Please install the drive in an XP or Vista system to test the drive following the information below. --- etc. So, doesn't look like I'll get any kind of reasonable response personally. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
If I'm not mistaken then the WD2002FYPS is an enterprise model: WD RE4-GP (RAID Edition, Green Power), so you almost certainly have the firmware that allows (1) the idle time before spindown to be modified with WDIDLE3.EXE and (2) the error reporting time to be modified with WDTLER.EXE. So I expect your drives are spinning down to save power as they are Green series drives. But if this spindown is causing odd things to happen you could see if it's possible to increase the spindown time with WDIDLE3.EXE. Let us know if you get any news back from WD. Cheers, Simon -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> That's good to hear. Which revision are they: 00R6B0 > or 00P8B0? It's marked on the drive top. Interesting. I wonder if this is the issue too with the 01U1B0 2.0TB drives? I have 24 WD2002FYPS-01U1B0 drives under OpenSolaris with an LSI 1068E controller that have weird timeout issues and I have 4 more on a 3ware 9650SE (a 9650SE-4LPML to be exact) under Linux which often show up with a status of DEVICE-ERROR even though the drive otherwise appears to be fine. All of this could be explained by these drives not responding in a timely manner I think to the controller. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Hi Tonmaus, That's good to hear. Which revision are they: 00R6B0 or 00P8B0? It's marked on the drive top. >From what I've seen elsewhere, people seem to be complaining about the newer >00P8B0 revision, so I'd be interested to hear from you. These revision numbers >are listed in the first post of the thread below, and refer to the 1.5TB model >(WD15EADS), but might also be applicable to the WD20EADS model too. http://opensolaris.org/jive/thread.jspa?threadID=121871 Cheers, Simon -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Hi Simon, I am running 5 WD20EADS in a raidz-1+spare on ahci controller without any problems I could relate to TLER or head parking. Cheers, Tonmaus -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> My timeout issue is definitely the WD10EARS disks. > WD has chosen to cripple their consumer grade disks > when used in quantities greater than one. > > I'll now need to evaluate alternative supplers of low > cost disks for low end high volume storage. > > Mark. > > typo ST32000542AS not NS This was the conclusion I came to. I'm also on the hunt for some decent consumer-priced drives for use in a ZFS RAID setup, and I created a thread to try to find which ones people recommend. See here: http://opensolaris.org/jive/thread.jspa?threadID=121871 So far, I'm inclined to think that the Samsung HD154UI 1.5TB, and possibly the Samsung HD203WI 2TB drives might be the most reliable choices at the moment, based on the data in that thread and checking user reports. Cheers, Simon http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/ -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
The results are in: My timeout issue is definitely the WD10EARS disks. Although differences in the error rate was seen with different LSI firmware revisions, the errors persisted. The more disks on the expander, the higher the number with iostat errors. This then causes zpool issues (disk failures, resilvering etc.) I replaced 24 of them with ST32000542NS (f/w CC34), and the problem departed with the WD disks. Full scrub of 1.5Tb, not one error seen anywhere. WD has chosen to cripple their consumer grade disks when used in quantities greater than one. I'll now need to evaluate alternative supplers of low cost disks for low end high volume storage. Mark. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
An update: Well things didn't quite turn out as expected. I decided to follow the path right to the disks for clues. Digging into the adapter diags with LSIUTIL, revealed an Adapter Link issue. Adapter Phy 5: Link Down Invalid DWord Count 5,969,575 Running Disparity Error Count 5,782,581 Loss of DWord Synch Count 0 Phy Reset Problem Count 0 After replacing cables, I eventually replaced the controller and then things really went pear shaped. It turns out the backplane, that ran without major issues on the Supermicro controller, refused to operate with the LSI SAS3081E-R (with latest code)- card wouldn't initialise, links only ran at 1.5Mb/s, most disks offline etc. Replacing the backplane (whole jbod) fixed the Adapter Link problems, but timeouts still occur when scrubbing. Oh look, the dev names moved. they used to start at c4t8d0, but it has "made it right" all by itself. EYHOBG! iostat -X -e -n s/w h/w trn tot device 0 0 0 0 c4t0d0 0 0 0 0 c4t1d0 0 2 8 10 c4t2d0 0 3 18 21 c4t3d0 0 0 0 0 c4t4d0 0 2 12 14 c4t5d0 0 1 8 9 c4t6d0 0 2 15 17 c4t7d0 0 0 0 0 c4t8d0 0 0 0 0 c4t9d0 0 0 0 0 c4t10d0 0 0 0 0 c4t11d0 0 0 0 0 c4t12d0 0 0 0 0 c4t13d0 0 11 84 95 c4t41d0 0 8 62 70 c4t42d0 0 10 72 82 c4t43d0 0 19 147 166 c4t44d0 0 12 102 114 c4t45d0 0 19 145 164 c4t46d0 0 13 108 121 c4t47d0 0 7 62 69 c4t48d0 0 14 113 127 c4t49d0 0 11 96 107 c4t50d0 0 11 91 102 c4t51d0 0 8 64 72 c4t52d0 0 13 108 121 c4t53d0 0 11 106 117 c4t54d0 0 10 82 92 c4t55d0 0 10 88 98 c4t56d0 0 12 85 97 c4t57d0 0 6 38 44 c4t58d0 and status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. c4t2d0 ONLINE 0 0 1 25.5K repaired c4t55d0 ONLINE 0 0 4 102K repaired I do note that after these errors, there are no errors in the lsi adapter diag logs. Data disks are all new WD10EARS. If the OpenSolaris and ZFS combination wasn't so robust, this would have ended badly. Next step will be trying different timeout settings on the controller and see if that helps. P.S. I have a client with a "suspect", nearly full, 20Tb zpool to try to scrub, so this is a big issue for me. A resilver of a 1Tb disk takes up to 40 hrs., so I expect a scrub to be a week (or two), and at present, would probably result in multiple disk failures. Mark. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
I would definitely be interested to see if the newer firmware fixes the problem for you. I have a very similar setup to yours, and finally forcing the firmware flash to 1.26.00 of my on-board LSI 1068E on a SuperMicro H8DI3+ running snv_131 seemed to address the issue. I'm still waiting to see if that's entirely the case, but so far so good (even with a LOT of disk activity to clean up the very messy zpool which had resulted from all of the disk timeouts/errors previously). -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
I can produce the timeout error on multiple, similar servers. These are storage servers, so no zones or gui running. Hardware: Supermicro X7DWN with AOC-USASLP-L8i controller E1 (single port) backplanes (16 & 24 bay) (LSILOGICSASX28 A.0 and LSILOGICSASX36 A.1) up to 36 1Tb WD Sata disks This server has 2 x quad core Intel CPU & 16Gb ram. Disks: WD 1Tb c4t12d0 to c4t47d0 as single raidz pool. (6 disks per set) Running dev 131. I see problem on 2009.06 as well. I note that the latest AOC-USASLP-L8i firmware is LSI Rev 1.26.00.00, which I believe does not support MSI. (working on Supermicro to update the firmware) I have an LSI controller to swap for the AOC-USASLP-L8i with latest firmware, which I can retest with. After a few hours of light load, no errors appear unless I initiate a scrub. iostat -X -e -n errors --- s/w h/w trn tot device 0 0 0 0 fd0 0 9 0 9 c5t1d0 0 0 0 0 c4t8d0 0 0 0 0 c4t9d0 0 0 0 0 c4t12d0 0 0 0 0 c4t13d0 0 0 0 0 c4t14d0 0 0 0 0 c4t15d0 0 0 0 0 c4t16d0 0 0 0 0 c4t17d0 0 0 0 0 c4t18d0 0 0 0 0 c4t19d0 0 0 0 0 c4t20d0 0 0 0 0 c4t21d0 0 0 0 0 c4t22d0 0 0 0 0 c4t23d0 0 0 0 0 c4t30d0 0 1 10 11 c4t31d0 0 2 20 22 c4t32d0 0 0 0 0 c4t33d0 0 0 0 0 c4t34d0 0 0 0 0 c4t35d0 0 0 0 0 c4t36d0 0 0 0 0 c4t37d0 0 0 0 0 c4t38d0 0 0 0 0 c4t39d0 0 0 0 0 c4t40d0 0 0 0 0 c4t41d0 0 0 0 0 c4t42d0 0 1 10 11 c4t43d0 0 3 31 34 c4t44d0 0 1 10 11 c4t45d0 0 2 20 22 c4t46d0 0 1 10 11 c4t47d0 0 0 0 0 c4t48d0 0 0 0 0 c4t49d0 0 0 0 0 c4t50d0 0 0 0 0 c4t51d0 0 0 0 0 c4t52d0 In this instance, all errors are on the same (24 bay) backplane. I have also had them on the 16 bay backplane with this 2 chassis configuration. The problem becomes more of a pain when drives drop off for a short period, then reconnect and resilver or occassionally just stop until a reboot or hot plug. The robustness of ZFS certainly helps keep things running. Mark. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Can't say when the problems may have been introduced, but it looks like we've got my report (b104) and another report from b111 of issues with the 1068E. The IR firmware seems to do some sort of internal multipathing while the IT firmware doesn't do any. With the IT firmware, I enabled multipathing (in-between frequent reboots due to the card hanging under load) and had very good performance with the round-robin policy. When I reverted to the IR firmware to fix my issues, I found that round-robin slowed down my disk benchmark tests (IOZone). My guess is because it was trying to use pathways that the IR firmware had marked standby internally. Disabling the round-robin policy almost doubled throughput under the IR firmware. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
I'm glad I was able to help someone. My card is also a 3081E-R (B3). It shipped to me with the IR firmware, and I immediately flashed the IT firmware on it because I had heard it was supposed to be (better, faster, stable, shiny) with Solaris and ZFS. The motherboard on that server has an LSI 2008 (SAS 2.0) chip onboard. My hope is that I'll be able to upgrade in the future with less excitement than I had trying to get the 1068E to work. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
I've spent all weekend fighting this problem on our storage server after installing a ZFS log device, and your suggestion fixed it! I also have a LSI 3081E-R adapter (B3 revision) connected to a SAS expander backplane with 7 drives on it. None of the /etc/system options mentioned in this thread worked, but after switching to the LSI 1.29 IR firmware, the controller no longer hangs constantly. I'm using OpenSolaris 9.06 (snv 111b). I still see warnings like this in dmesg occassionally: Dec 6 14:15:04 wid scsi: [ID 365881 kern.info] /p...@0,0/pci1166,1...@9/pci1000,3...@0 (mpt0): Dec 6 14:15:04 wid Log info 0x31123000 received for target 26. Dec 6 14:15:04 wid scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc but I have not observed any problems yet while I scrub the zpool. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
I was under the impression that the problem affecting most of us was introduced much later than b104, sometime between ~114 and ~118. When I first started using my LSI 3081 cards, they had the IR firmware on them, and it caused me all kinds of problems. The disks showed up but I couldn't write to them, I believe. Eventually I found that I needed the IT firmware for it to work properly, which is what I have used ever since, but maybe some builds do work with IR firmware? I remember, then, when I was originally trying to set them up with the IR firmware, Opensolaris saw my two cards as one device, whereas with the IT firmware they were always mpt0 and mpt1. Could also be the IR works with one card but not well when two cards are combine... Chad On Sat, Dec 05, 2009 at 02:47:55PM -0800, Calvin Morrow wrote: > I found this thread after fighting the same problem in Nexenta which uses the > OpenSolaris kernel from b104. Thankfully, I think I have (for the moment) > solved my problem. > > Background: > > I have an LSI 3081e-R (1068E based) adapter which experiences the same > disconnected command timeout error under relatively light load. This card > connects to a Supermicro chassis using 2 MiniSAS cables to redundant > expanders that are attached to 18 SAS drives. The card ran the latest IT > firmware (1.29?). > > This server is a new install, and even installing from the CD to two disks in > a mirrored ZFS root would randomly cause the disconnect error. The system > remained unresponsive until after a reboot. > > I tried the workarounds mentioned in this thread, namely using "set > mpt:mpt_enable_msi = 0" and "set xpv_psm:xen_support_msi = -1" in > /etc/system. Once I added those lines, the system never really became > unresponsive, however there were partial read and partial write messages that > littered dmesg. At one point there appeared to be a disconnect error ( can > not confirm ) that the system recovered from. > > Eventually, I became desperate and flashed the IR (Integrated Raid) firmware > over the top of the IT firmware. Since then, I have had no errors in dmesg > of any kind. > > I even removed the workarounds from /etc/system and still have had no issues. > The mpt driver is exceptionally quiet now. > > I'm interested to know if anyone who has a 1068E based card is having these > problems using the IR firmware, or if they all seem to be IT (initiator > target) related. > -- > This message posted from opensolaris.org > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
I found this thread after fighting the same problem in Nexenta which uses the OpenSolaris kernel from b104. Thankfully, I think I have (for the moment) solved my problem. Background: I have an LSI 3081e-R (1068E based) adapter which experiences the same disconnected command timeout error under relatively light load. This card connects to a Supermicro chassis using 2 MiniSAS cables to redundant expanders that are attached to 18 SAS drives. The card ran the latest IT firmware (1.29?). This server is a new install, and even installing from the CD to two disks in a mirrored ZFS root would randomly cause the disconnect error. The system remained unresponsive until after a reboot. I tried the workarounds mentioned in this thread, namely using "set mpt:mpt_enable_msi = 0" and "set xpv_psm:xen_support_msi = -1" in /etc/system. Once I added those lines, the system never really became unresponsive, however there were partial read and partial write messages that littered dmesg. At one point there appeared to be a disconnect error ( can not confirm ) that the system recovered from. Eventually, I became desperate and flashed the IR (Integrated Raid) firmware over the top of the IT firmware. Since then, I have had no errors in dmesg of any kind. I even removed the workarounds from /etc/system and still have had no issues. The mpt driver is exceptionally quiet now. I'm interested to know if anyone who has a 1068E based card is having these problems using the IR firmware, or if they all seem to be IT (initiator target) related. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Perhaps. As I noted though, it also occurs on the onboard NVidia SATA controller when MSI is enabled. I had already put a line in /etc/system to disable MSI for that controller per a forum thread and it worked great. I'm now running with all MSI disabled via XVM as the mpt controller is giving me the same problems. As it's happening on totally different controller types, cable types, and drive types, I have to go with software issues. I know for sure the NVidia issue didn't come up on 2009.06. It makes the system take forever to boot, so it's very noticeable. It happened when I first went to dev builds, I want to say it was around b118. I updated for better XVM support for newer Linux kernels. The NVidia controller causes similar log messages. Command timeouts. Disabling MSIs fixes it as well. Motherboard is an Asus M4N82 Deluxe. NVIDIA nForce 980a SLI chipset. I expect the root cause is the same, and I would guess that something is causing the drivers to miss or not receive some interrupts. However, my programming at this level is limited, so perhaps I'm misdiagnosing the issue. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Just an update, my scrub completed without any timeout errors in the log. XVM with MSI disabled globally. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> Chenbro 16 hotswap bay case. It has 4 mini backplanes that each connect via > an SFF-8087 cable > StarTech HSB430SATBK hmm, both are passive backplanes with one SATA tunnel per link... no SAS Expanders (LSISASx36) like those found in SuperMicro or J4x00 with 4 links per connection. wonder if there is a LSI issue with too many links in HBA mode? Rob ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> (1) disabling MSI support in xVM makes the problem go > away Yes here. > (6) mpt(7d) without MSI support is sloow. That does seem to be the case. It's not so bad overall, and at least the performance is consistent. It would be nice if this were improved. > For those of you who have been running xVM without > MSI support, > could you please confirm whether the devices > exhibiting the problem > are internal to your host, or connected via jbod. And > if via jbod, > please confirm the model number and cables. Direct connect. The drives are in hot-swap racks, but they are passive devices. No expanders or anything like that in there. In case it's interesting, the racks are StarTech HSB430SATBK devices. I'm using SAS to SATA breakout cables to connect them. I have tried different lengths with the same result. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Hi, I just posted a summary of a similiar issue I'm having with non-Sun hardware. For the record, it's in a Chenbro RM41416 chassis with 4 chenbro SAS backplanes but no expanders (each backplane is 4 disks connected by SFF-8087 cable). Each of my LSI brand SAS3081E PCI-E cards is connected to two backplanes with 1m SFF-8087 (both ends) cables. For more details if they are important see my other post. I haven't tried the MSI workaround yet (although I'm not sure what MSI is) but from what I've read the workaround won't fix the issues in my case with non-sun hardware. Thanks, Chad On Tue, Dec 01, 2009 at 12:36:33PM +1000, James C. McPherson wrote: > Hi all, > I believe it's an accurate summary of the emails on this thread > over the last 18 hours to say that > > (1) disabling MSI support in xVM makes the problem go away > > (2) disabling MSI support on bare metal when you only have > disks internal to your host (no jbods), makes the problem > go away > (several reports of this) > > (3) disabling MSI support on bare metal when you have a non-Sun > jbod (and cables) does _not_ make the problem go away. > (several reports of this) > > (4) the problem is not seen with a Sun-branded jbod and cables > (only one report of this) > > (5) problem is seen with both mpt(7d) and itmpt(7d). > > (6) mpt(7d) without MSI support is sloow. > > > For those who've been suffering this problem and who have non-Sun > jbods, could you please let me know what model of jbod and cables > (including length thereof) you have in your configuration. > > For those of you who have been running xVM without MSI support, > could you please confirm whether the devices exhibiting the problem > are internal to your host, or connected via jbod. And if via jbod, > please confirm the model number and cables. > > Please note that Jianfei and I are not making assumptions about the > root cause here, we're just trying to nail down specifics of what > seems to be a likely cause. > > > thankyou in advance, > James C. McPherson > -- > Senior Kernel Software Engineer, Solaris > Sun Microsystems > http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Hi all, I believe it's an accurate summary of the emails on this thread over the last 18 hours to say that (1) disabling MSI support in xVM makes the problem go away (2) disabling MSI support on bare metal when you only have disks internal to your host (no jbods), makes the problem go away (several reports of this) (3) disabling MSI support on bare metal when you have a non-Sun jbod (and cables) does _not_ make the problem go away. (several reports of this) (4) the problem is not seen with a Sun-branded jbod and cables (only one report of this) (5) problem is seen with both mpt(7d) and itmpt(7d). (6) mpt(7d) without MSI support is sloow. For those who've been suffering this problem and who have non-Sun jbods, could you please let me know what model of jbod and cables (including length thereof) you have in your configuration. For those of you who have been running xVM without MSI support, could you please confirm whether the devices exhibiting the problem are internal to your host, or connected via jbod. And if via jbod, please confirm the model number and cables. Please note that Jianfei and I are not making assumptions about the root cause here, we're just trying to nail down specifics of what seems to be a likely cause. thankyou in advance, James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Carson Gaspar wrote: Mark Johnson wrote: I think there are two different bugs here... I think there is a problem with MSIs and some variant of mpt card on xVM. These seem to be showing up as timeout errors. Disabling MSIs for this adapter seems to fix this problem. For folks seeing this problem, what HBA adapter are you using that you see this problem on? I have just confirmed that adding "set mpt:mpt_enable_msi = 0" and removing "set xpv_psm:xen_support_msi = -1" in /etc/system also fixes the problem for me. I am running an LSI branded SAS3081E-R with directly attached SATA disks. See my previous email for full system info. For the record, I am _not_ seeing the other command error problem. But I don't have an external chassis, expanders, etc. I spoke too soon, I came back home to: Nov 30 15:52:19 gandalf.taltos.org scsi: [ID 107833 kern.warning] WARNING: /p...@0,0/pci8086,2...@1c/pci1000,3...@0 (mpt0): Nov 30 15:52:19 gandalf.taltos.org Disconnected command timeout for Target 11 Nov 30 15:52:19 gandalf.taltos.org scsi: [ID 365881 kern.notice] /p...@0,0/pci8086,2...@1c/pci1000,3...@0 (mpt0): Nov 30 15:52:19 gandalf.taltos.org Log info 0x3114 received for target 11. Nov 30 15:52:19 gandalf.taltos.org scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc Nov 30 15:52:19 gandalf.taltos.org scsi: [ID 365881 kern.notice] /p...@0,0/pci8086,2...@1c/pci1000,3...@0 (mpt0): Nov 30 15:52:19 gandalf.taltos.org Log info 0x3114 received for target 11. Nov 30 15:52:19 gandalf.taltos.org scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc Nov 30 15:52:19 gandalf.taltos.org scsi: [ID 365881 kern.notice] /p...@0,0/pci8086,2...@1c/pci1000,3...@0 (mpt0): Nov 30 15:52:19 gandalf.taltos.org Log info 0x3113 received for target 11. Nov 30 15:52:19 gandalf.taltos.org scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc Reverting to "set xpv_psm:xen_support_msi = -1" now... -- Carson ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
On Nov 30, 2009, at 2:14 PM, Carson Gaspar wrote: Mark Johnson wrote: I think there are two different bugs here... I think there is a problem with MSIs and some variant of mpt card on xVM. These seem to be showing up as timeout errors. Disabling MSIs for this adapter seems to fix this problem. For folks seeing this problem, what HBA adapter are you using that you see this problem on? I have just confirmed that adding "set mpt:mpt_enable_msi = 0" and removing "set xpv_psm:xen_support_msi = -1" in /etc/system also fixes the problem for me. I am running an LSI branded SAS3081E-R with directly attached SATA disks. See my previous email for full system info. For the record, I am _not_ seeing the other command error problem. But I don't have an external chassis, expanders, etc. I'm using a LSI Logic SAS1068E controller (according to lsiutil) and NOT using XVM and seeing these problems. I just put 'set mpt:mpt_enable_msi = 0' into my /etc/system and rebooted, we'll see how that works. If this seems to solve the issue for this machine I'll do it on all of our others as well (we currently have about 12 of these running, about 1.5PB of data online on them) and report back. I just basically wanted to chime in and mention that I'm also having these problems but NOT using XVM as other people are. -Jeremy ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Mark Johnson wrote: I think there are two different bugs here... I think there is a problem with MSIs and some variant of mpt card on xVM. These seem to be showing up as timeout errors. Disabling MSIs for this adapter seems to fix this problem. For folks seeing this problem, what HBA adapter are you using that you see this problem on? I have just confirmed that adding "set mpt:mpt_enable_msi = 0" and removing "set xpv_psm:xen_support_msi = -1" in /etc/system also fixes the problem for me. I am running an LSI branded SAS3081E-R with directly attached SATA disks. See my previous email for full system info. For the record, I am _not_ seeing the other command error problem. But I don't have an external chassis, expanders, etc. -- Carson ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> o The problems are not seen with Sun's version of > this card Unable to comment as I don't have a Sun card here. If Sun would like to send me one, I would be willing to test it compared to the cards I do have. I'm running Supermicro USAS-L8i cards (LSI 1068e based). > o The problems are not seen with LSI's version of > the driver I haven't tried it as comments from Sun staff here have indicated that it's not a good idea. > o The problems are seen with the latest LSI > firmware Yes. When I checked, the LSI site was listing the version I see at boot. > o Errors still occur if MSIs are disabled. I haven't seen any command timeout errors since disabling MSIs. I tried using the command to disable MSI only for the MPT driver, but I get a similar error from the NVidia driver at that point as it has my boot drives. It seems to me that the issue seems to have more in common with MSIs than the drivers themselves. I do have a scrub scheduled for 12/1, so I can check the logs after than to see if it appears from that. My other tests have not triggered the issue since disabling MSIs. I'm currently running with "set xpv_psm:xen_support_msi = -1". I am not using any jbod enclosures. My setup uses SAS to SATA breakout cables and connect directly to the drives. I have tried different cables and lengths. The timeouts affected drives in a seemingly random fashion. I would get timeouts on both controllers and every drive over time. I have never had command errors here. Just the timeouts. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> Can folks confirm/deny each of these? > > o The problems are not seen with Sun's version of > this card On the Thumper x4540 (which uses 6 of the same LSI 1068E controller chips), we do not see this problem. Then again, it uses a one-to-one mapping of controller PHY ports to internal disks; no JBODs or expanders here. 1 controller per 8 disks is a much more performance-oriented ratio, so I don`t expect to see the problem there. We have not tried using a Sun re-branded LSI controller for the external JBODs. My understanding is that Sun uses a custom firmware derived from LSIs public offerings. > o The problems are not seen with LSI's version of > the driver Incorrect. We have tried using the latest itmpt driver from LSI and see the same problem. > o The problems are seen with the latest LSI > firmware Correct. We`ve tried Phase 15, 16 and 17. All exhibit the same problem. > o Errors still occur if MSIs are disabled. They > seem to > occur less frequently. Were timeouts being seen > before > MSIs were disabled? If so, are timeouts being > seen after > MSIs were disabled? Correct: disabling the MSIs did not affect the problem, although they did slow the IO on the system down enough to delay the onset of the problem a few hours. Timeouts were being seen before disabling MSIs and they are usually coupled with bus resets, which is standard behaviour for the sd driver if an IO is timed out for too long, I believe. > folks seeing the command failures, what are you using > for > a jbod? Is there firmware on the jbod, and if so, is > it up > to date? Have you tried a different jbod? Are the > command failures > tied to a subset of the disks or effect all of them? > Have you tried > a different length cable? > We use Dell DCS J23 JBODs (23 disk enclosure), 2 per LSI3801E, fully populated with enterprise-grade WD SATA drives. We`ve tried both R105 and R106 firmware (both latest production-grade firmware) on them with no differences. The problem affects all disks in the JBOD(s), not specific ones. Usually one or two disks start to timeout which snowballs into all of them when the bus resets. We have 15 of these systems running, all with the same config using 2 foot external cables...changing cables doesn`t help. We have not tried using a different JBOD. - Adam -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
James C. McPherson wrote: Adam Cheal wrote: I thought you had just set set xpv_psm:xen_support_msi = -1 which is different, because that sets the xen_support_msi variable which lives inside the xpv_psm module. Setting mptsas:* will have no effect on your system if you do not have an mptsas card installed. The mptsas cards are not generally available yet (they're 2nd generation), so I would be surprised if you had one. No...I had set the other two variables after Mark contacted me offline to do some testing mainly to verify the problem was, indeed, not xVM specific. I had added the mptsas line as well, as per his recommendations, because I wasn't sure if there was some crossover between it and using the MPT driver for a SAS card. Thanks for clearing that up though...obviously we don't need it for the LSI3801E we are using. Ah, ok I think there are two different bugs here... I think there is a problem with MSIs and some variant of mpt card on xVM. These seem to be showing up as timeout errors. Disabling MSIs for this adapter seems to fix this problem. For folks seeing this problem, what HBA adapter are you using that you see this problem on? The second problem is there appears to be a problem with mpt and the LSI3801E. These seem to be command errors than timeouts? Not sure if they were seeing timeouts too? I believe the following things were stated across the thread.. Can folks confirm/deny each of these? o The problems are not seen with Sun's version of this card o The problems are not seen with LSI's version of the driver o The problems are seen with the latest LSI firmware o Errors still occur if MSIs are disabled. They seem to occur less frequently. Were timeouts being seen before MSIs were disabled? If so, are timeouts being seen after MSIs were disabled? i.e. are there two different problems here, timeouts, and then commands failing.. If so, maybe these are unrelated problems? For folks seeing the command failures, what are you using for a jbod? Is there firmware on the jbod, and if so, is it up to date? Have you tried a different jbod? Are the command failures tied to a subset of the disks or effect all of them? Have you tried a different length cable? Thanks, MRJ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Tru Huynh wrote: On Sat, Nov 21, 2009 at 07:08:20PM +1000, James C. McPherson wrote: If you and everybody else who is seeing this problem could provide details about your configuration (output from cfgadm -lva, raidctl -l, prtconf -v, what your zpool configs are, and the firmware rev of each disk in each zpool) that would help us sort through and find any commonalities and hopefully a fix. On a supermicro board, with 3 hw raid6 vdev joined in a single pool, random hangs ( nothing). Attached the requested files on 2 machines having the same issue. Two things here: (1) your hba is a MegaRAID SAS ELP, which uses the mega_sas driver not mpt, and (2) If you've got nothing in the logs, then you need to do more investigation to work out where the problem lies. Since your system is not using mpt, you have a different problem. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Adam Cheal wrote: I thought you had just set set xpv_psm:xen_support_msi = -1 which is different, because that sets the xen_support_msi variable which lives inside the xpv_psm module. Setting mptsas:* will have no effect on your system if you do not have an mptsas card installed. The mptsas cards are not generally available yet (they're 2nd generation), so I would be surprised if you had one. No...I had set the other two variables after Mark contacted me offline to do some testing mainly to verify the problem was, indeed, not xVM specific. I had added the mptsas line as well, as per his recommendations, because I wasn't sure if there was some crossover between it and using the MPT driver for a SAS card. Thanks for clearing that up though...obviously we don't need it for the LSI3801E we are using. Ah, ok Can you explain the "regen the boot archive" request in more detail though? This has me wondering if there were additional steps we needed to take when testing out other /etc/system tweaks, such as the vdev queue limitation. I want to make sure we eliminate sources of the problem, if possible, as the chain of possible blame is still quite long right now. The reboot command should have automatically run bootadm update-archive for you, I have this habit of running it by hand whenever I change a driver or /etc/system to make sure that I have an up to date boot archive from that point in time onwards. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> > I thought you had just set > > set xpv_psm:xen_support_msi = -1 > > which is different, because that sets the > xen_support_msi variable > which lives inside the xpv_psm module. > > Setting mptsas:* will have no effect on your system > if you do not > have an mptsas card installed. The mptsas cards are > not generally > available yet (they're 2nd generation), so I would be > surprised if > you had one. No...I had set the other two variables after Mark contacted me offline to do some testing mainly to verify the problem was, indeed, not xVM specific. I had added the mptsas line as well, as per his recommendations, because I wasn't sure if there was some crossover between it and using the MPT driver for a SAS card. Thanks for clearing that up though...obviously we don't need it for the LSI3801E we are using. Can you explain the "regen the boot archive" request in more detail though? This has me wondering if there were additional steps we needed to take when testing out other /etc/system tweaks, such as the vdev queue limitation. I want to make sure we eliminate sources of the problem, if possible, as the chain of possible blame is still quite long right now. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Adam Cheal wrote: Hi Adam, thanks for this info. I've talked with my colleagues in Beijing (since I'm in Beijing this week) and we'd like you to try disabling MSI/MSI-X for your mpt instances. In /etc/system, add set mpt:mpt_enable_msi = 0 then regen your boot archive and reboot. I had already done this at Mark Johnson's request, though I had just added: set mpt:mpt_enable_msi=0 set mptsas:mptsas_enable_msi=0 ...to the /etc/system file and did a full reboot. I didn't know I had to regen the boot archive manually for those new settings to take effect: how/why would I do this? The fact the IO rate changed for me during the test indicated that the new settings had "taken". Long story short, I still had the problems after making this change though they took longer to appear. I thought you had just set set xpv_psm:xen_support_msi = -1 which is different, because that sets the xen_support_msi variable which lives inside the xpv_psm module. Setting mptsas:* will have no effect on your system if you do not have an mptsas card installed. The mptsas cards are not generally available yet (they're 2nd generation), so I would be surprised if you had one. Longer story: After making the change, rebooting and starting a scrub on the pool I watched iostat for hints of trouble (i.e. error column changes). The IO rate to the disk was definately slower after this change, with individual disks never getting more than 50% busy and 2 active commands. About three hours later, the read errors/bus resets started to appear. I assume the longer delay before errors was just because the reduced IO was putting less of a strain on the driver/hardware. Let me know if you want me to refine this test or any other diagnostics that would help you out. I think that's sufficient to go on for the moment, thankyou. cheers, James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> Hi Adam, > thanks for this info. I've talked with my colleagues > in Beijing (since > I'm in Beijing this week) and we'd like you to try > disabling MSI/MSI-X > for your mpt instances. In /etc/system, add > > set mpt:mpt_enable_msi = 0 > > then regen your boot archive and reboot. > I had already done this at Mark Johnson's request, though I had just added: set mpt:mpt_enable_msi=0 set mptsas:mptsas_enable_msi=0 ...to the /etc/system file and did a full reboot. I didn't know I had to regen the boot archive manually for those new settings to take effect: how/why would I do this? The fact the IO rate changed for me during the test indicated that the new settings had "taken". Long story short, I still had the problems after making this change though they took longer to appear. Longer story: After making the change, rebooting and starting a scrub on the pool I watched iostat for hints of trouble (i.e. error column changes). The IO rate to the disk was definately slower after this change, with individual disks never getting more than 50% busy and 2 active commands. About three hours later, the read errors/bus resets started to appear. I assume the longer delay before errors was just because the reduced IO was putting less of a strain on the driver/hardware. Let me know if you want me to refine this test or any other diagnostics that would help you out. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Adam Cheal wrote: Thankyou for all who've procvided data about this. I've updated the bugs mentioned earlier and I believe we can now make progress on diagnosis. The new synopsis (should show up on b.o.o tomorrow) is as follows: 6894775 mpt's msi support is suboptimal with xVM FYI, as the original submitter of 6894775 I can tell you that we don't use XVM so the title change is misleading; we are using a simple physical server with a LSI3801E attached to 2 JBODs. My case has really gone nowhere and the original engineer in charge of it only made a few suggestions. Namely: Hi Adam, thanks for this info. I've talked with my colleagues in Beijing (since I'm in Beijing this week) and we'd like you to try disabling MSI/MSI-X for your mpt instances. In /etc/system, add set mpt:mpt_enable_msi = 0 then regen your boot archive and reboot. I've added this to the public comments field of the CR, and removed the reference to xVM from the synopsis - hopefully the mail gateway will send your copy reasonably soon :-) Best regards, James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Thankyou for all who've procvided data about this. I've updated the bugs mentioned earlier and I believe we can now make progress on diagnosis. The new synopsis (should show up on b.o.o tomorrow) is as follows: 6894775 mpt's msi support is suboptimal with xVM James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> Travis Tabbal wrote: > > I have a possible workaround. Mark Johnson > has > > been emailing me today about this issue and he > proposed the > > following: > > > >> You can try adding the following to /etc/system, > then rebooting... > >> set xpv_psm:xen_support_msi = -1 > > I am also running XVM, and after modifying > /etc/system and rebooting, my > zpool scrub test is runing along merrily with no > hangs so far, where > usually I would expect to see several by now. > > Can the other folks who have seen this please test > and report back? I'd > hate to think we solved it only to discover there > were overlapping bugs. > > Fingers crossed, and many thanks to those who have > worked to track this > down! Nice to see we have one confirmed report that things are working. Hopefully we get a few more! Even if it's just a workaround until a real fix makes it in, it gets us running. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> > On Nov 23, 2009, at 7:28 PM, Travis Tabbal wrote: > > > I have a possible workaround. Mark Johnson > > > has been emailing me today about this issue and he > proposed the > > following: > > > >> You can try adding the following to /etc/system, > then rebooting... > >> set xpv_psm:xen_support_msi = -1 > > would this change affect systems not using XVM? we > are just using > these as backup storage. Probably not. Are you seeing the issue without XVM installed? We had one other user report that the issue went away when they removed XVM, so I had thought it wouldn't affect other users. If you are getting the same issue without XVM, there may be overlapping bugs in play. Someone at Sun might be able to tell you how to disable MSI on the controller. Someone told me how to do it for the NVidia SATA controller when there was a bug in that driver. I would think there is a way to do it for the MPT driver. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
On Nov 23, 2009, at 7:28 PM, Travis Tabbal wrote: I have a possible workaround. Mark Johnson has been emailing me today about this issue and he proposed the following: You can try adding the following to /etc/system, then rebooting... set xpv_psm:xen_support_msi = -1 would this change affect systems not using XVM? we are just using these as backup storage. Thanks! -Jeremy ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Travis Tabbal wrote: I have a possible workaround. Mark Johnson has been emailing me today about this issue and he proposed the following: You can try adding the following to /etc/system, then rebooting... set xpv_psm:xen_support_msi = -1 I am also running XVM, and after modifying /etc/system and rebooting, my zpool scrub test is runing along merrily with no hangs so far, where usually I would expect to see several by now. Can the other folks who have seen this please test and report back? I'd hate to think we solved it only to discover there were overlapping bugs. Fingers crossed, and many thanks to those who have worked to track this down! -- Carson ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
I have a possible workaround. Mark Johnson has been emailing me today about this issue and he proposed the following: > You can try adding the following to /etc/system, then rebooting... > set xpv_psm:xen_support_msi = -1 I have been able to format a ZVOL container from a VM 3 times while other activity is going on the system and it's working. I think performance is down a bit, but it's still acceptable. More importantly, it does so without killing the server. I would get the stall every time I would try this test before. So at least 1 case seems to be helped by doing this. I'll watch the server over the next few days to see if it stays improved. He mentioned that there is a fix being worked on for MSI handling in XVM that might make it into b129 that could fix this problem. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Travis Tabbal wrote: I will give you all of this information on monday. This is great news :) Indeed. I will also be posting this information when I get to the server tonight. Perhaps it will help. I don't think I want to try using that old driver though, it seems too risky for my taste. Definitely not recommended. Is there a command to get the disk firmware rev from OpenSolaris while booted up? I know of some boot CDs that can get to it, but I'm unsure about accessing it while the server is running. Of course. Use prtconf -v and look for the disk node hardware properties. Example: Hardware properties: name='devid' type=string items=1 value='id1,s...@sata_samsung_hm320ji___s19fj10pc45360' name='inquiry-device-type' type=int items=1 value= name='inquiry-revision-id' type=string items=1 value='2SS00_01' name='inquiry-product-id' type=string items=1 value='HM320JI' name='inquiry-vendor-id' type=string items=1 value='SAMSUNG' which tells you that this is direct-access device (device type 0), has a non-standard revision id field (2SS00_01) of which we take the last 4 bytes as the actual revision field, and the vendor and product ids. The devid information helps here too. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
> I will give you all of this information on monday. > This is great news :) Indeed. I will also be posting this information when I get to the server tonight. Perhaps it will help. I don't think I want to try using that old driver though, it seems too risky for my taste. Is there a command to get the disk firmware rev from OpenSolaris while booted up? I know of some boot CDs that can get to it, but I'm unsure about accessing it while the server is running. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
On Nov 21, 2009, at 1:08 AM, James C. McPherson wrote: We currently have two bugs open on what I believe to be the same issue, namely 6894775 mpt driver timeouts and bus resets under load 6900767 Server hang with LSI 1068E based SAS controller under load If you and everybody else who is seeing this problem could provide details about your configuration (output from cfgadm -lva, raidctl -l, prtconf -v, what your zpool configs are, and the firmware rev of each disk in each zpool) that would help us sort through and find any commonalities and hopefully a fix. I will give you all of this information on monday. This is great news :) -Jeremy ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Workaround for mpt timeouts in snv_127
Carson Gaspar wrote: For all of those suffering from mpt timeouts in snv_127, I decided to give the ancient itmpt driver a whirl. It works fine, and in my brief testing a zfs scrub that would generate about 1 timeout every 2 minutes or so now runs with no problems. The downside is that lsiutil and raidctl both fail to work :-( ... and that you don't have FMA support, or MPxIO. I also tried the mpt driver from Solaris 10 x86 patch 143129-01, but that fails to load with undefined symbols, as do the mpt drivers from snv_111b and anv_118 (unless I got something wrong with my bootadm update-archive invocation...) The version of mpt(7d) that's in Solaris 10, Solaris 10 Updates and Solaris 10 patches will NOT work on OpenSolaris or SXCE. The codebase has diverged between Solaris 10 and now. Likewise, the mpt(7d) driver has diverged from the version that's in snv_111b / OpenSolaris 2009.06, and even in the 10 builds between 118 and 127. Frankly, I'm surprised you didn't panic your system, especially with the Solaris 10 version. We currently have two bugs open on what I believe to be the same issue, namely 6894775 mpt driver timeouts and bus resets under load 6900767 Server hang with LSI 1068E based SAS controller under load If you and everybody else who is seeing this problem could provide details about your configuration (output from cfgadm -lva, raidctl -l, prtconf -v, what your zpool configs are, and the firmware rev of each disk in each zpool) that would help us sort through and find any commonalities and hopefully a fix. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Workaround for mpt timeouts in snv_127
For all of those suffering from mpt timeouts in snv_127, I decided to give the ancient itmpt driver a whirl. It works fine, and in my brief testing a zfs scrub that would generate about 1 timeout every 2 minutes or so now runs with no problems. The downside is that lsiutil and raidctl both fail to work :-( I also tried the mpt driver from Solaris 10 x86 patch 143129-01, but that fails to load with undefined symbols, as do the mpt drivers from snv_111b and anv_118 (unless I got something wrong with my bootadm update-archive invocation...) For reference, I'm running firmware 1.29.00.00 (the latest), and the old errors were of the form: Nov 19 03:34:43 gandalf.taltos.org scsi: [ID 107833 kern.warning] WARNING: /p...@0,0/pci8086,2...@1c/pci1000,3...@0 (mpt0): Nov 19 03:34:43 gandalf.taltos.org Disconnected command timeout for Target 13 With the target number changing. When the hangs would happen, iostat would show "stuck" I/Os on all 8 SATA disks attached to my controller (which are all in the same raidz2 pool) -- Carson ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
