Re: [OmniOS-discuss] iSCSI target hang, no way to restart but server reboot

2015-03-31 Thread Narayan Desai
We were primarily using the machines for serving iscsi to VMs, and we'd see bad cascading failures (iscsi lun timeouts would cause the watchdog to kick in on the linux hosts, resetting the initiator, meanwhile the VM would decide that the virtio devices in the VM were dead, requiring a client

Re: [OmniOS-discuss] iSCSI target hang, no way to restart but server reboot

2015-03-31 Thread Matej Zerovnik
On 27. 03. 2015 16:13, Narayan Desai wrote: Having been on the receiving end of similar advice, it is a frustrating situation to be in, since you have (and will likely continue to have) the hardware in production, without much option for replacement. When we had systems like this, we had a

Re: [OmniOS-discuss] iSCSI target hang, no way to restart but server reboot

2015-03-27 Thread Matej Zerovnik
It just happened about 2 hours ago... The whole system did not crash, but 2 clients lost the connection. This is what I see in logs: Mar 27 13:55:51 storage.host.org scsi: [ID 107833 kern.notice] /pci@0,0/pci8086,3c02@1/pci1000,3040@0 (mpt_sas0): Mar 27 13:55:51 storage.host.org

Re: [OmniOS-discuss] iSCSI target hang, no way to restart but server reboot

2015-03-27 Thread Dan McDonald
On Mar 27, 2015, at 10:54 AM, Matej Zerovnik ma...@zunaj.si wrote: I read in the archives, that this errors happens when you have SATA drives on a SAS expander and one of the drives misbehaves: A command did not complete and the mpt driver reset the target. If that target is an expander,

Re: [OmniOS-discuss] iSCSI target hang, no way to restart but server reboot

2015-03-27 Thread Matej Zerovnik
On 27. 03. 2015 15:56, Dan McDonald wrote: iostat -Ei | grep Error reports that one device has 6 hard errors and 6 device not ready errors, but that is a local drive, attached to a different controller (LSI Megaraid). LSI Megaraid, ESPECIALLY with 006, is not going to be as good as either

Re: [OmniOS-discuss] iSCSI target hang, no way to restart but server reboot

2015-03-27 Thread Narayan Desai
Having been on the receiving end of similar advice, it is a frustrating situation to be in, since you have (and will likely continue to have) the hardware in production, without much option for replacement. When we had systems like this, we had a lot of success being aggressive in swapping out

Re: [OmniOS-discuss] iSCSI target hang, no way to restart but server reboot

2015-03-27 Thread Dave Pooser
Having been on the receiving end of similar advice, it is a frustrating situation to be in, since you have (and will likely continue to have) the hardware in production, without much option for replacement. When we had systems like this, we had a lot of success being aggressive in swapping out