Re: 'Device not ready' issue on mpt2sas since 3.1.10
On 08/16/2012 04:24 PM, Matthias Prager wrote: > > Not yet, but it is in James scsi misc tree and last I heard was > scheduled for inclusion in the 3.6 kernel. Close enough. :-) I didn't track the changes on the SCSI tree and I just wanted to make sure that it didn't slip through the cracks. Thanks to all involved for all of the help and a speedy fix! -- Robert -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 'Device not ready' issue on mpt2sas since 3.1.10
On 07/25/2012 03:55 PM, James Bottomley wrote: > > Well, reading it, so do I. Unfortunately, we get to deal with the world > as it is rather than as we would wish it to be. We likely have this > problem with a lot of USB SATLs as well ... Has this patch made it into the main git trees yet? I haven't seen anything about it in nearly a month, but I've been using the James' patch since he posted it and the sleep/wakeup behavior seems improved/correct. -- Robert -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 'Device not ready' issue on mpt2sas since 3.1.10
On 07/25/2012 06:35 PM, tomm wrote: > > If this is a driver or firmware bug, then why would commit > 85ef06d1d252f6a2e73b678591ab71caad4667bb > cause this to happen? What is the interaction between this issue > and this commit which just flushes events? That's confusing to me as well. Tejun's patch seems very unrelated to anything related to power-state on non-removable disks. > Also this issue does not happen with mvsas, only with mpt2sas. Now _that_ is a useful data point. Is that with SATA disks attached? Why is it limited (so far) to just the mpt2sas controller? -- Robert -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 'Device not ready' issue on mpt2sas since 3.1.10
On 07/25/2012 07:56 PM, Matthias Prager wrote: > > I don't yet understand all the code but I'm following your discussion > with Tejun: I've set up a minimal vm running gentoo with a mpt2sas > driven controller in passthrough mode. I've applied your proposed patch > against the vanilla 3.5.0 kernel (which includes Tejun's commit), and > I'm happy to report the problem does seem to get fixed by it. I can confirm this on my hardware as well with both 3.4.4 and 3.5.0. Without James' patch the kernels will immediately drop the I/O and with the patch both kernels will wake the SATA disks and then complete the I/O successfully. -- Robert -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 'Device not ready' issue on mpt2sas since 3.1.10
On 07/09/2012 09:51 PM, Robert Trace wrote: > > Huh.. I just retested this and I'm seeing really random behavior. Ok, with a refined test I've been able to reliably reproduce this and I bisected it back to commit 85ef06d1d252f6a2e73b678591ab71caad4667bb in Linus' tree (introduced between 3.0 and 3.1): commit 85ef06d1d252f6a2e73b678591ab71caad4667bb Author: Tejun Heo Date: Fri Jul 1 16:17:47 2011 +0200 block: flush MEDIA_CHANGE from drivers on close(2) Prior to the above commit, sleeping disks will spin up as a result of I/O sent to them. With the above commit, they don't spin up and immediately return an I/O failure. That's all the further I've gotten so far. I'll be happy to test any patches or suggestions. -- Rob -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 'Device not ready' issue on mpt2sas since 3.1.10
On 07/09/2012 08:21 PM, Matthias Prager wrote: > > I haven't checked the scsi logging side, but about the only commands > that wake up the disks are 'smartctl -a /dev/sda' and 'sg_start' > (smartcl maybe issuing a START UNIT command on it's own). smartctl -a does appear to wake the disks. The scsi log shows an IDENTIFY and then several ATA passthrough commands (one of which takes ~10 seconds to complete). So, I don't see an explicit START UNIT, but one of those ATA commands which I didn't decode could certainly trigger the wakeup. -- Rob -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 'Device not ready' issue on mpt2sas since 3.1.10
[removed linux-raid since the md layer seems unrelated]
On 07/09/2012 08:12 PM, Matthias Prager wrote:
>>
>> I've reproduced this behavior on the raw disks themselves, no MD layer
>> involved (although the freak-out by my MD layer is what alerted me to
>> this issue too... Having your entire array punted the first time you
>> access it is a little scary :-). I'm also on raw hardware and I've seen
>> this behavior on kernels 3.0.33 through 3.4.4.
> This is interesting - are you sure about 3.0.33? I'm running this kernel
> atm for it gives me no trouble (as opposed to >=3.1.10). The SATA disks
> are spun up when I access data on them.
Huh.. I just retested this and I'm seeing really random behavior.
I tried 3.0.33 a few days ago after I saw your initial e-mail to this
list. At that time, the one disk I tried didn't wake up when I sent I/O
to it.
My first retest (just now), on 3.0.33 with four disks, showed the
behavior you initially reported. Two of the disks woke up from the I/O,
but not all of them.
Repeating the test without rebooting made two disks wake up, but only
one of the same disks from the first test. The second disk that woke up
was different.
After rebooting and running the test again, none of the disks woke up.
Rebooting again and all of the disks are waking up.
(FYI, here's the test I ran:
1. hdparm -y /dev/sd[lmjk]
2. hdparm -C /dev/sd[lmjk] (to verify disks in standby)
3. for i in l m j k; do sg_turs -v /dev/sd${i}; done
(All disks reported "Not Ready")
4. echo 3 > /proc/sys/vm/drop_caches
5. for i in l m j k; do dd if=/dev/sd${i} of=/dev/null bs=512 count=1
skip=; done
I've been manually changing the skip= because I've seen the dd
command complete successfully without the disk waking up. I think this
is because the disk is satisfying the read from its own cache. Changing
where on the disk I'm reading should thwart this.
)
I'm confused. I'll try more recent kernels again and see if the
behavior becomes predictable.
-- Rob
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 'Device not ready' issue on mpt2sas since 3.1.10
On 07/09/2012 04:45 PM, Darrick J. Wong wrote: > > I suspect that /sys/devices//manage_start_stop = 0 > for the SATA devices hanging off the SAS controller. Yep, looks like you're right. For my system: # cat /sys/block/sd?/device/scsi_disk/*/manage_start_stop 1 1 1 1 1 0 0 0 0 0 0 0 0 Those first 5 disks are SATA disks on SATA controllers. The last 8 disks are SATA disks on the SAS controller. > Setting that sysfs > attribute to 1 is supposed to enable the SCSI layer to send TUR when it sees > "LU not ready", as well as spin down the drives at suspend/poweroff time. Setting it to 1 doesn't seem to have made any difference, however. # cat /sys/block/sdm/device/scsi_disk/14\:0\:7\:0/manage_start_stop 0 # echo 1 > /sys/block/sdm/device/scsi_disk/14\:0\:7\:/manage_start_stop # cat /sys/block/sdm/device/scsi_disk/14\:0\:7\:0/manage_start_stop 1 # hdparm -y /dev/sdm /dev/sdm: issuing standby command # hdparm -C /dev/sdm /dev/sdm: drive state is: standby # dd if=/dev/sdm of=/dev/null bs=512 count=1 dd: reading `/dev/sdm': Input/output error 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00117802 s, 0.0 kB/s ... and on the scsi logging side, I see the read(10) to the disk which immediately returns "Not Ready" and the I/O failure bubbles up the chain. And afterwards, the disk is still asleep. # hdparm -C /dev/sdm /dev/sdm: drive state is: standby Also, TURs don't appear to actually wake the disk up (should they?). The only thing I've found that'll wake the disk up is an explicit START UNIT command. -- Rob -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 'Device not ready' issue on mpt2sas since 3.1.10
> I did some further research regarding my problem. > It appears to me the fault does not lie with the mpt2sas driver (not > that I can definitely exclude it), but with the md implementation. I'm actually discovering some of the same issues (LSI 9211-8i w/ SATA disks), but I've come to a slightly different conclusion. I noticed that when my SATA disks are on a SATA controller and they spin down (or are spun down via hdparm -y), then they response to TUR (TEST UNIT READY) commands with an OK. Any I/O sent to these disks simply wait while the disks spin up and then complete as usual. However, my SATA disks on the SAS controller respond to TUR with the sense error "Not Ready/Initializing command required". Any I/O sent to these disks immediately fails. You saw this in your logging: > [ 604.838640] sd 2:0:0:0: [sda] Device not ready > [ 604.838645] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK > driverbyte=DRIVER_SENSE > [ 604.838655] sd 2:0:0:0: [sda] Sense Key : Not Ready [current] > [ 604.838663] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready, > initializing command required > [ 604.838668] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00 > 20 00 > [ 604.838680] end_request: I/O error, dev sda, sector 2048 > [ 604.838688] Buffer I/O error on device md127, logical block 0 > [ 604.838695] Buffer I/O error on device md127, logical block 1 > [ 604.838699] Buffer I/O error on device md127, logical block 2 > [ 604.838702] Buffer I/O error on device md127, logical block 3 Sending an explicit START UNIT command to these sleeping disks will wake them up and then they behave normally. (BTW, you can issue TURs and START UNITs via the sg_turs and sg_start commands). I've reproduced this behavior on the raw disks themselves, no MD layer involved (although the freak-out by my MD layer is what alerted me to this issue too... Having your entire array punted the first time you access it is a little scary :-). I'm also on raw hardware and I've seen this behavior on kernels 3.0.33 through 3.4.4. So, SATA disks respond differently depending on the controller they're on. I don't know if this is a SCSI thing, a SAS thing or a firmware/driver thing for the 9211. Now, whether or not the MD layer should be assembling arrays from "failed" disks is, I think, a separate issue. -- Rob -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
