Re: [PATCH] scsi-mq: fix hw queue hang caused by timeout
On 09/19/2014 08:18 AM, Ming Lei wrote: > On Fri, Sep 19, 2014 at 9:07 PM, Ming Lei wrote: >> On Fri, Sep 19, 2014 at 1:03 AM, Jens Axboe wrote: >>> On 2014-09-18 10:35, Christoph Hellwig wrote: On Thu, Sep 18, 2014 at 11:59:10PM +0800, Ming Lei wrote: > > If there are two requests or more timed out, the dispatch queue > is put into stopped state and never be recoverd, and there > is no such problem in non-mq mode. > > This patch trys to recover the stopped queue when the queue > becomes unbusy, then the following retries can move on. > > Basically this patch maintains same behavior for this situation > with non-mq mode. This looks somewhat similar to the issues that Doug reported, and I remember when he was last running into boot problems it was timeout related, too. As far as the implementation is concerned I think the correct fix is to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list. >>> >>> >>> Since that's the kick part of the requeue, auto-starting the queue for that >>> makes a lot of sense. I say that's the way we go. >> >> Yeah, that looks better. >> >> But it doesn't work after the simple change, and I need to >> investigate further. > > It is because of the timer miss, now it starts to work. Excellent. I think most new issues should be fixed in for-linus for inclusion in this round. It's much bigger than I hoped for this late in the cycle, but lots of us have run a lot of testing, so that's not a huge worry. -- Jens Axboe -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi-mq: fix hw queue hang caused by timeout
On Fri, Sep 19, 2014 at 9:07 PM, Ming Lei wrote: > On Fri, Sep 19, 2014 at 1:03 AM, Jens Axboe wrote: >> On 2014-09-18 10:35, Christoph Hellwig wrote: >>> >>> On Thu, Sep 18, 2014 at 11:59:10PM +0800, Ming Lei wrote: If there are two requests or more timed out, the dispatch queue is put into stopped state and never be recoverd, and there is no such problem in non-mq mode. This patch trys to recover the stopped queue when the queue becomes unbusy, then the following retries can move on. Basically this patch maintains same behavior for this situation with non-mq mode. >>> >>> >>> This looks somewhat similar to the issues that Doug reported, and I >>> remember >>> when he was last running into boot problems it was timeout related, too. >>> >>> As far as the implementation is concerned I think the correct fix is >>> to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list. >> >> >> Since that's the kick part of the requeue, auto-starting the queue for that >> makes a lot of sense. I say that's the way we go. > > Yeah, that looks better. > > But it doesn't work after the simple change, and I need to > investigate further. It is because of the timer miss, now it starts to work. Thanks, -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi-mq: fix hw queue hang caused by timeout
On Fri, Sep 19, 2014 at 1:03 AM, Jens Axboe wrote: > On 2014-09-18 10:35, Christoph Hellwig wrote: >> >> On Thu, Sep 18, 2014 at 11:59:10PM +0800, Ming Lei wrote: >>> >>> If there are two requests or more timed out, the dispatch queue >>> is put into stopped state and never be recoverd, and there >>> is no such problem in non-mq mode. >>> >>> This patch trys to recover the stopped queue when the queue >>> becomes unbusy, then the following retries can move on. >>> >>> Basically this patch maintains same behavior for this situation >>> with non-mq mode. >> >> >> This looks somewhat similar to the issues that Doug reported, and I >> remember >> when he was last running into boot problems it was timeout related, too. >> >> As far as the implementation is concerned I think the correct fix is >> to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list. > > > Since that's the kick part of the requeue, auto-starting the queue for that > makes a lot of sense. I say that's the way we go. Yeah, that looks better. But it doesn't work after the simple change, and I need to investigate further. Thanks, -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi-mq: fix hw queue hang caused by timeout
On 2014-09-18 10:35, Christoph Hellwig wrote: On Thu, Sep 18, 2014 at 11:59:10PM +0800, Ming Lei wrote: If there are two requests or more timed out, the dispatch queue is put into stopped state and never be recoverd, and there is no such problem in non-mq mode. This patch trys to recover the stopped queue when the queue becomes unbusy, then the following retries can move on. Basically this patch maintains same behavior for this situation with non-mq mode. This looks somewhat similar to the issues that Doug reported, and I remember when he was last running into boot problems it was timeout related, too. As far as the implementation is concerned I think the correct fix is to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list. Since that's the kick part of the requeue, auto-starting the queue for that makes a lot of sense. I say that's the way we go. -- Jens Axboe -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi-mq: fix hw queue hang caused by timeout
On Thu, Sep 18, 2014 at 11:59:10PM +0800, Ming Lei wrote: > If there are two requests or more timed out, the dispatch queue > is put into stopped state and never be recoverd, and there > is no such problem in non-mq mode. > > This patch trys to recover the stopped queue when the queue > becomes unbusy, then the following retries can move on. > > Basically this patch maintains same behavior for this situation > with non-mq mode. This looks somewhat similar to the issues that Doug reported, and I remember when he was last running into boot problems it was timeout related, too. As far as the implementation is concerned I think the correct fix is to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list. -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] scsi-mq: fix hw queue hang caused by timeout
If there are two requests or more timed out, the dispatch queue
is put into stopped state and never be recoverd, and there
is no such problem in non-mq mode.
This patch trys to recover the stopped queue when the queue
becomes unbusy, then the following retries can move on.
Basically this patch maintains same behavior for this situation
with non-mq mode.
Signed-off-by: Ming Lei
---
drivers/scsi/scsi_lib.c | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 3b92c39..dfbc028 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -142,6 +142,8 @@ static void __scsi_queue_insert(struct scsi_cmnd *cmd, int
reason, int unbusy)
struct scsi_device *device = cmd->device;
struct request_queue *q = device->request_queue;
unsigned long flags;
+ bool restart = false;
+ bool blocked = !!atomic_read(&device->device_blocked);
SCSI_LOG_MLQUEUE(1, scmd_printk(KERN_INFO, cmd,
"Inserting command %p into mlqueue\n", cmd));
@@ -152,9 +154,14 @@ static void __scsi_queue_insert(struct scsi_cmnd *cmd, int
reason, int unbusy)
* Decrement the counters, since these commands are no longer
* active on the host/device.
*/
- if (unbusy)
+ if (unbusy) {
scsi_device_unbusy(device);
+ /* need to restart hw queue if it was stopped */
+ if (!atomic_read(&device->device_busy) && blocked)
+ restart = true;
+ }
+
/*
* Requeue this command. It will go before all other commands
* that are already in the queue. Schedule requeue work under
@@ -164,6 +171,8 @@ static void __scsi_queue_insert(struct scsi_cmnd *cmd, int
reason, int unbusy)
cmd->result = 0;
if (q->mq_ops) {
scsi_mq_requeue_cmd(cmd);
+ if (restart)
+ blk_mq_start_stopped_hw_queues(q, true);
return;
}
spin_lock_irqsave(q->queue_lock, flags);
--
1.7.9.5
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
