Re: [PATCH 1/6] megaraid_sas: Do not wait forever
On 01/24/2014 08:46 AM, Desai, Kashyap wrote:
Hannes:
We have already worked on wait_event usage in megasas_issue_blocked_cmd.
That code will be posted by LSI once we received test result from
LSI Q/A team.
If you see the current OCR code in Linux Driver we do re-send the IOCTL
command.
MR product does not want IOCTL timeout due to some reason. That is why even if
FW faulted, Driver will do OCR and re-send all existing
Management commands
(IOCTL comes under management commands).
Just for info. (see below snippet in OCR code)
/* Re-fire management commands */
for (j = 0 ; j instance-max_fw_cmds; j++) {
cmd_fusion = fusion-cmd_list[j];
if (cmd_fusion-sync_cmd_idx !=
(u32)ULONG_MAX) {
cmd_mfi =
instance-cmd_list[cmd_fusion-sync_cmd_idx];
if (cmd_mfi-frame-dcmd.opcode ==
MR_DCMD_LD_MAP_GET_INFO) {
megasas_return_cmd(instance,
cmd_mfi);
megasas_return_cmd_fusion(instance, cmd_fusion);
Current MR Driver is not designed to add timeout for DCMD and IOCTL path.
[ I added timeout only for limited DCMDs, which are harmless to
continue after timeout ]
As of now, you can skip this patch and we will be submitting patch to fix
similar issue.
But note, we cannot add complete wait_event_timeout due to day-1 design,
but will
try to cover wait_event_timout for some valid cases.
Ouch.
The reason I sent this patch is that I've got an Intel box here,
which blocks megaraid_sas initialisation when the IOMMU is turned on:
[ 21.867264] megasas: io_request_frames 880800f5
[ 21.867363] megasas: init frame fff57000
[ 22.223234] megasas: frame status 00
[ 22.223235] megasas: IOC Init cmd success
[ 22.223282] megasas: ld map 88080b60
[ 22.223289] megasas: issue dcmd 05 opcode 300e101
[ 22.244184] dmar: DRHD: handling fault status reg 2
[ 22.244186] dmar: DMAR:[DMA Read] Request device [06:00.0] fault
addr 698
[ 22.244186] DMAR:[fault reason 06] PTE Read access is not set
[ 22.247223] megasas: frame status 00
[ 22.247231] megasas: issue dcmd 05 opcode 300e101
[ 22.247231] megasas: INIT adapter done
[ 22.247237] megasas: pd list 88080cfd size 8192
[ 22.247237] megasas: issue dcmd 05 opcode 2010100
[ 22.253516] dmar: DRHD: handling fault status reg 102
[ 22.253518] dmar: DMAR:[DMA Write] Request device [06:00.0] fault
addr e3f
[ 22.253518] DMAR:[fault reason 05] PTE Write access is not set
[ 22.253521] dmar: DMAR:[DMA Write] Request device [06:00.0] fault
addr e3f
[ 22.253521] DMAR:[fault reason 05] PTE Write access is not set
[ 22.253523] dmar: DMAR:[DMA Write] Request device [06:00.0] fault
addr e3f
[ Some more DMAR messages snipped ]
[ 22.273199] dmar: DRHD: handling fault status reg 2
[ 22.273201] dmar: DMAR:[DMA Read] Request device [06:00.0] fault
addr 6cef000
[ 22.273201] DMAR:[fault reason 06] PTE Read access is not set
[ .. ]
[ 94.222456] megasas: frame status ff
[ 94.240946] megasas: failed to get PD list
(I've inserted some debugging messages :-)
This is really weird. The 'write' faults do correspond with the
number of (megaraid) commands, reserved at the initial step.
(This is a 'Fury' card, btw).
What is more puzzling is that the INIT command and the initial
LD List command goes through, but the PD List command gets blocked.
Incidentally, this is not consistent; occasionally even the LD List
command gets blocked, and the DMAR messages occur earlier.
Anyway. Point is, if we cannot timout these initial commands
the megaraid_sas driver will be stuck during initialisation (as the
loop _never_ terminates).
Which in turn means that the modprobe command hangs indefinitely,
and you cannot even unload the module.
The only way to recover here is a reboot.
Nasty.
Hence the patch for the timeout; when this triggers the HBA is
pretty much hosed anyway, so the state of the firmware is pretty
much irrelevant here. But at least you can continue to boot.
(And OCR doesn't work at this point, neither. But that's a different
story).
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries Storage
h...@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line unsubscribe linux-scsi in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 1/6] megaraid_sas: Do not wait forever
-Original Message-
From: Hannes Reinecke [mailto:h...@suse.de]
Sent: Friday, January 24, 2014 1:54 PM
To: Desai, Kashyap; James Bottomley
Cc: linux-scsi@vger.kernel.org; Adam Radford; Saxena, Sumit
Subject: Re: [PATCH 1/6] megaraid_sas: Do not wait forever
On 01/24/2014 08:46 AM, Desai, Kashyap wrote:
Hannes:
We have already worked on wait_event usage in
megasas_issue_blocked_cmd.
That code will be posted by LSI once we received test result from
LSI Q/A team.
If you see the current OCR code in Linux Driver we do re-send the IOCTL
command.
MR product does not want IOCTL timeout due to some reason. That is why
even if FW faulted, Driver will do OCR and re-send all existing
Management commands
(IOCTL comes under management commands).
Just for info. (see below snippet in OCR code)
/* Re-fire management commands */
for (j = 0 ; j instance-max_fw_cmds; j++) {
cmd_fusion = fusion-cmd_list[j];
if (cmd_fusion-sync_cmd_idx !=
(u32)ULONG_MAX) {
cmd_mfi =
instance-cmd_list[cmd_fusion-
sync_cmd_idx];
if (cmd_mfi-frame-dcmd.opcode ==
MR_DCMD_LD_MAP_GET_INFO) {
megasas_return_cmd(instance, cmd_mfi);
megasas_return_cmd_fusion(instance, cmd_fusion);
Current MR Driver is not designed to add timeout for DCMD and IOCTL
path.
[ I added timeout only for limited DCMDs, which are harmless to
continue after timeout ]
As of now, you can skip this patch and we will be submitting patch to fix
similar issue.
But note, we cannot add complete wait_event_timeout due to day-1
design, but will try to cover wait_event_timout for some valid cases.
Ouch.
The reason I sent this patch is that I've got an Intel box here, which blocks
megaraid_sas initialisation when the IOMMU is turned on:
[ 21.867264] megasas: io_request_frames 880800f5
[ 21.867363] megasas: init frame fff57000
[ 22.223234] megasas: frame status 00
[ 22.223235] megasas: IOC Init cmd success
[ 22.223282] megasas: ld map 88080b60
[ 22.223289] megasas: issue dcmd 05 opcode 300e101
[ 22.244184] dmar: DRHD: handling fault status reg 2
[ 22.244186] dmar: DMAR:[DMA Read] Request device [06:00.0] fault
addr 698
[ 22.244186] DMAR:[fault reason 06] PTE Read access is not set
[ 22.247223] megasas: frame status 00
[ 22.247231] megasas: issue dcmd 05 opcode 300e101
[ 22.247231] megasas: INIT adapter done
[ 22.247237] megasas: pd list 88080cfd size 8192
[ 22.247237] megasas: issue dcmd 05 opcode 2010100
[ 22.253516] dmar: DRHD: handling fault status reg 102
[ 22.253518] dmar: DMAR:[DMA Write] Request device [06:00.0] fault
addr e3f
[ 22.253518] DMAR:[fault reason 05] PTE Write access is not set
[ 22.253521] dmar: DMAR:[DMA Write] Request device [06:00.0] fault
addr e3f
[ 22.253521] DMAR:[fault reason 05] PTE Write access is not set
[ 22.253523] dmar: DMAR:[DMA Write] Request device [06:00.0] fault
addr e3f
[ Some more DMAR messages snipped ]
[ 22.273199] dmar: DRHD: handling fault status reg 2
[ 22.273201] dmar: DMAR:[DMA Read] Request device [06:00.0] fault
addr 6cef000
[ 22.273201] DMAR:[fault reason 06] PTE Read access is not set
[ .. ]
[ 94.222456] megasas: frame status ff
[ 94.240946] megasas: failed to get PD list
(I've inserted some debugging messages :-)
This is really weird. The 'write' faults do correspond with the number of
(megaraid) commands, reserved at the initial step.
(This is a 'Fury' card, btw).
Fury card has iMR FW and we have seen issue with iMR FW if IOMMU is ON, but not
like driver load failure.
Is your OS driver behind Fury ? What is a Raid type used on your setup ?
Which system you are using ?
What is more puzzling is that the INIT command and the initial LD List
command goes through, but the PD List command gets blocked.
Incidentally, this is not consistent; occasionally even the LD List command
gets blocked, and the DMAR messages occur earlier.
LD command use megasas_issue_polled which is already timeout based mechanism.
Below are list of DCMD command which use infinite timeout.
megasas_get_seq_num
megasas_flush_cache
megasas_shutdown_controller
megasas_mgmt_fw_ioctl
We can convert all DCMD except IOCTL with timeout value. For you
megasas_get_seq_num might be hang in FW. It cannot be megasas_get_ld_list.
Anyway. Point is, if we cannot timout these initial commands the
megaraid_sas driver will be stuck during initialisation (as the loop _never_
terminates).
Which in turn means that the modprobe command hangs indefinitely, and
you cannot even unload the module.
The only way to recover here is a reboot.
Nasty.
Hence the patch for the timeout; when this triggers the HBA
Re: [PATCH 1/6] megaraid_sas: Do not wait forever
On 01/24/2014 09:34 AM, Desai, Kashyap wrote:
-Original Message-
From: Hannes Reinecke [mailto:h...@suse.de]
Sent: Friday, January 24, 2014 1:54 PM
To: Desai, Kashyap; James Bottomley
Cc: linux-scsi@vger.kernel.org; Adam Radford; Saxena, Sumit
Subject: Re: [PATCH 1/6] megaraid_sas: Do not wait forever
On 01/24/2014 08:46 AM, Desai, Kashyap wrote:
Hannes:
We have already worked on wait_event usage in
megasas_issue_blocked_cmd.
That code will be posted by LSI once we received test result from
LSI Q/A team.
If you see the current OCR code in Linux Driver we do re-send the IOCTL
command.
MR product does not want IOCTL timeout due to some reason. That is why
even if FW faulted, Driver will do OCR and re-send all existing
Management commands
(IOCTL comes under management commands).
Just for info. (see below snippet in OCR code)
/* Re-fire management commands */
for (j = 0 ; j instance-max_fw_cmds; j++) {
cmd_fusion = fusion-cmd_list[j];
if (cmd_fusion-sync_cmd_idx !=
(u32)ULONG_MAX) {
cmd_mfi =
instance-cmd_list[cmd_fusion-
sync_cmd_idx];
if (cmd_mfi-frame-dcmd.opcode ==
MR_DCMD_LD_MAP_GET_INFO) {
megasas_return_cmd(instance, cmd_mfi);
megasas_return_cmd_fusion(instance, cmd_fusion);
Current MR Driver is not designed to add timeout for DCMD and IOCTL
path.
[ I added timeout only for limited DCMDs, which are harmless to
continue after timeout ]
As of now, you can skip this patch and we will be submitting patch to fix
similar issue.
But note, we cannot add complete wait_event_timeout due to day-1
design, but will try to cover wait_event_timout for some valid cases.
Ouch.
The reason I sent this patch is that I've got an Intel box here, which blocks
megaraid_sas initialisation when the IOMMU is turned on:
[ 21.867264] megasas: io_request_frames 880800f5
[ 21.867363] megasas: init frame fff57000
[ 22.223234] megasas: frame status 00
[ 22.223235] megasas: IOC Init cmd success
[ 22.223282] megasas: ld map 88080b60
[ 22.223289] megasas: issue dcmd 05 opcode 300e101
[ 22.244184] dmar: DRHD: handling fault status reg 2
[ 22.244186] dmar: DMAR:[DMA Read] Request device [06:00.0] fault
addr 698
[ 22.244186] DMAR:[fault reason 06] PTE Read access is not set
[ 22.247223] megasas: frame status 00
[ 22.247231] megasas: issue dcmd 05 opcode 300e101
[ 22.247231] megasas: INIT adapter done
[ 22.247237] megasas: pd list 88080cfd size 8192
[ 22.247237] megasas: issue dcmd 05 opcode 2010100
[ 22.253516] dmar: DRHD: handling fault status reg 102
[ 22.253518] dmar: DMAR:[DMA Write] Request device [06:00.0] fault
addr e3f
[ 22.253518] DMAR:[fault reason 05] PTE Write access is not set
[ 22.253521] dmar: DMAR:[DMA Write] Request device [06:00.0] fault
addr e3f
[ 22.253521] DMAR:[fault reason 05] PTE Write access is not set
[ 22.253523] dmar: DMAR:[DMA Write] Request device [06:00.0] fault
addr e3f
[ Some more DMAR messages snipped ]
[ 22.273199] dmar: DRHD: handling fault status reg 2
[ 22.273201] dmar: DMAR:[DMA Read] Request device [06:00.0] fault
addr 6cef000
[ 22.273201] DMAR:[fault reason 06] PTE Read access is not set
[ .. ]
[ 94.222456] megasas: frame status ff
[ 94.240946] megasas: failed to get PD list
(I've inserted some debugging messages :-)
This is really weird. The 'write' faults do correspond with the number of
(megaraid) commands, reserved at the initial step.
(This is a 'Fury' card, btw).
Fury card has iMR FW and we have seen issue with iMR FW if IOMMU is ON, but
not like driver load failure.
Is your OS driver behind Fury ? What is a Raid type used on your setup ?
It's SLES12 (alpha), basically plain 3.13
Which system you are using ?
Pre-production hardware, admittedly. So it _might_ be a BIOS issue.
What is more puzzling is that the INIT command and the initial LD List
command goes through, but the PD List command gets blocked.
Incidentally, this is not consistent; occasionally even the LD List command
gets blocked, and the DMAR messages occur earlier.
LD command use megasas_issue_polled which is already timeout based mechanism.
Below are list of DCMD command which use infinite timeout.
megasas_get_seq_num
megasas_flush_cache
megasas_shutdown_controller
megasas_mgmt_fw_ioctl
We can convert all DCMD except IOCTL with timeout value. For you
megasas_get_seq_num
might be hang in FW. It cannot be megasas_get_ld_list.
Ahh. Okay, will try be modify megasas_get_seq_num() and see if that
works.
Cheers,
Hannes
P.S.: I've also send an earlier patch named 'megaraid_sas: disable
controller reset for PPC' to linux-scsi. Care to review it, too?
Thanks.
--
Dr. Hannes
RE: [PATCH 1/6] megaraid_sas: Do not wait forever
Hannes:
We have already worked on wait_event usage in megasas_issue_blocked_cmd.
That code will be posted by LSI once we received test result from LSI Q/A team.
If you see the current OCR code in Linux Driver we do re-send the IOCTL
command.
MR product does not want IOCTL timeout due to some reason. That is why even if
FW faulted, Driver will do OCR and re-send all existing Management commands
(IOCTL comes under management commands).
Just for info. (see below snippet in OCR code)
/* Re-fire management commands */
for (j = 0 ; j instance-max_fw_cmds; j++) {
cmd_fusion = fusion-cmd_list[j];
if (cmd_fusion-sync_cmd_idx != (u32)ULONG_MAX)
{
cmd_mfi =
instance-cmd_list[cmd_fusion-sync_cmd_idx];
if (cmd_mfi-frame-dcmd.opcode ==
MR_DCMD_LD_MAP_GET_INFO) {
megasas_return_cmd(instance,
cmd_mfi);
megasas_return_cmd_fusion(instance, cmd_fusion);
Current MR Driver is not designed to add timeout for DCMD and IOCTL path. [
I added timeout only for limited DCMDs, which are harmless to continue after
timeout ]
As of now, you can skip this patch and we will be submitting patch to fix
similar issue.
But note, we cannot add complete wait_event_timeout due to day-1 design, but
will try to cover wait_event_timout for some valid cases.
` Kashyap
-Original Message-
From: Hannes Reinecke [mailto:h...@suse.de]
Sent: Thursday, January 16, 2014 3:56 PM
To: James Bottomley
Cc: linux-scsi@vger.kernel.org; Hannes Reinecke; Desai, Kashyap; Adam
Radford
Subject: [PATCH 1/6] megaraid_sas: Do not wait forever
If the firmware is incommunicado for whatever reason the driver will wait
forever during initialisation, causing all sorts of hangcheck timers to
trigger.
We should rather wait for a defined time, and give up on the command if no
response was received.
Cc: Kashyap Desai kashyap.de...@lsi.com
Cc: Adam Radford aradf...@gmail.com
Signed-off-by: Hannes Reinecke h...@suse.de
---
drivers/scsi/megaraid/megaraid_sas_base.c | 43 ++
-
1 file changed, 25 insertions(+), 18 deletions(-)
diff --git a/drivers/scsi/megaraid/megaraid_sas_base.c
b/drivers/scsi/megaraid/megaraid_sas_base.c
index 3b7ad10..95d4e5c 100644
--- a/drivers/scsi/megaraid/megaraid_sas_base.c
+++ b/drivers/scsi/megaraid/megaraid_sas_base.c
@@ -911,9 +911,11 @@ megasas_issue_blocked_cmd(struct
megasas_instance *instance,
instance-instancet-issue_dcmd(instance, cmd);
- wait_event(instance-int_cmd_wait_q, cmd-cmd_status !=
ENODATA);
+ wait_event_timeout(instance-int_cmd_wait_q,
+cmd-cmd_status != ENODATA,
+MEGASAS_INTERNAL_CMD_WAIT_TIME * HZ);
- return 0;
+ return cmd-cmd_status == ENODATA ? -ENODATA : 0;
}
/**
@@ -932,11 +934,12 @@ megasas_issue_blocked_abort_cmd(struct
megasas_instance *instance, {
struct megasas_cmd *cmd;
struct megasas_abort_frame *abort_fr;
+ int status;
cmd = megasas_get_cmd(instance);
if (!cmd)
- return -1;
+ return -ENOMEM;
abort_fr = cmd-frame-abort;
@@ -960,11 +963,14 @@ megasas_issue_blocked_abort_cmd(struct
megasas_instance *instance,
/*
* Wait for this cmd to complete
*/
- wait_event(instance-abort_cmd_wait_q, cmd-cmd_status !=
0xFF);
+ wait_event_timeout(instance-abort_cmd_wait_q,
+cmd-cmd_status != 0xFF,
+MEGASAS_INTERNAL_CMD_WAIT_TIME * HZ);
cmd-sync_cmd = 0;
+ status = cmd-cmd_status;
megasas_return_cmd(instance, cmd);
- return 0;
+ return status == 0xFF ? -ENODATA : 0;
}
/**
@@ -3902,6 +3908,7 @@ megasas_get_seq_num(struct megasas_instance
*instance,
struct megasas_dcmd_frame *dcmd;
struct megasas_evt_log_info *el_info;
dma_addr_t el_info_h = 0;
+ int rc;
cmd = megasas_get_cmd(instance);
@@ -3933,23 +3940,23 @@ megasas_get_seq_num(struct
megasas_instance *instance,
dcmd-sgl.sge32[0].phys_addr = cpu_to_le32(el_info_h);
dcmd-sgl.sge32[0].length = cpu_to_le32(sizeof(struct
megasas_evt_log_info));
- megasas_issue_blocked_cmd(instance, cmd);
-
- /*
- * Copy the data back into callers buffer
- */
- eli-newest_seq_num = le32_to_cpu(el_info-newest_seq_num);
- eli-oldest_seq_num = le32_to_cpu(el_info-oldest_seq_num);
- eli-clear_seq_num = le32_to_cpu(el_info-clear_seq_num);
- eli-shutdown_seq_num = le32_to_cpu(el_info-
shutdown_seq_num);
- eli-boot_seq_num = le32_to_cpu(el_info-boot_seq_num);
-
+ rc = megasas_issue_blocked_cmd(instance, cmd);
+ if (!rc) {
+ /*
+ * Copy the
