Re: [PATCHv3 0/9] New EH command timeout handler
Hi, Hannes: On 07/15/2013 02:05 PM, Ren Mingxin wrote: On 07/12/2013 06:27 PM, Hannes Reinecke wrote: On 07/12/2013 12:00 PM, Ren Mingxin wrote: On 07/12/2013 02:09 PM, Hannes Reinecke wrote: On 07/12/2013 06:14 AM, Ren Mingxin wrote: On 07/01/2013 10:24 PM, Hannes Reinecke wrote: With the original SCSI EH I got: # time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 142.652 s, 118 kB/s real2m22.657s user0m0.013s sys0m0.145s With this patchset I got: # time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 52.1579 s, 322 kB/s real0m52.163s user0m0.012s sys0m0.145s Test was to disable RSCN on the target port, disable the target port, and then start the 'dd' command as indicated. Do you mean disabling RSCN/port is enough? I'm afraid I couldn't reproduce the problem by your steps. Both with and without your patchset are the same 'dd' result: 27s. Please let me know where I neglected or mistook: 1) I made a dm-multipath target 'dm-0' whose grouping policy was failover; 2) Disable RSCN/port via brocade fc switch: SW300:root portcfg rscnsupr 15 --enable; portDisable 15 3) Start the 'dd' command: # time dd if=/dev/zero of=/dev/dm-0 bs=4k count=4k oflag=direct dd: writing `/dev/sde': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 27.8588 s, 0.0 kB/s real0m27.860s user0m0.001s sys 0m0.000s You are aware that you have to disable RSCNs on the _target_ port, right? Disabling RSCNs on the _initiator_ ports is a well-tested case, and the one which actually makes sense (and is even implemented in QLogic switches). Disabling RSCNs for the _target_ port, OTOH, has a very questionable nature (hence QLogic switches don't even allow you to do this). You're right. By disabling RSCNs on target port, I've reproduced this problem. Thank you so much. But I've encountered the bug I said before. I'll test again with your new patchset once you send. Could you check with the attached patch? That should convert it to delayed_work and avoid this issue. Unfortunately, the login prompt couldn't be entered in and BUGs were printed ceaselessly while os booting with this patch. The BUGs are like below: BUG: scheduling while atomic: swapper/0/0/0x1100 Modules linked in: mptsas(F+) mptscsih(F) mptbase(F) scsi_transport_sas(F) CPU: 0 PID: 0 Comm: swapper/0 Tainted: GF3.10.0hannes+ #10 Hardware name: FUJITSU-SV PRIMEQUEST 1800E/SB-8GDIMM-CN, BIOS PRIMEQUEST 1000 Series BIOS Version 1.39 11/16/2012 88047ee03b68 8153ada4 88047ee03b78 8107389d 88047ee03c08 8153ca26 81a01fd8 00012d00 81a00010 00012d00 00012d00 Call Trace: IRQ [8153ada4] dump_stack+0x19/0x1d [8107389d] __schedule_bug+0x4d/0x60 [8153ca26] __schedule+0x646/0x6f0 [8107749a] __cond_resched+0x2a/0x40 [8153cb60] _cond_resched+0x30/0x40 [8105fecc] start_flush_work+0x2c/0x140 [8105fffa] flush_work+0x1a/0x40 [8105fb39] ? try_to_grab_pending+0x109/0x190 [8106027e] __cancel_work_timer+0x7e/0x110 [81060323] cancel_delayed_work_sync+0x13/0x20 [81374ec5] scsi_put_command+0x65/0xa0 This bug is caused by the sync function 'cancel_delayed_work_sync' which is invoked in the interrupt context. By replacing it by non- sync function 'cancel_delayed_work' in 'scsi_put_command' can avoid. Do you think there is such need to sync in the function 'scsi_put_ command'? Since SCSI command block will be freed here, it is NOT necessary to wait for the abort work to finish on it, yes? Thanks, Ren [8137d5aa] scsi_next_command+0x3a/0x60 [8137dedb] scsi_end_request+0xab/0xb0 [8137e1ef] scsi_io_completion+0x9f/0x670 [813744e4] scsi_finish_command+0xd4/0x140 [8137e927] scsi_softirq_done+0x147/0x170 [81239534] blk_done_softirq+0x74/0x90 [81049a4f] __do_softirq+0xef/0x260 [81049cb5] irq_exit+0xb5/0xc0 [81548406] do_IRQ+0x66/0xe0 [8153e5ea] common_interrupt+0x6a/0x6a EOI [8109b5f2] ? clockevents_notify+0x52/0x150 [8142dce3] ? cpuidle_enter_state+0x53/0xd0 [8142dcdf] ? cpuidle_enter_state+0x4f/0xd0 [8142e10f] cpuidle_idle_call+0xcf/0x160 [8100ab1e] arch_cpu_idle+0xe/0x30 [81093275] cpu_idle_loop+0x65/0x1f0 [81093470] cpu_startup_entry+0x70/0x80 [81529427] rest_init+0x77/0x80 [81b0e1bb] start_kernel+0x41a/0x427 [81b0dbbf] ? repair_env_string+0x5b/0x5b [81b0d5a1] x86_64_start_reservations+0x2a/0x2c [81b0d6d2] x86_64_start_kernel+0x12f/0x136 -- To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a
Re: [PATCHv3 0/9] New EH command timeout handler
On 08/07/2013 12:08 PM, Ren Mingxin wrote: Hi, Hannes: On 07/15/2013 02:05 PM, Ren Mingxin wrote: On 07/12/2013 06:27 PM, Hannes Reinecke wrote: On 07/12/2013 12:00 PM, Ren Mingxin wrote: On 07/12/2013 02:09 PM, Hannes Reinecke wrote: On 07/12/2013 06:14 AM, Ren Mingxin wrote: On 07/01/2013 10:24 PM, Hannes Reinecke wrote: With the original SCSI EH I got: # time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 142.652 s, 118 kB/s real2m22.657s user0m0.013s sys0m0.145s With this patchset I got: # time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 52.1579 s, 322 kB/s real0m52.163s user0m0.012s sys0m0.145s Test was to disable RSCN on the target port, disable the target port, and then start the 'dd' command as indicated. Do you mean disabling RSCN/port is enough? I'm afraid I couldn't reproduce the problem by your steps. Both with and without your patchset are the same 'dd' result: 27s. Please let me know where I neglected or mistook: 1) I made a dm-multipath target 'dm-0' whose grouping policy was failover; 2) Disable RSCN/port via brocade fc switch: SW300:root portcfg rscnsupr 15 --enable; portDisable 15 3) Start the 'dd' command: # time dd if=/dev/zero of=/dev/dm-0 bs=4k count=4k oflag=direct dd: writing `/dev/sde': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 27.8588 s, 0.0 kB/s real0m27.860s user0m0.001s sys 0m0.000s You are aware that you have to disable RSCNs on the _target_ port, right? Disabling RSCNs on the _initiator_ ports is a well-tested case, and the one which actually makes sense (and is even implemented in QLogic switches). Disabling RSCNs for the _target_ port, OTOH, has a very questionable nature (hence QLogic switches don't even allow you to do this). You're right. By disabling RSCNs on target port, I've reproduced this problem. Thank you so much. But I've encountered the bug I said before. I'll test again with your new patchset once you send. Could you check with the attached patch? That should convert it to delayed_work and avoid this issue. Unfortunately, the login prompt couldn't be entered in and BUGs were printed ceaselessly while os booting with this patch. The BUGs are like below: BUG: scheduling while atomic: swapper/0/0/0x1100 Modules linked in: mptsas(F+) mptscsih(F) mptbase(F) scsi_transport_sas(F) CPU: 0 PID: 0 Comm: swapper/0 Tainted: GF3.10.0hannes+ #10 Hardware name: FUJITSU-SV PRIMEQUEST 1800E/SB-8GDIMM-CN, BIOS PRIMEQUEST 1000 Series BIOS Version 1.39 11/16/2012 88047ee03b68 8153ada4 88047ee03b78 8107389d 88047ee03c08 8153ca26 81a01fd8 00012d00 81a00010 00012d00 00012d00 Call Trace: IRQ [8153ada4] dump_stack+0x19/0x1d [8107389d] __schedule_bug+0x4d/0x60 [8153ca26] __schedule+0x646/0x6f0 [8107749a] __cond_resched+0x2a/0x40 [8153cb60] _cond_resched+0x30/0x40 [8105fecc] start_flush_work+0x2c/0x140 [8105fffa] flush_work+0x1a/0x40 [8105fb39] ? try_to_grab_pending+0x109/0x190 [8106027e] __cancel_work_timer+0x7e/0x110 [81060323] cancel_delayed_work_sync+0x13/0x20 [81374ec5] scsi_put_command+0x65/0xa0 This bug is caused by the sync function 'cancel_delayed_work_sync' which is invoked in the interrupt context. By replacing it by non- sync function 'cancel_delayed_work' in 'scsi_put_command' can avoid. Do you think there is such need to sync in the function 'scsi_put_ command'? Since SCSI command block will be freed here, it is NOT necessary to wait for the abort work to finish on it, yes? You are right, cancel_delayed_work() should be sufficient here. I'll give it a spin and repost the patchset. Cheers, Hannes -- Dr. Hannes Reinecke zSeries Storage h...@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv3 0/9] New EH command timeout handler
Hi, Hannes: On 07/12/2013 06:27 PM, Hannes Reinecke wrote: On 07/12/2013 12:00 PM, Ren Mingxin wrote: On 07/12/2013 02:09 PM, Hannes Reinecke wrote: On 07/12/2013 06:14 AM, Ren Mingxin wrote: On 07/01/2013 10:24 PM, Hannes Reinecke wrote: With the original SCSI EH I got: # time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 142.652 s, 118 kB/s real2m22.657s user0m0.013s sys0m0.145s With this patchset I got: # time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 52.1579 s, 322 kB/s real0m52.163s user0m0.012s sys0m0.145s Test was to disable RSCN on the target port, disable the target port, and then start the 'dd' command as indicated. Do you mean disabling RSCN/port is enough? I'm afraid I couldn't reproduce the problem by your steps. Both with and without your patchset are the same 'dd' result: 27s. Please let me know where I neglected or mistook: 1) I made a dm-multipath target 'dm-0' whose grouping policy was failover; 2) Disable RSCN/port via brocade fc switch: SW300:root portcfg rscnsupr 15 --enable; portDisable 15 3) Start the 'dd' command: # time dd if=/dev/zero of=/dev/dm-0 bs=4k count=4k oflag=direct dd: writing `/dev/sde': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 27.8588 s, 0.0 kB/s real0m27.860s user0m0.001s sys 0m0.000s You are aware that you have to disable RSCNs on the _target_ port, right? Disabling RSCNs on the _initiator_ ports is a well-tested case, and the one which actually makes sense (and is even implemented in QLogic switches). Disabling RSCNs for the _target_ port, OTOH, has a very questionable nature (hence QLogic switches don't even allow you to do this). You're right. By disabling RSCNs on target port, I've reproduced this problem. Thank you so much. But I've encountered the bug I said before. I'll test again with your new patchset once you send. Could you check with the attached patch? That should convert it to delayed_work and avoid this issue. Unfortunately, the login prompt couldn't be entered in and BUGs were printed ceaselessly while os booting with this patch. The BUGs are like below: BUG: scheduling while atomic: swapper/0/0/0x1100 Modules linked in: mptsas(F+) mptscsih(F) mptbase(F) scsi_transport_sas(F) CPU: 0 PID: 0 Comm: swapper/0 Tainted: GF3.10.0hannes+ #10 Hardware name: FUJITSU-SV PRIMEQUEST 1800E/SB-8GDIMM-CN, BIOS PRIMEQUEST 1000 Series BIOS Version 1.39 11/16/2012 88047ee03b68 8153ada4 88047ee03b78 8107389d 88047ee03c08 8153ca26 81a01fd8 00012d00 81a00010 00012d00 00012d00 Call Trace: IRQ [8153ada4] dump_stack+0x19/0x1d [8107389d] __schedule_bug+0x4d/0x60 [8153ca26] __schedule+0x646/0x6f0 [8107749a] __cond_resched+0x2a/0x40 [8153cb60] _cond_resched+0x30/0x40 [8105fecc] start_flush_work+0x2c/0x140 [8105fffa] flush_work+0x1a/0x40 [8105fb39] ? try_to_grab_pending+0x109/0x190 [8106027e] __cancel_work_timer+0x7e/0x110 [81060323] cancel_delayed_work_sync+0x13/0x20 [81374ec5] scsi_put_command+0x65/0xa0 [8137d5aa] scsi_next_command+0x3a/0x60 [8137dedb] scsi_end_request+0xab/0xb0 [8137e1ef] scsi_io_completion+0x9f/0x670 [813744e4] scsi_finish_command+0xd4/0x140 [8137e927] scsi_softirq_done+0x147/0x170 [81239534] blk_done_softirq+0x74/0x90 [81049a4f] __do_softirq+0xef/0x260 [81049cb5] irq_exit+0xb5/0xc0 [81548406] do_IRQ+0x66/0xe0 [8153e5ea] common_interrupt+0x6a/0x6a EOI [8109b5f2] ? clockevents_notify+0x52/0x150 [8142dce3] ? cpuidle_enter_state+0x53/0xd0 [8142dcdf] ? cpuidle_enter_state+0x4f/0xd0 [8142e10f] cpuidle_idle_call+0xcf/0x160 [8100ab1e] arch_cpu_idle+0xe/0x30 [81093275] cpu_idle_loop+0x65/0x1f0 [81093470] cpu_startup_entry+0x70/0x80 [81529427] rest_init+0x77/0x80 [81b0e1bb] start_kernel+0x41a/0x427 [81b0dbbf] ? repair_env_string+0x5b/0x5b [81b0d5a1] x86_64_start_reservations+0x2a/0x2c [81b0d6d2] x86_64_start_kernel+0x12f/0x136 If there is any info I havn't expatiated, please let me know. Thanks, Ren -- To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv3 0/9] New EH command timeout handler
On 07/12/2013 06:14 AM, Ren Mingxin wrote: Hi, Hannes: On 07/01/2013 10:24 PM, Hannes Reinecke wrote: With the original SCSI EH I got: # time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 142.652 s, 118 kB/s real2m22.657s user0m0.013s sys0m0.145s With this patchset I got: # time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 52.1579 s, 322 kB/s real0m52.163s user0m0.012s sys0m0.145s Test was to disable RSCN on the target port, disable the target port, and then start the 'dd' command as indicated. Do you mean disabling RSCN/port is enough? I'm afraid I couldn't reproduce the problem by your steps. Both with and without your patchset are the same 'dd' result: 27s. Please let me know where I neglected or mistook: 1) I made a dm-multipath target 'dm-0' whose grouping policy was failover; 2) Disable RSCN/port via brocade fc switch: SW300:root portcfg rscnsupr 15 --enable; portDisable 15 3) Start the 'dd' command: # time dd if=/dev/zero of=/dev/dm-0 bs=4k count=4k oflag=direct dd: writing `/dev/sde': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 27.8588 s, 0.0 kB/s real0m27.860s user0m0.001s sys 0m0.000s You are aware that you have to disable RSCNs on the _target_ port, right? Disabling RSCNs on the _initiator_ ports is a well-tested case, and the one which actually makes sense (and is even implemented in QLogic switches). Disabling RSCNs for the _target_ port, OTOH, has a very questionable nature (hence QLogic switches don't even allow you to do this). [ .. ] Another question: I also tried to produce timeouts by modifying Yasui's module(please see APPENDIX A): http://www.spinics.net/lists/linux-scsi/msg35091.html But I got a bug with your this patchset by follwing steps(there was not such bug without your patchset): # grep lpfc_template /proc/kallsyms a00f9240 d lpfc_template[lpfc] # multipath -ll ... mpathb (36000b5d0006a006a14e7000c) dm-1 FUJITSU,ETERNUS_DX400 size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=130 status=active | `- 2:0:0:1 sdf 8:80 active ready running `-+- policy='round-robin 0' prio=130 status=enabled `- 3:0:0:1 sdh 8:112 active ready running # insmod scsi_tmo_mod.ko param=0xa00f9240,2:0:0:1; time dd if=/dev/zero of=/dev/dm-1 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 151.194 s, 111 kB/s real2m31.195s user0m0.004s sys0m0.111s Please see logs in APPENDIX B. Do you think this bug is irrelevant to your patchset? Hmm. No, sadly not. 'cancel_work_sync' cannot be called from an interrupt context; guess I'll need to convert it to delayed work. Thanks for testing; will be updating the patchset. Cheers, Hannes -- Dr. Hannes Reinecke zSeries Storage h...@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv3 0/9] New EH command timeout handler
Hi, Hannes: On 07/12/2013 02:09 PM, Hannes Reinecke wrote: On 07/12/2013 06:14 AM, Ren Mingxin wrote: On 07/01/2013 10:24 PM, Hannes Reinecke wrote: With the original SCSI EH I got: # time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 142.652 s, 118 kB/s real2m22.657s user0m0.013s sys0m0.145s With this patchset I got: # time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 52.1579 s, 322 kB/s real0m52.163s user0m0.012s sys0m0.145s Test was to disable RSCN on the target port, disable the target port, and then start the 'dd' command as indicated. Do you mean disabling RSCN/port is enough? I'm afraid I couldn't reproduce the problem by your steps. Both with and without your patchset are the same 'dd' result: 27s. Please let me know where I neglected or mistook: 1) I made a dm-multipath target 'dm-0' whose grouping policy was failover; 2) Disable RSCN/port via brocade fc switch: SW300:root portcfg rscnsupr 15 --enable; portDisable 15 3) Start the 'dd' command: # time dd if=/dev/zero of=/dev/dm-0 bs=4k count=4k oflag=direct dd: writing `/dev/sde': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 27.8588 s, 0.0 kB/s real0m27.860s user0m0.001s sys 0m0.000s You are aware that you have to disable RSCNs on the _target_ port, right? Disabling RSCNs on the _initiator_ ports is a well-tested case, and the one which actually makes sense (and is even implemented in QLogic switches). Disabling RSCNs for the _target_ port, OTOH, has a very questionable nature (hence QLogic switches don't even allow you to do this). You're right. By disabling RSCNs on target port, I've reproduced this problem. Thank you so much. But I've encountered the bug I said before. I'll test again with your new patchset once you send. Thanks, Ren [ .. ] Another question: I also tried to produce timeouts by modifying Yasui's module(please see APPENDIX A): http://www.spinics.net/lists/linux-scsi/msg35091.html But I got a bug with your this patchset by follwing steps(there was not such bug without your patchset): # grep lpfc_template /proc/kallsyms a00f9240 d lpfc_template[lpfc] # multipath -ll ... mpathb (36000b5d0006a006a14e7000c) dm-1 FUJITSU,ETERNUS_DX400 size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=130 status=active | `- 2:0:0:1 sdf 8:80 active ready running `-+- policy='round-robin 0' prio=130 status=enabled `- 3:0:0:1 sdh 8:112 active ready running # insmod scsi_tmo_mod.ko param=0xa00f9240,2:0:0:1; time dd if=/dev/zero of=/dev/dm-1 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 151.194 s, 111 kB/s real2m31.195s user0m0.004s sys0m0.111s Please see logs in APPENDIX B. Do you think this bug is irrelevant to your patchset? Hmm. No, sadly not. 'cancel_work_sync' cannot be called from an interrupt context; guess I'll need to convert it to delayed work. Thanks for testing; will be updating the patchset. -- To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv3 0/9] New EH command timeout handler
On 07/12/2013 12:00 PM, Ren Mingxin wrote:
Hi, Hannes:
On 07/12/2013 02:09 PM, Hannes Reinecke wrote:
On 07/12/2013 06:14 AM, Ren Mingxin wrote:
On 07/01/2013 10:24 PM, Hannes Reinecke wrote:
With the original SCSI EH I got:
# time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct
4096+0 records in
4096+0 records out
16777216 bytes (17 MB) copied, 142.652 s, 118 kB/s
real2m22.657s
user0m0.013s
sys0m0.145s
With this patchset I got:
# time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct
4096+0 records in
4096+0 records out
16777216 bytes (17 MB) copied, 52.1579 s, 322 kB/s
real0m52.163s
user0m0.012s
sys0m0.145s
Test was to disable RSCN on the target port, disable the
target port, and then start the 'dd' command as indicated.
Do you mean disabling RSCN/port is enough? I'm afraid I couldn't
reproduce the problem by your steps. Both with and without your
patchset are the same 'dd' result: 27s. Please let me know where I
neglected or mistook:
1) I made a dm-multipath target 'dm-0' whose grouping policy was
failover;
2) Disable RSCN/port via brocade fc switch:
SW300:root portcfg rscnsupr 15 --enable; portDisable 15
3) Start the 'dd' command:
# time dd if=/dev/zero of=/dev/dm-0 bs=4k count=4k oflag=direct
dd: writing `/dev/sde': Input/output error
1+0 records in
0+0 records out
0 bytes (0 B) copied, 27.8588 s, 0.0 kB/s
real0m27.860s
user0m0.001s
sys 0m0.000s
You are aware that you have to disable RSCNs on the _target_ port,
right?
Disabling RSCNs on the _initiator_ ports is a well-tested case, and
the one which actually makes sense (and is even implemented in
QLogic switches).
Disabling RSCNs for the _target_ port, OTOH, has a very questionable
nature (hence QLogic switches don't even allow you to do this).
You're right. By disabling RSCNs on target port, I've reproduced this
problem. Thank you so much. But I've encountered the bug I said
before. I'll test again with your new patchset once you send.
Could you check with the attached patch? That should convert it to
delayed_work and avoid this issue.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries Storage
h...@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
From 2d0851bea02e3d15bf403e6800cc7164f2e211f2 Mon Sep 17 00:00:00 2001
From: Hannes Reinecke h...@suse.de
Date: Fri, 12 Jul 2013 12:06:19 +0200
Subject: [PATCH] scsi: use delayed_work to trigger aborts
Command aborts are invoked from an interrupt, so we cannot use
cancel_work_sync() when an outstanding abort needs to be cancelled.
So convert the mechanism to delayed_work, which can be cancelled
even from an interrupt context.
Signed-off-by: Hannes Reinecke h...@suse.de
diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
index a62ae84..13a3418 100644
--- a/drivers/scsi/scsi.c
+++ b/drivers/scsi/scsi.c
@@ -297,7 +297,7 @@ struct scsi_cmnd *scsi_get_command(struct scsi_device *dev, gfp_t gfp_mask)
cmd-device = dev;
INIT_LIST_HEAD(cmd-list);
- INIT_WORK(cmd-abort_work, scmd_eh_abort_handler);
+ INIT_DELAYED_WORK(cmd-abort_work, scmd_eh_abort_handler);
spin_lock_irqsave(dev-list_lock, flags);
list_add_tail(cmd-list, dev-cmd_list);
spin_unlock_irqrestore(dev-list_lock, flags);
@@ -354,6 +354,8 @@ void scsi_put_command(struct scsi_cmnd *cmd)
list_del_init(cmd-list);
spin_unlock_irqrestore(cmd-device-list_lock, flags);
+ cancel_delayed_work_sync(cmd-abort_work);
+
__scsi_put_command(cmd-device-host, cmd, sdev-sdev_gendev);
}
EXPORT_SYMBOL(scsi_put_command);
diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
index 28c272f..cd443b2 100644
--- a/drivers/scsi/scsi_error.c
+++ b/drivers/scsi/scsi_error.c
@@ -187,7 +187,7 @@ void
scmd_eh_abort_handler(struct work_struct *work)
{
struct scsi_cmnd *scmd =
- container_of(work, struct scsi_cmnd, abort_work);
+ container_of(work, struct scsi_cmnd, abort_work.work);
struct scsi_device *sdev = scmd-device;
unsigned long flags;
int rtn;
@@ -254,7 +254,7 @@ scsi_abort_command(struct scsi_cmnd *scmd)
SCSI_LOG_ERROR_RECOVERY(3,
scmd_printk(KERN_INFO, scmd,
scmd %p abort timeout\n, scmd));
- cancel_work_sync(scmd-abort_work);
+ cancel_delayed_work(scmd-abort_work);
return BLK_EH_NOT_HANDLED;
}
@@ -279,7 +279,7 @@ scsi_abort_command(struct scsi_cmnd *scmd)
SCSI_LOG_ERROR_RECOVERY(3,
scmd_printk(KERN_INFO, scmd,
scmd %p abort scheduled\n, scmd));
- schedule_work(scmd-abort_work);
+ schedule_delayed_work(scmd-abort_work, HZ / 100);
return BLK_EH_SCHEDULED;
}
EXPORT_SYMBOL_GPL(scsi_abort_command);
diff --git a/include/scsi/scsi_cmnd.h b/include/scsi/scsi_cmnd.h
index a2f062e..e3137e3 100644
--- a/include/scsi/scsi_cmnd.h
+++ b/include/scsi/scsi_cmnd.h
@@ -55,7 +55,7 @@ struct scsi_cmnd {
Re: [PATCHv3 0/9] New EH command timeout handler
Hi, Hannes:
On 07/01/2013 10:24 PM, Hannes Reinecke wrote:
With the original SCSI EH I got:
# time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct
4096+0 records in
4096+0 records out
16777216 bytes (17 MB) copied, 142.652 s, 118 kB/s
real2m22.657s
user0m0.013s
sys 0m0.145s
With this patchset I got:
# time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct
4096+0 records in
4096+0 records out
16777216 bytes (17 MB) copied, 52.1579 s, 322 kB/s
real0m52.163s
user0m0.012s
sys 0m0.145s
Test was to disable RSCN on the target port, disable the
target port, and then start the 'dd' command as indicated.
Do you mean disabling RSCN/port is enough? I'm afraid I couldn't
reproduce the problem by your steps. Both with and without your
patchset are the same 'dd' result: 27s. Please let me know where I
neglected or mistook:
1) I made a dm-multipath target 'dm-0' whose grouping policy was
failover;
2) Disable RSCN/port via brocade fc switch:
SW300:root portcfg rscnsupr 15 --enable; portDisable 15
3) Start the 'dd' command:
# time dd if=/dev/zero of=/dev/dm-0 bs=4k count=4k oflag=direct
dd: writing `/dev/sde': Input/output error
1+0 records in
0+0 records out
0 bytes (0 B) copied, 27.8588 s, 0.0 kB/s
real0m27.860s
user0m0.001s
sys 0m0.000s
#) Corresponding logs in /var/log/messages
Jul 9 14:56:06 build kernel: lpfc :0d:00.1: 1:1305 Link Down Event
x4 received Data: x4 x20 x110 x0 x0
Jul 9 14:56:36 build kernel: rport-3:0-2: blocked FC remote port time
out: removing target and saving binding
Jul 9 14:56:36 build kernel: sd 3:0:0:0: rejecting I/O to offline device
Jul 9 14:56:36 build kernel: lpfc :0d:00.1: 1:(0):0203 Devloss
timeout on WWPN 20:41:00:0b:5d:6a:14:e7 NPort x620700 Data: x0 x8 x0
Jul 9 14:56:36 build kernel: sd 3:0:0:0: [sde] Synchronizing SCSI cache
Jul 9 14:56:36 build kernel: sd 3:0:0:0: [sde]
Jul 9 14:56:36 build kernel: Result: hostbyte=DID_NO_CONNECT
driverbyte=DRIVER_OK
Jul 9 14:56:36 build kernel: sd 3:0:0:1: [sdf] Synchronizing SCSI cache
Jul 9 14:56:36 build kernel: sd 3:0:0:1: [sdf]
Jul 9 14:56:36 build kernel: Result: hostbyte=DID_NO_CONNECT
driverbyte=DRIVER_OK
Jul 9 14:56:36 build multipathd: sdf: remove path (uevent)
Jul 9 14:56:36 build multipathd: mpatha: load table [0 104857600
multipath 1 queue_if_no_path 0 1 1 round-robin 0 1 1 8:112 1]
Jul 9 14:56:36 build multipathd: sdf: path removed from map mpatha
Jul 9 14:56:36 build udevd-work[8420]: error opening
ATTR{/sys/devices/pci:00/:00:03.0/:01:00.0/:02:01.0/:0a:00.0/:0b:01.0/:0d:00.1/host3/rport-3:0-2/target3:0:0/3:0:0:0/block/sde/queue/iosched/slice_idle}
for writing: No such file or directory
Jul 9 14:56:36 build udevd-work[8420]: error opening
ATTR{/sys/devices/pci:00/:00:03.0/:01:00.0/:02:01.0/:0a:00.0/:0b:01.0/:0d:00.1/host3/rport-3:0-2/target3:0:0/3:0:0:0/block/sde/queue/iosched/quantum}
for writing: No such file or directory
Jul 9 14:56:36 build multipathd: sde: remove path (uevent)
Jul 9 14:56:36 build multipathd: mpathb: load table [0 104857600
multipath 1 queue_if_no_path 0 1 1 round-robin 0 1 1 8:96 1]
Jul 9 14:56:36 build multipathd: sde: path removed from map mpathb
* there are two disks sde and sdf connected via port 15
Another question:
I also tried to produce timeouts by modifying Yasui's module(please
see APPENDIX A):
http://www.spinics.net/lists/linux-scsi/msg35091.html
But I got a bug with your this patchset by follwing steps(there was
not such bug without your patchset):
# grep lpfc_template /proc/kallsyms
a00f9240 d lpfc_template[lpfc]
# multipath -ll
...
mpathb (36000b5d0006a006a14e7000c) dm-1 FUJITSU,ETERNUS_DX400
size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| `- 2:0:0:1 sdf 8:80 active ready running
`-+- policy='round-robin 0' prio=130 status=enabled
`- 3:0:0:1 sdh 8:112 active ready running
# insmod scsi_tmo_mod.ko param=0xa00f9240,2:0:0:1; time dd
if=/dev/zero of=/dev/dm-1 bs=4k count=4k oflag=direct
4096+0 records in
4096+0 records out
16777216 bytes (17 MB) copied, 151.194 s, 111 kB/s
real2m31.195s
user0m0.004s
sys0m0.111s
Please see logs in APPENDIX B. Do you think this bug is irrelevant to
your patchset?
Thanks,
Ren
APPENDIX A:
/*
* scsi timeout injection module
*/
#include linux/module.h
#include scsi/scsi_cmnd.h
#include scsi/scsi_host.h
#include scsi/scsi_device.h
static struct scsi_host_template *sht;
static char config[32];
static struct target {
short host;
uint channel;
uint id;
uint lun;
} st;
static int (*org_qc)(struct Scsi_Host *, struct scsi_cmnd *);
static inline int check_dev(struct target *st, struct scsi_cmnd *cmd)
{
return (st-host == cmd-device-host-host_no
st-channel == cmd-device-channel
st-id ==
[PATCHv3 0/9] New EH command timeout handler
This patchset implements a new SCSI EH command timeout handler which will be sending command aborts inline without actually engaging SCSI EH. SCSI EH will only be invoked if command abort fails. In addition the commands will be returned directly if the command abort succeeded, cutting down recovery times dramatically. With the original SCSI EH I got: # time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 142.652 s, 118 kB/s real2m22.657s user0m0.013s sys 0m0.145s With this patchset I got: # time dd if=/dev/zero of=/dev/dm-2 bs=4k count=4k oflag=direct 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 52.1579 s, 322 kB/s real0m52.163s user0m0.012s sys 0m0.145s Test was to disable RSCN on the target port, disable the target port, and then start the 'dd' command as indicated. As a proof-of-concept I've also enabled the new timeout handler for virtio, so that things can be tested out more easily. As requested I've also hooked in the new SCSI timeout handler to SAS. Changes to the original version: - Use a private list in scsi_eh_abort_handler to avoid list starvation (pointed out by Joern Engel) - Terminate command aborts when the first abort fails - Do not attempt command aborts if the host is already in recovery or if the device is removed. - Flush abort workqueue if the device is removed. Changes to v2: - Removed eh_entry initialisation - Convert to per-command workqueue Comments etc are welcome. Hannes Reinecke (9): scsi: Fix erratic device offline during EH blk-timeout: add BLK_EH_SCHEDULED return code scsi: improved eh timeout handler virtio_scsi: Enable new EH timeout handler libsas: Enable new EH timeout handler mptsas: Enable new EH timeout handler mpt2sas: Enable new EH timeout handler mpt3sas: Enable new EH timeout handler scsi_transport_fc: Enable new EH timeout handler drivers/message/fusion/mptsas.c | 3 +- drivers/message/fusion/mptscsih.c| 7 ++ drivers/message/fusion/mptscsih.h| 1 + drivers/scsi/libsas/sas_scsi_host.c | 2 +- drivers/scsi/mpt2sas/mpt2sas_scsih.c | 13 ++- drivers/scsi/mpt3sas/mpt3sas_scsih.c | 11 +++ drivers/scsi/scsi.c | 1 + drivers/scsi/scsi_error.c| 171 ++- drivers/scsi/scsi_priv.h | 2 + drivers/scsi/scsi_transport_fc.c | 2 +- drivers/scsi/virtio_scsi.c | 8 ++ include/linux/blkdev.h | 1 + include/scsi/scsi_cmnd.h | 2 + 13 files changed, 199 insertions(+), 25 deletions(-) -- 1.7.12.4 -- To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
