Would be nice to be pointed to the correct place to report major
regressions, if not here.
Note that the same error occurs:
* on 4.18.19
* on 4.19.8-rc1
* with the latest firmware (16.00.01)
* on a number of other peoples' powerpc64/power9 hardware.
Note that both 4.18.X and 4.19.Y will, in addition to the above error,
also occasionally hit EEH fences (on 4.18 this appears to be the
predominant error, on 4.19 this is less common).
[ 1711.273181] EEH: Frozen PHB#31-PE#fd detected
[ 1711.273235] EEH: PE location: N/A, PHB location: N/A
[ 1711.273297] CPU: 13 PID: 10993 Comm: kworker/u128:4 Tainted: G
W 4.18.0-3-powerpc64le #1 Debian 4.18.20-2
[ 1711.273307] Workqueue: poll_mpt3sas0_statu _base_fault_reset_work
[mpt3sas]
[ 1711.273310] Call Trace:
[ 1711.273314] [c000000fb5623a80] [c00000000097690c]
dump_stack+0xb0/0xf4 (unreliable)
[ 1711.273318] [c000000fb5623ac0] [c00000000003cad0]
eeh_dev_check_failure+0x5e0/0x600
[ 1711.273321] [c000000fb5623b70] [c00000000003cb7c]
eeh_check_failure+0x8c/0xd0
[ 1711.273326] [c000000fb5623bb0] [c008000008b6935c]
mpt3sas_base_get_iocstate+0x94/0xb0 [mpt3sas]
[ 1711.273330] [c000000fb5623bf0] [c008000008b6da10]
_base_fault_reset_work+0xd8/0x310 [mpt3sas]
[ 1711.273334] [c000000fb5623c80] [c000000000129040]
process_one_work+0x260/0x530
[ 1711.273336] [c000000fb5623d20] [c000000000129398]
worker_thread+0x88/0x5d0
[ 1711.273340] [c000000fb5623dc0] [c000000000132238] kthread+0x1a8/0x1b0
[ 1711.273344] [c000000fb5623e30] [c00000000000bdd4]
ret_from_kernel_thread+0x5c/0x88
[ 1711.273347] mpt3sas_cm0: SAS host is non-operational !!!!
[ 1711.273690] EEH: Detected PCI bus error on PHB#31-PE#fd
[ 1711.273693] EEH: This PCI device has failed 1 times in the last hour
and will be permanently disabled after 5 failures.
[ 1711.273694] EEH: Notify device drivers to shutdown
[ 1711.273697] EEH: Beginning: 'error_detected(IO frozen)'
[ 1711.273699] EEH: PE#fd (PCI 0031:01:00.0): Invoking
mpt3sas->error_detected(IO frozen)
[ 1711.273701] mpt3sas_cm0: PCI error: detected callback, state(2)!!
[ 1711.274838] EEH: PE#fd (PCI 0031:01:00.0): mpt3sas driver reports:
'need reset'
[ 1711.274839] EEH: Finished:'error_detected(IO frozen)' with aggregate
recovery state:'need reset'
[ 1711.274842] EEH: Collect temporary log
[ 1711.274861] EEH: of node=0031:01:00.0
[ 1711.274863] EEH: PCI device/vendor: 00c41000
[ 1711.274864] EEH: PCI cmd/status register: 00100142
[ 1711.274865] EEH: PCI-E capabilities and status follow:
[ 1711.274872] EEH: PCI-E 00: 0002d010 10008025 00002950 00415083
[ 1711.274878] EEH: PCI-E 10: 10830000 00000000 00000000 00000000
[ 1711.274879] EEH: PCI-E 20: 00000000
[ 1711.274879] EEH: PCI-E AER capability register set follows:
[ 1711.274886] EEH: PCI-E AER 00: 1e020001 00000000 00000000 00462031
[ 1711.274892] EEH: PCI-E AER 10: 00000000 00002000 000001e0 00000000
[ 1711.274897] EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000
[ 1711.274899] EEH: PCI-E AER 30: 00000000 00010015
[ 1711.274900] PHB4 PHB#49 Diag-data (Version: 1)
[ 1711.274901] brdgCtl: 00000002
[ 1711.274902] RootSts: 00060040 00402000 a0830008 00100107 00000800
[ 1711.274902] PhbSts: 0000001c00000000 0000001c00000000
[ 1711.274903] Lem: 0000000110000000 0000000000000000
0000000010000000
[ 1711.274904] PhbErr: 0000088000000000 0000080000000000
2148000098000240 a008400000000000
[ 1711.274905] RxeArbErr: 0000000800000000 0000000800000000
0717010000000076 8a8201000000008a
[ 1711.274906] PblErr: 0000000000020000 0000000000020000
0000000000000000 0000000000000000
[ 1711.274907] RegbErr: 0040004000000000 0000004000000000
8800000400000000 0000000000000000
[ 1711.274908] PE[0fd] A/B: 8200802301000000 8a8201000000008a
[ 1711.274909] PE[100] A/B: 8000000000003bfe 00000000300d2593
[ 1711.274910] PE[1fd] A/B: 8000000000000000 8000000000000000
[ 1711.274910] EEH: Reset without hotplug activity
[ 1716.201556] EEH: Notify device drivers the completion of reset
[ 1716.201559] EEH: Beginning: 'slot_reset'
[ 1716.201568] EEH: PE#fd (PCI 0031:01:00.0): Invoking mpt3sas->slot_reset()
[ 1716.201569] mpt3sas_cm0: PCI error: slot reset callback!!
[ 1716.201649] mpt3sas 0031:01:00.0: Using 64-bit DMA iommu bypass
[ 1716.201656] mpt3sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED,
total mem (131847972 kB)
[ 1716.256003] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default
host page size to 4k
[ 1716.256024] mpt3sas_cm0: MSI-X vectors supported: 96, no of cores:
64, max_msix_vectors: -1
[ 1716.258361] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 75
[ 1716.258363] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 76
[ 1716.258364] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 77
[ 1716.258365] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 78
[ 1716.258366] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 79
[ 1716.258367] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 89
[ 1716.258368] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 90
[ 1716.258368] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 91
[ 1716.258369] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 92
[ 1716.258370] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 93
[ 1716.258371] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 94
[ 1716.258372] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 95
[ 1716.258373] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 97
[ 1716.258374] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 98
[ 1716.258375] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 99
[ 1716.258376] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 100
[ 1716.258377] mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 101
[ 1716.258378] mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 102
[ 1716.258379] mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 103
[ 1716.258380] mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 104
[ 1716.258381] mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 105
[ 1716.258382] mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 106
[ 1716.258383] mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 107
[ 1716.258384] mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 108
[ 1716.258385] mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 109
[ 1716.258385] mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 110
[ 1716.258386] mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 111
[ 1716.258387] mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 112
[ 1716.258388] mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 113
[ 1716.258389] mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 114
[ 1716.258390] mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 115
[ 1716.258391] mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 116
[ 1716.258392] mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 117
[ 1716.258393] mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 118
[ 1716.258394] mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 119
[ 1716.258395] mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 120
[ 1716.258396] mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 121
[ 1716.258397] mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 122
[ 1716.258398] mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 123
[ 1716.258399] mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 124
[ 1716.258400] mpt3sas0-msix40: PCI-MSI-X enabled: IRQ 125
[ 1716.258401] mpt3sas0-msix41: PCI-MSI-X enabled: IRQ 126
[ 1716.258402] mpt3sas0-msix42: PCI-MSI-X enabled: IRQ 127
[ 1716.258403] mpt3sas0-msix43: PCI-MSI-X enabled: IRQ 128
[ 1716.258403] mpt3sas0-msix44: PCI-MSI-X enabled: IRQ 129
[ 1716.258404] mpt3sas0-msix45: PCI-MSI-X enabled: IRQ 130
[ 1716.258405] mpt3sas0-msix46: PCI-MSI-X enabled: IRQ 131
[ 1716.258407] mpt3sas0-msix47: PCI-MSI-X enabled: IRQ 132
[ 1716.258408] mpt3sas0-msix48: PCI-MSI-X enabled: IRQ 133
[ 1716.258410] mpt3sas0-msix49: PCI-MSI-X enabled: IRQ 134
[ 1716.258411] mpt3sas0-msix50: PCI-MSI-X enabled: IRQ 135
[ 1716.258413] mpt3sas0-msix51: PCI-MSI-X enabled: IRQ 136
[ 1716.258414] mpt3sas0-msix52: PCI-MSI-X enabled: IRQ 137
[ 1716.258416] mpt3sas0-msix53: PCI-MSI-X enabled: IRQ 138
[ 1716.258417] mpt3sas0-msix54: PCI-MSI-X enabled: IRQ 139
[ 1716.258419] mpt3sas0-msix55: PCI-MSI-X enabled: IRQ 140
[ 1716.258420] mpt3sas0-msix56: PCI-MSI-X enabled: IRQ 141
[ 1716.258421] mpt3sas0-msix57: PCI-MSI-X enabled: IRQ 142
[ 1716.258422] mpt3sas0-msix58: PCI-MSI-X enabled: IRQ 143
[ 1716.258423] mpt3sas0-msix59: PCI-MSI-X enabled: IRQ 144
[ 1716.258424] mpt3sas0-msix60: PCI-MSI-X enabled: IRQ 145
[ 1716.258425] mpt3sas0-msix61: PCI-MSI-X enabled: IRQ 146
[ 1716.258426] mpt3sas0-msix62: PCI-MSI-X enabled: IRQ 147
[ 1716.258427] mpt3sas0-msix63: PCI-MSI-X enabled: IRQ 148
[ 1716.258429] mpt3sas_cm0: iomem(0x000620c080100000),
mapped(0x000000004a4e4187), size(65536)
[ 1716.258431] mpt3sas_cm0: ioport(0x0000000000000000), size(0)
[ 1716.308411] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default
host page size to 4k
[ 1716.454968] mpt3sas_cm0: _base_display_fwpkg_version: complete
[ 1716.455303] mpt3sas_cm0: LSISAS3224: FWVersion(09.00.100.00),
ChipRevision(0x01), BiosVersion(00.00.00.00)
[ 1716.455305] mpt3sas_cm0: Protocol=(
[ 1716.455305] Initiator
[ 1716.455306] ,Target
[ 1716.455307] ),
[ 1716.455308] Capabilities=(
[ 1716.455309] TLR
[ 1716.455310] ,EEDP
[ 1716.455311] ,Snapshot Buffer
[ 1716.455312] ,Diag Trace Buffer
[ 1716.455313] ,Task Set Full
[ 1716.455313] ,NCQ
[ 1716.455314] )
[ 1716.455372] mpt3sas_cm0: sending port enable !!
[ 1727.531265] mpt3sas_cm0: port enable: SUCCESS
[ 1727.531452] mpt3sas_cm0: search for end-devices: start
[ 1727.534191] scsi target0:0:0: handle(0x0019),
sas_addr(0x4433221100000000)
[ 1727.534195] scsi target0:0:0: enclosure logical
id(0x500062b203842300), slot(3)
[ 1727.534281] scsi target0:0:1: handle(0x001a),
sas_addr(0x4433221103000000)
[ 1727.534283] scsi target0:0:1: enclosure logical
id(0x500062b203842300), slot(1)
[ 1727.534371] scsi target0:0:3: handle(0x001b),
sas_addr(0x4433221105000000)
[ 1727.534374] scsi target0:0:3: enclosure logical
id(0x500062b203842300), slot(6)
[ 1727.534461] scsi target0:0:2: handle(0x001c),
sas_addr(0x4433221104000000)
[ 1727.534463] scsi target0:0:2: enclosure logical
id(0x500062b203842300), slot(7)
[ 1727.534552] scsi target0:0:5: handle(0x001d),
sas_addr(0x4433221107000000)
[ 1727.534553] scsi target0:0:5: enclosure logical
id(0x500062b203842300), slot(5)
[ 1727.534641] scsi target0:0:4: handle(0x001e),
sas_addr(0x4433221106000000)
[ 1727.534643] scsi target0:0:4: enclosure logical
id(0x500062b203842300), slot(4)
[ 1727.534733] scsi target0:0:6: handle(0x001f),
sas_addr(0x4433221110000000)
[ 1727.534735] scsi target0:0:6: enclosure logical
id(0x500062b203842300), slot(11)
[ 1727.534823] scsi target0:0:7: handle(0x0020),
sas_addr(0x4433221111000000)
[ 1727.534825] scsi target0:0:7: enclosure logical
id(0x500062b203842300), slot(10)
[ 1727.534915] scsi target0:0:9: handle(0x0021),
sas_addr(0x4433221113000000)
[ 1727.534918] scsi target0:0:9: enclosure logical
id(0x500062b203842300), slot(9)
[ 1727.535004] scsi target0:0:8: handle(0x0022),
sas_addr(0x4433221112000000)
[ 1727.535006] scsi target0:0:8: enclosure logical
id(0x500062b203842300), slot(8)
[ 1727.535588] mpt3sas_cm0: search for end-devices: complete
[ 1727.535589] mpt3sas_cm0: search for end-devices: start
[ 1727.535590] mpt3sas_cm0: search for PCIe end-devices: complete
[ 1727.535591] mpt3sas_cm0: search for expanders: start
[ 1727.535592] mpt3sas_cm0: search for expanders: complete
[ 1727.535598] mpt3sas_cm0: hard reset: success
[ 1727.535601] EEH: PE#fd (PCI 0031:01:00.0): mpt3sas driver reports:
'recovered'
[ 1727.535602] EEH: Finished:'slot_reset' with aggregate recovery
state:'recovered'
[ 1727.535604] EEH: Notify device driver to resume
[ 1727.535606] EEH: Beginning: 'resume'
[ 1727.535608] EEH: PE#fd (PCI 0031:01:00.0): Invoking mpt3sas->resume()
[ 1727.535610] mpt3sas_cm0: PCI error: resume callback!!
[ 1727.536115] mpt3sas_cm0: removing unresponding devices: start
[ 1727.536118] mpt3sas_cm0: removing unresponding devices: end-devices
[ 1727.536121] mpt3sas_cm0: Removing unresponding devices: pcie end-devices
[ 1727.536123] mpt3sas_cm0: removing unresponding devices: expanders
[ 1727.536124] mpt3sas_cm0: removing unresponding devices: complete
[ 1727.536132] mpt3sas_cm0: scan devices: start
[ 1727.536357] EEH: PE#fd (PCI 0031:01:00.0): mpt3sas driver reports: 'none'
[ 1727.536359] EEH: Finished:'resume'
[ 1727.536360] EEH: Recovery successful.
[ 1727.551703] mpt3sas_cm0: scan devices: expanders start
[ 1727.552312] mpt3sas_cm0: break from expander scan:
ioc_status(0x0022), loginfo(0x310f0400)
[ 1727.552313] mpt3sas_cm0: scan devices: expanders complete
[ 1727.552314] mpt3sas_cm0: scan devices: end devices start
[ 1727.561673] mpt3sas_cm0: break from end device scan:
ioc_status(0x0022), loginfo(0x310f0400)
[ 1727.561675] mpt3sas_cm0: scan devices: end devices complete
[ 1727.561676] mpt3sas_cm0: scan devices: pcie end devices start
[ 1727.561706] mpt3sas_cm0: log_info(0x3003011d): originator(IOP),
code(0x03), sub_code(0x011d)
[ 1727.561755] mpt3sas_cm0: log_info(0x3003011d): originator(IOP),
code(0x03), sub_code(0x011d)
[ 1727.561759] mpt3sas_cm0: break from pcie end device scan:
ioc_status(0x0021), loginfo(0x3003011d)
[ 1727.561760] mpt3sas_cm0: pcie devices: pcie end devices complete
[ 1727.561761] mpt3sas_cm0: scan devices: complete
[ 1728.031230] sd 0:0:0:0: Power-on or device reset occurred
[ 1728.031235] sd 0:0:1:0: Power-on or device reset occurred
[ 1728.031236] sd 0:0:4:0: Power-on or device reset occurred
[ 1728.031238] sd 0:0:2:0: Power-on or device reset occurred
[ 1728.031240] sd 0:0:7:0: Power-on or device reset occurred
[ 1728.031243] sd 0:0:6:0: Power-on or device reset occurred
[ 1728.031245] sd 0:0:8:0: Power-on or device reset occurred
[ 1728.031260] sd 0:0:9:0: Power-on or device reset occurred
[ 1728.031719] sd 0:0:5:0: Power-on or device reset occurred
[ 1728.819966] mpt3sas_cm0: log_info(0x31120320): originator(PL),
code(0x12), sub_code(0x0320)
[ 1729.281480] sd 0:0:8:0: Power-on or device reset occurred
[ 1729.443974] sd 0:0:3:0: Power-on or device reset occurred
[ 1730.031108] sd 0:0:8:0: Power-on or device reset occurred
On 11/18/18 7:52 PM, Matt Corallo wrote:
(not subscribed to lists, please keep me on CC)
When upgrading from 4.17.2 to 4.19.2, my 9305-16i started faulting on
load (well, not so much load given its all spinning disks, but once
every hour or five during a btrfs rebalance). Mostly the disks would
all come back online and it would just result in 30 seconds of disks
offline, but occasionally it would fall offline completely. dmesg is
below, though its not so useful. Downgrading to 4.17.2 again fixed the
issue completely.
[19983.155887] mpt3sas_cm0: fault_state(0x5853)!
[19983.155932] mpt3sas_cm0: sending diag reset !!
[19984.093087] mpt3sas_cm0: diag reset: SUCCESS
[19984.155056] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default
host page size to 4k
[19984.301067] mpt3sas_cm0: _base_display_fwpkg_version: complete
[19984.301400] mpt3sas_cm0: LSISAS3224: FWVersion(09.00.100.00),
ChipRevision(0x01), BiosVersion(00.00.00.00)
[19984.301403] mpt3sas_cm0: Protocol=(
[19984.301404] Initiator
[19984.301405] ,Target
[19984.301406] ),
[19984.301407] Capabilities=(
[19984.301408] TLR
[19984.301410] ,EEDP
[19984.301411] ,Snapshot Buffer
[19984.301412] ,Diag Trace Buffer
[19984.301413] ,Task Set Full
[19984.301414] ,NCQ
[19984.301415] )
[19984.301473] mpt3sas_cm0: sending port enable !!
[19995.149962] mpt3sas_cm0: port enable: SUCCESS
[19995.150077] mpt3sas_cm0: search for end-devices: start
[19995.151143] scsi target0:0:0: handle(0x0019),
sas_addr(0x4433221100000000)
[19995.151147] scsi target0:0:0: enclosure logical
id(0x500062b203842300), slot(3)
[19995.151196] scsi target0:0:1: handle(0x001a),
sas_addr(0x4433221103000000)
[19995.151199] scsi target0:0:1: enclosure logical
id(0x500062b203842300), slot(1)
[19995.151245] scsi target0:0:3: handle(0x001b),
sas_addr(0x4433221105000000)
[19995.151247] scsi target0:0:3: enclosure logical
id(0x500062b203842300), slot(6)
[19995.151293] scsi target0:0:2: handle(0x001c),
sas_addr(0x4433221104000000)
[19995.151296] scsi target0:0:2: enclosure logical
id(0x500062b203842300), slot(7)
[19995.151342] scsi target0:0:4: handle(0x001d),
sas_addr(0x4433221106000000)
[19995.151345] scsi target0:0:4: enclosure logical
id(0x500062b203842300), slot(4)
[19995.151391] scsi target0:0:6: handle(0x001e),
sas_addr(0x4433221110000000)
[19995.151393] scsi target0:0:6: enclosure logical
id(0x500062b203842300), slot(11)
[19995.151439] scsi target0:0:7: handle(0x001f),
sas_addr(0x4433221111000000)
[19995.151441] scsi target0:0:7: enclosure logical
id(0x500062b203842300), slot(10)
[19995.151487] scsi target0:0:9: handle(0x0020),
sas_addr(0x4433221113000000)
[19995.151490] scsi target0:0:9: enclosure logical
id(0x500062b203842300), slot(9)
[19995.151535] scsi target0:0:10: handle(0x0021),
sas_addr(0x4433221112000000)
[19995.151537] scsi target0:0:10: enclosure logical
id(0x500062b203842300), slot(8)
[19995.151607] mpt3sas_cm0: search for end-devices: complete
[19995.151609] mpt3sas_cm0: search for end-devices: start
[19995.151611] mpt3sas_cm0: search for PCIe end-devices: complete
[19995.151613] mpt3sas_cm0: search for expanders: start
[19995.151614] mpt3sas_cm0: search for expanders: complete
[19995.151624] mpt3sas_cm0: _base_fault_reset_work: hard reset: success
[19995.151630] mpt3sas_cm0: removing unresponding devices: start
[19995.151631] mpt3sas_cm0: removing unresponding devices: end-devices
[19995.151633] mpt3sas_cm0: Removing unresponding devices: pcie
end-devices
[19995.151635] mpt3sas_cm0: removing unresponding devices: expanders
[19995.151636] mpt3sas_cm0: removing unresponding devices: complete
[19995.151642] mpt3sas_cm0: scan devices: start
[19995.152075] mpt3sas_cm0: scan devices: expanders start
[19995.152139] mpt3sas_cm0: break from expander scan:
ioc_status(0x0022), loginfo(0x310f0400)
[19995.152141] mpt3sas_cm0: scan devices: expanders complete
[19995.152142] mpt3sas_cm0: scan devices: end devices start
[19995.156007] mpt3sas_cm0: break from end device scan:
ioc_status(0x0022), loginfo(0x310f0400)
[19995.156009] mpt3sas_cm0: scan devices: end devices complete
[19995.156010] mpt3sas_cm0: scan devices: pcie end devices start
[19995.156028] mpt3sas_cm0: log_info(0x3003011d): originator(IOP),
code(0x03), sub_code(0x011d)
[19995.156047] mpt3sas_cm0: log_info(0x3003011d): originator(IOP),
code(0x03), sub_code(0x011d)
[19995.156053] mpt3sas_cm0: break from pcie end device scan:
ioc_status(0x0021), loginfo(0x3003011d)
[19995.156054] mpt3sas_cm0: pcie devices: pcie end devices complete
[19995.156055] mpt3sas_cm0: scan devices: complete
[19995.650024] sd 0:0:0:0: Power-on or device reset occurred
[19995.650052] sd 0:0:6:0: Power-on or device reset occurred
[19996.239565] sd 0:0:10:0: Power-on or device reset occurred
[19996.341924] sd 0:0:9:0: Power-on or device reset occurred
[19996.650155] sd 0:0:1:0: Power-on or device reset occurred
[19996.650184] sd 0:0:3:0: Power-on or device reset occurred
[19996.650197] sd 0:0:2:0: Power-on or device reset occurred
[19996.650205] sd 0:0:4:0: Power-on or device reset occurred
[19996.650214] sd 0:0:7:0: Power-on or device reset occurred
