------- Comment From lakun...@in.ibm.com 2017-08-24 14:43 EDT------- hmc : daan1.isst.aus.stglabs.ibm.com has address 9.3.48.91 (hscroot/abc123) CEC : rose lpar : roselp2.isst.aus.stglabs.ibm.com has address 10.33.23.15 (root/don2rry)
to access the console of lpar rmvterm -m rose -p roselp2 mkvterm -m rose -p roselp2 lpar is at xmon and ready to debug -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1661684 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04.2: drop in xmon when running dlpar tests under stress Status in The Ubuntu-power-systems project: Opinion Status in linux package in Ubuntu: Incomplete Bug description: == Comment: #0 - Ping Tian Han <pt...@cn.ibm.com> - 2016-12-26 21:59:52 == ---Problem Description--- When testing DLPAR, include slot/cpu/mem, under stress on roselp4, system dropped into xmon: roselp4 login: [ 95.511790] sysrq: SysRq : Changing Loglevel [ 95.511816] sysrq: Loglevel set to 9 [ 289.363833] mlx4_en 0292:60:00.0: removed PHC [ 293.123896] iommu: Removing device 0292:60:00.0 from group 3 [ 303.173744] pci_bus 0292:60: busn_res: [bus 60-ff] is released [ 303.173865] rpadlpar_io: slot PHB 658 removed [ 335.853779] iommu: Removing device 0021:01:00.0 from group 0 [ 345.893764] pci_bus 0021:01: busn_res: [bus 01-ff] is released [ 345.893869] rpadlpar_io: slot PHB 33 removed [ 382.204003] min_free_kbytes is not updated to 16885 because user defined value 551564 is preferred [ 446.143648] cpu 152 (hwid 152) Ready to die... [ 446.464057] cpu 153 (hwid 153) Ready to die... [ 446.473525] cpu 154 (hwid 154) Ready to die... [ 446.474077] cpu 155 (hwid 155) Ready to die... [ 446.483529] cpu 156 (hwid 156) Ready to die... [ 446.493532] cpu 157 (hwid 157) Ready to die... [ 446.494078] cpu 158 (hwid 158) Ready to die... [ 446.503527] cpu 159 (hwid 159) Ready to die... [ 446.664534] cpu 144 (hwid 144) Ready to die... [ 446.964113] cpu 145 (hwid 145) Ready to die... [ 446.973525] cpu 146 (hwid 146) Ready to die... [ 446.974094] cpu 147 (hwid 147) Ready to die... [ 446.983944] cpu 148 (hwid 148) Ready to die... [ 446.984062] cpu 149 (hwid 149) Ready to die... [ 446.993518] cpu 150 (hwid 150) Ready to die... [ 446.993543] Querying DEAD? cpu 150 (150) shows 2 [ 446.994098] cpu 151 (hwid 151) Ready to die... [ 447.133726] cpu 136 (hwid 136) Ready to die... [ 447.403532] cpu 137 (hwid 137) Ready to die... [ 447.403772] cpu 138 (hwid 138) Ready to die... [ 447.403839] cpu 139 (hwid 139) Ready to die... [ 447.403887] cpu 140 (hwid 140) Ready to die... [ 447.403937] cpu 141 (hwid 141) Ready to die... [ 447.403979] cpu 142 (hwid 142) Ready to die... [ 447.404038] cpu 143 (hwid 143) Ready to die... [ 447.513546] cpu 128 (hwid 128) Ready to die... [ 447.693533] cpu 129 (hwid 129) Ready to die... [ 447.693999] cpu 130 (hwid 130) Ready to die... [ 447.703530] cpu 131 (hwid 131) Ready to die... [ 447.704087] Querying DEAD? cpu 132 (132) shows 2 [ 447.704102] cpu 132 (hwid 132) Ready to die... [ 447.713534] cpu 133 (hwid 133) Ready to die... [ 447.714064] Querying DEAD? cpu 134 (134) shows 2 cpu 0x86: Vector: 300 (Data Access) at [c000000007b0fd40] pc: 000000001ec3072c lr: 000000001ec2fee0 sp: 1faf6bd0 msr: 8000000102801000 dar: 212d6c1a2a20c dsisr: 42000000 current = 0xc000000474c6d600 paca = 0xc000000007b6b600 softe: 0 irq_happened: 0x01 pid = 0, comm = swapper/134 Linux version 4.8.0-34-generic (buildd@bos01-ppc64el-026) (gcc version 5.4.0 20160609 (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.4) ) #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 (Ubuntu 4.8.0-34.36~16.04.1-generic 4.8.11) WARNING: exception is not recoverable, can't continue enter ? for help SP (1faf6bd0) is in userspace 86:mon> 86:mon> t SP (1faf6bd0) is in userspace 86:mon> r R00 = 000212d6c1a2a20f R16 = c000000000ff1c38 R01 = 000000001faf6bd0 R17 = c000000474c9c080 R02 = 000000001ed1be80 R18 = c000000474c9c000 R03 = 000000001faf6c80 R19 = c0000000013fdf08 R04 = 0000000000000018 R20 = c000000474c9c080 R05 = 00000000000000e0 R21 = c0000000013e8ad0 R06 = 0000000000009e04 R22 = c000000474c9c000 R07 = 000000001faf6d30 R23 = c00000047a9a1c40 R08 = 000000001faf6d28 R24 = 0000000000000002 R09 = 000212d6c1a2a20c R25 = c000000000fd4e6c R10 = 000000001ec1b118 R26 = c000000000fd4e6c R11 = 000000001ee7e040 R27 = c0000000014daae0 R12 = 000000000163c1d8 R28 = 0000000000000000 R13 = c000000007b6b600 R29 = 0000000000000086 R14 = c0000000014defb0 R30 = c000000000fd4e68 R15 = 0000000000000001 R31 = 000000001faf6bd0 pc = 000000001ec3072c cfar= 000000001ec2fedc lr = 000000001ec2fee0 msr = 8000000102801000 cr = 42000000 ctr = 000000001ec48788 xer = 0000000000000020 trap = 300 dar = 000212d6c1a2a20c dsisr = 42000000 86:mon> Contact Information = Ping Tian Han/pt...@cn.ibm.com ---uname output--- Linux roselp4 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux Machine Type = lpar ---Debugger Data--- cpu 0x86: Vector: 300 (Data Access) at [c000000007b0fd40] pc: 000000001ec3072c lr: 000000001ec2fee0 sp: 1faf6bd0 msr: 8000000102801000 dar: 212d6c1a2a20c dsisr: 42000000 current = 0xc000000474c6d600 paca = 0xc000000007b6b600 softe: 0 irq_happened: 0x01 pid = 0, comm = swapper/134 Linux version 4.8.0-34-generic (buildd@bos01-ppc64el-026) (gcc version 5.4.0 20160609 (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.4) ) #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 (Ubuntu 4.8.0-34.36~16.04.1-generic 4.8.11) WARNING: exception is not recoverable, can't continue enter ? for help SP (1faf6bd0) is in userspace 86:mon> e cpu 0x86: Vector: 300 (Data Access) at [c000000007b0fd40] pc: 000000001ec3072c lr: 000000001ec2fee0 sp: 1faf6bd0 msr: 8000000102801000 dar: 212d6c1a2a20c dsisr: 42000000 current = 0xc000000474c6d600 paca = 0xc000000007b6b600 softe: 0 irq_happened: 0x01 pid = 0, comm = swapper/134 Linux version 4.8.0-34-generic (buildd@bos01-ppc64el-026) (gcc version 5.4.0 20160609 (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.4) ) #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 (Ubuntu 4.8.0-34.36~16.04.1-generic 4.8.11) 86:mon> t SP (1faf6bd0) is in userspace 86:mon> r R00 = 000212d6c1a2a20f R16 = c000000000ff1c38 R01 = 000000001faf6bd0 R17 = c000000474c9c080 R02 = 000000001ed1be80 R18 = c000000474c9c000 R03 = 000000001faf6c80 R19 = c0000000013fdf08 R04 = 0000000000000018 R20 = c000000474c9c080 R05 = 00000000000000e0 R21 = c0000000013e8ad0 R06 = 0000000000009e04 R22 = c000000474c9c000 R07 = 000000001faf6d30 R23 = c00000047a9a1c40 R08 = 000000001faf6d28 R24 = 0000000000000002 R09 = 000212d6c1a2a20c R25 = c000000000fd4e6c R10 = 000000001ec1b118 R26 = c000000000fd4e6c R11 = 000000001ee7e040 R27 = c0000000014daae0 R12 = 000000000163c1d8 R28 = 0000000000000000 R13 = c000000007b6b600 R29 = 0000000000000086 R14 = c0000000014defb0 R30 = c000000000fd4e68 R15 = 0000000000000001 R31 = 000000001faf6bd0 pc = 000000001ec3072c cfar= 000000001ec2fedc lr = 000000001ec2fee0 msr = 8000000102801000 cr = 42000000 ctr = 000000001ec48788 xer = 0000000000000020 trap = 300 dar = 000212d6c1a2a20c dsisr = 42000000 86:mon> ---System Hang--- drop into xmon ---Steps to Reproduce--- 1. run IO stress tests on roselp4 2. run slot/cpu/mem dlpar tests on roselp4 Stack trace output: no Oops output: no System Dump Info: The system was configured to capture a dump, however a dump was not produced. *Additional Instructions for Ping Tian Han/pt...@cn.ibm.com: -Post a private note with access information to the machine that is currently in the debugger. -Attach sysctl -a output output to the bug. == Comment: #4 - PAWAN K. SINGH <pawak...@in.ibm.com> - 2016-12-27 02:19:58 == == Comment: #7 - Ping Tian Han <pt...@cn.ibm.com> - 2016-12-27 20:53:59 == == Comment: #8 - Ping Tian Han <pt...@cn.ibm.com> - 2016-12-27 20:59:04 == == Comment: #14 - Ping Tian Han <pt...@cn.ibm.com> - 2016-12-28 03:17:50 == FYI. With default min_free_kbytes, roselp4 still drops into xmon: Ubuntu 16.04.1 LTS roselp4 hvc0 roselp4 login: [ 260.094141] sysrq: SysRq : Changing Loglevel [ 260.094161] sysrq: Loglevel set to 9 [ 266.614273] cpu 152 (hwid 152) Ready to die... [ 266.794136] cpu 153 (hwid 153) Ready to die... [ 266.794694] cpu 154 (hwid 154) Ready to die... [ 266.804248] cpu 155 (hwid 155) Ready to die... [ 266.804302] cpu 156 (hwid 156) Ready to die... [ 266.804354] cpu 157 (hwid 157) Ready to die... [ 266.804410] cpu 158 (hwid 158) Ready to die... [ 266.804465] cpu 159 (hwid 159) Ready to die... [ 266.935065] cpu 144 (hwid 144) Ready to die... [ 267.144140] cpu 145 (hwid 145) Ready to die... [ 267.144683] cpu 146 (hwid 146) Ready to die... [ 267.154692] cpu 147 (hwid 147) Ready to die... [ 267.164134] cpu 148 (hwid 148) Ready to die... [ 267.164702] cpu 149 (hwid 149) Ready to die... [ 267.174819] cpu 150 (hwid 150) Ready to die... [ 267.184684] cpu 151 (hwid 151) Ready to die... [ 267.324831] cpu 136 (hwid 136) Ready to die... [ 267.614138] cpu 137 (hwid 137) Ready to die... [ 267.614745] cpu 138 (hwid 138) Ready to die... [ 267.624135] cpu 139 (hwid 139) Ready to die... [ 267.624716] cpu 140 (hwid 140) Ready to die... [ 267.634637] Querying DEAD? cpu 141 (141) shows 2 cpu 0x8d: Vector: 300 (Data Access) at [c000000007ad7d40] pc: 000000001ec26be0 lr: 000000001ec26ab4 sp: 1faf6920 msr: 8000000102801000 dar: fffffe801faf6bc0 dsisr: 40000000 current = 0xc000000474c51e00 paca = 0xc000000007b6f500 softe: 0 irq_happened: 0x01 pid = 0, comm = swapper/141 Linux version 4.8.0-34-generic (buildd@bos01-ppc64el-026) (gcc version 5.4.0 20160609 (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.4) ) #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 (Ubuntu 4.8.0-34.36~16.04.1-generic 4.8.11) WARNING: exception is not recoverable, can't continue enter ? for help SP (1faf6920) is in userspace 8d:mon> cpu 0x8e: Vector: 300 (Data Access) at [c000000007acfd40] pc: 000000001ec22614 lr: 000000001ec22d5c sp: 1faf6b00 msr: 8000000102801000 dar: 20000000 dsisr: 40000000 current = 0xc000000474c7c800 paca = 0xc000000007b6fe00 softe: 0 irq_happened: 0x01 pid = 0, comm = swapper/142 Linux version 4.8.0-34-generic (buildd@bos01-ppc64el-026) (gcc version 5.4.0 20160609 (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.4) ) #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 (Ubuntu 4.8.0-34.36~16.04.1-generic 4.8.11) WARNING: exception is not recoverable, can't continue 8d:mon> Unrecognized command: \x1be (type ? for help) 8d:mon> e cpu 0x8d: Vector: 300 (Data Access) at [c000000007ad7d40] pc: 000000001ec26be0 lr: 000000001ec26ab4 sp: 1faf6920 msr: 8000000102801000 dar: fffffe801faf6bc0 dsisr: 40000000 current = 0xc000000474c51e00 paca = 0xc000000007b6f500 softe: 0 irq_happened: 0x01 pid = 0, comm = swapper/141 Linux version 4.8.0-34-generic (buildd@bos01-ppc64el-026) (gcc version 5.4.0 20160609 (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.4) ) #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 (Ubuntu 4.8.0-34.36~16.04.1-generic 4.8.11) 8d:mon> t SP (1faf6920) is in userspace 8d:mon> == Comment: #15 - Ping Tian Han <pt...@cn.ibm.com> - 2016-12-28 03:22:22 == == Comment: #19 - Ping Tian Han <pt...@cn.ibm.com> - 2016-12-29 00:13:39 == == Comment: #20 - Kevin W. Rudd <ru...@us.ibm.com> - 2016-12-29 12:49:15 == Nathan or Laurent, In the dmesg output, I'm seeing similar behavior to the problem reported in Bug 146931. The following error and completely bogus NIP/LR values appear to be the same scenario: pseries-hotplug-cpu: Failed to release drc (10000098) for CPU PowerPC,POWER8, rc: -17 The NIP and LR values appear to be completely bogus, so I'm not sure what about the Bug 146931 scenario matched the issue being tracked in Bug 146776. This looks to be a side issue of doing hotplugging on the CPUs Please review and provide your thoughts on this observed behavior. Thanks. == Comment: #25 - Nathan D. Fontenot <nfont...@us.ibm.com> - 2017-01-13 13:32:51 == My first thought in looking at this is that it appears that the swapper thread for a cpu is scheduled to run on a cpu that has been removed. This may explain the bogus pc and lr values. There have been a lot of updates to the generic kernel cpu hotplug code recently, perhaps some update there could be causing this. It would be interesting to see if this occurs on older kernels. As for the rtas set-indicator call returning -17, I don't know how that is possible. A return value of -17 is not even a defined return value in the PAPR. This could be a side effect of what is causing the crash though so that should be resolved first and then see if this still occurs. == Comment: #31 - Fernando Seiti Furusato <ferse...@br.ibm.com> - 2017-02-02 11:37:37 == Mirroring so Canonical is aware of this bug. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1661684/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
