Default Comment by Bridge ** Attachment added: "Kdump console logs" https://bugs.launchpad.net/bugs/1751835/+attachment/5063610/+files/kdump_failure_on_proposed_4.15.txt
** Changed in: ubuntu Assignee: (unassigned) => Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) ** Package changed: ubuntu => kexec-tools (Ubuntu) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1751835 Title: Ubuntu 18.04 kdump does not work on bionic-proposed 4.15 ppc64el with AC922 Status in The Ubuntu-power-systems project: New Status in kexec-tools package in Ubuntu: New Bug description: no guest was running, only Ubuntu 18.04 Host installed and trying kdump only on the host =========== tried on the ubuntu released proposed kernel 4.15.0-10-generic and kdump process was able to proceed (did not dump complete vmcore though) but hit issue, please check below console log ======== root@ltciofvtr-spoon4:~# dmesg | grep -i crash [ 0.000000] Reserving 6144MB of memory at 128MB for crashkernel (System RAM: 524288MB) [ 0.000000] Kernel command line: root=UUID=749429f9-83b9-4776-9a3e-147e297cdf99 ro quiet splash crashkernel=6144M root@ltciofvtr-spoon4:~# dmesg | grep -i reserv [ 0.000000] Reserving 6144MB of memory at 128MB for crashkernel (System RAM: 524288MB) [ 0.000000] cma: Reserved 26224 MiB at 0x0000203995000000 [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] Memory: 502952192K/536870912K available (13376K kernel code, 2048K rwdata, 3648K rodata, 4800K init, 3037K bss, 7065344K reserved, 26853376K cma-reserved) root@ltciofvtr-spoon4:~# uname -a Linux ltciofvtr-spoon4 4.15.0-10-generic #11-Ubuntu SMP Tue Feb 13 18:21:52 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux root@ltciofvtr-spoon4:~# kdump kdump kdump-config root@ltciofvtr-spoon4:~# kdump kdump kdump-config root@ltciofvtr-spoon4:~# kdump-config Usage: /usr/sbin/kdump-config {help|test|show|status|load|unload|savecore|propagate|symlinks kernel-version} root@ltciofvtr-spoon4:~# kdump-config status current state : ready to kdump root@ltciofvtr-spoon4:~# echo 1 > /proc/sys/kernel/sysrq root@ltciofvtr-spoon4:~# echo c > /proc/sysrq-trigger [ 164.604204] sysrq: SysRq : Trigger a crash [ 164.604264] Unable to handle kernel paging request for data at address 0x00000000 [ 164.604395] Faulting instruction address: 0xc0000000007ea268 [ 164.604481] Oops: Kernel access of bad area, sig: 11 [#1] [ 164.604559] LE SMP NR_CPUS=2048 NUMA PowerNV [ 164.604628] Modules linked in: idt_89hpesx ofpart ipmi_powernv ipmi_devintf ipmi_msghandler ibmpowernv cmdlinepart vmx_crypto powernv_flash at24 uio_pdrv_genirq uio mtd opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 mlx5_ib ib_core mlx5_core bnx2x ast i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mdio ahci mlxfw libcrc32c crc32c_vpmsum drm tg3 libahci devlink [ 164.605176] CPU: 72 PID: 3444 Comm: bash Not tainted 4.15.0-10-generic #11-Ubuntu [ 164.605305] NIP: c0000000007ea268 LR: c0000000007eb1a8 CTR: c0000000007ea240 [ 164.605413] REGS: c0002038f8c2f9f0 TRAP: 0300 Not tainted (4.15.0-10-generic) [ 164.605496] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28222222 XER: 20040000 [ 164.605641] CFAR: c0000000007eb1a4 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1 [ 164.605641] GPR00: c0000000007eb1a8 c0002038f8c2fc70 c0000000016ea600 0000000000000063 [ 164.605641] GPR04: c000203993d0ce18 c000203993d24368 9000000000009033 0000000000000808 [ 164.605641] GPR08: 0000000000000007 0000000000000001 0000000000000000 9000000000001003 [ 164.605641] GPR12: c0000000007ea240 c000000007a51800 000008f7fbb8e008 0000000000000000 [ 164.605641] GPR16: 000008f7fb19e9f0 000008f7fb231998 000008f7fb2319d0 000008f7fb268204 [ 164.605641] GPR20: 0000000000000000 0000000000000001 0000000000000000 00007fffed1e1b54 [ 164.605641] GPR24: 00007fffed1e1b50 000008f7fb26afc4 c0000000015e9930 0000000000000002 [ 164.605641] GPR28: 0000000000000063 0000000000000004 c000000001572a9c c0000000015e9cd0 [ 164.606714] NIP [c0000000007ea268] sysrq_handle_crash+0x28/0x30 [ 164.606796] LR [c0000000007eb1a8] __handle_sysrq+0xf8/0x2c0 [ 164.606865] Call Trace: [ 164.606919] [c0002038f8c2fc70] [c0000000007eb188] __handle_sysrq+0xd8/0x2c0 (unreliable) [ 164.607043] [c0002038f8c2fd10] [c0000000007eb9b4] write_sysrq_trigger+0x64/0x90 [ 164.607152] [c0002038f8c2fd40] [c00000000047c548] proc_reg_write+0x88/0xd0 [ 164.607252] [c0002038f8c2fd70] [c0000000003cf8dc] __vfs_write+0x3c/0x70 [ 164.607347] [c0002038f8c2fd90] [c0000000003cfb38] vfs_write+0xd8/0x220 [ 164.607445] [c0002038f8c2fde0] [c0000000003cfe58] SyS_write+0x68/0x110 [ 164.607545] [c0002038f8c2fe30] [c00000000000b184] system_call+0x58/0x6c [ 164.607640] Instruction dump: [ 164.607689] 4bfff9f1 4bfffe50 3c4c00f0 384203c0 7c0802a6 60000000 39200001 3d42001c [ 164.607805] 394a76b0 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00f0 38420390 [ 164.607925] ---[ end trace d9ded5212faa751b ]--- [ 165.612963] [ 165.613070] Sending IPI to other CPUs [ 165.623168] IPI complete [ 165.627790] kexec: waiting for cpu 1 (physical 1) to enter OPAL [ 165.629875] kexec: waiting fo[ 419.363342082,5] OPAL: Switch to big-endian OS r cpu 5 (physical 5) to enter[ 423.007803159,5] OPAL: Switch to little-endian OS OPAL [ 165.635949] kexec: waiting for cpu 7 (physical 7) to enter OPAL [ 167.322686] kexec: Starting switchover sequence. [ 1.245751] integrity: Unable to open file: /etc/keys/x509_ima.der (-2) [ 1.245755] integrity: Unable to open file: /etc/keys/x509_evm.der (-2) [ 1.314527] vio vio: uevent: failed to send synthetic uevent /dev/sda2: recovering journal /dev/sda2: clean, 134971/61054976 files, 5545559/244188416 blocks [ 6.725980] vio vio: uevent: failed to send synthetic uevent [ OK ] Started Show Plymouth Boot Screen. plymouth-start.service [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Forward Password Requests to Plymouth Directory Watch. [ OK ] Started Network Time Synchronization. systemd-timesyncd.service [ OK ] Reached target System Time Synchronized. [ OK ] Started AppArmor initialization. apparmor.service [ OK ] Reached target System Initialization. [ OK ] Started Wait for Network to be Configured. systemd-networkd-wait-online.service [ OK ] Reached target Network is Online. Starting Kernel crash dump capture service... [ 24.498357] kdump-tools[1424]: Starting kdump-tools: * running makedumpfile -c -d 31 /proc/vmcore /var/crash/201802160010/dump-incomplete [ 41.042895] Fatal Hypervisor Maintenance interrupt [Not recovered] [ 41.042962] Error detail: Malfunction Alert [ 41.043003] HMER: 8040000000000000 [ 41.043060] Unknown Malfunction Alert of type 0 [ 41.043113] opal: Hardware platform error: Unrecoverable HMI exception [ 41.849182] WARNING: CPU: 2 PID: 55 at /build/linux-jWa1Fv/linux-4.15.0/kernel/sched/core.c:1188 set_task_cpu+0x240/0x250 [ 41.849297] Modules linked in: mlx5_ib ibmpowernv vmx_crypto ipmi_powernv ofpart crct10dif_vpmsum ipmi_devintf ipmi_msghandler cmdlinepart powernv_flash opal_prd mtd idt_89hpesx ib_core bnx2x at24 mdio mlx5_core mlxfw libcrc32c devlink tg3 sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum ast i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci uio_pdrv_genirq drm uio libahci [ 41.849810] CPU: 2 PID: 55 Comm: kworker/2:1 Not tainted 4.15.0-10-generic #11-Ubuntu [ 41.849945] Workqueue: events hmi_event_handler [ 41.850025] NIP: c000000008149aa0 LR: c00000000814a6cc CTR: c000000008156600 [ 41.850154] REGS: c00000016feeeeb0 TRAP: 0700 Not tainted (4.15.0-10-generic) [ 41.850270] MSR: 900000000282b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28002244 XER: 00000000 [ 41.850408] CFAR: c00000000814990c SOFTE: 0 [ 41.850408] GPR00: c00000000814a6cc c00000016feef130 c0000000096ea600 c00000016f432b00 [ 41.850408] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 41.850408] GPR08: c000000009721ed8 0000000000000000 0000000000000000 0000000000000002 [ 41.850408] GPR12: 0000000028002244 c00000000fff1600 c000000008138be8 c00000016f54cf00 [ 41.850408] GPR16: 0000000000000000 0000000000000000 0000000000000000 000000017e0a0000 [ 41.850408] GPR20: c0000001872635c0 0000000000000000 0000000000000000 0000000000000000 [ 41.850408] GPR24: c00000016f432f28 c00000000971dd78 c0000000091d8580 0000000000000000 [ 41.850408] GPR28: 0000000000000004 0000000000000000 0000000000000000 c00000016f432b00 [ 41.851427] NIP [c000000008149aa0] set_task_cpu+0x240/0x250 [ 41.851507] LR [c00000000814a6cc] try_to_wake_up+0x1bc/0x660 [ 41.851605] Call Trace: [ 41.851649] [c00000016feef130] [c0000000091d8580] runqueues+0x0/0xc00 (unreliable) [ 41.851791] [c00000016feef170] [c00000000814a6cc] try_to_wake_up+0x1bc/0x660 [ 41.851921] [c00000016feef1f0] [c00000000816deb0] __wake_up_common+0xd0/0x200 [ 41.852053] [c00000016feef260] [c00000000843f020] ep_poll_callback+0xe0/0x2f0 [ 41.852159] [c00000016feef2c0] [c00000000816deb0] __wake_up_common+0xd0/0x200 [ 41.852271] [c00000016feef330] [c00000000816e09c] __wake_up_common_lock+0xbc/0x110 [ 41.852379] [c00000016feef3c0] [c00000000818ace0] wake_up_klogd_work_func+0x60/0xc0 [ 41.852492] [c00000016feef3f0] [c000000008291ef0] irq_work_run_list+0xb0/0x100 [ 41.852597] [c00000016feef430] [c0000000081b34e0] update_process_times+0x60/0x90 [ 41.852705] [c00000016feef460] [c0000000081cb1d4] tick_sched_handle.isra.5+0x34/0xd0 [ 41.852817] [c00000016feef490] [c0000000081cb2d0] tick_sched_timer+0x60/0xe0 [ 41.852921] [c00000016feef4d0] [c0000000081b4034] __hrtimer_run_queues+0x144/0x370 [ 41.853029] [c00000016feef550] [c0000000081b4f9c] hrtimer_interrupt+0xfc/0x350 [ 41.853137] [c00000016feef620] [c000000008024950] __timer_interrupt+0x90/0x260 [ 41.853234] [c00000016feef670] [c000000008024d68] timer_interrupt+0x98/0xe0 [ 41.853331] [c00000016feef6a0] [c000000008009014] decrementer_common+0x114/0x120 [ 41.853446] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 41.853446] LR = arch_local_irq_restore+0x74/0x90 [ 41.853602] [c00000016feef990] [c00000016feef9d0] 0xc00000016feef9d0 (unreliable) [ 41.853711] [c00000016feef9b0] [c000000008d03d00] _raw_spin_unlock_irqrestore+0x40/0xa0 [ 41.853825] [c00000016feef9d0] [c00000000857923c] pstore_dump+0x31c/0x3e0 [ 41.853922] [c00000016feefb10] [c00000000818af04] kmsg_dump+0x134/0x1a0 [ 41.854017] [c00000016feefb70] [c0000000080a1214] pnv_platform_error_reboot+0x94/0x110 [ 41.854127] [c00000016feefbe0] [c0000000080a648c] hmi_event_handler+0x1bc/0x1c0 [ 41.854236] [c00000016feefc90] [c00000000812fdb8] process_one_work+0x298/0x5a0 [ 41.854341] [c00000016feefd20] [c000000008130158] worker_thread+0x98/0x630 [ 41.854438] [c00000016feefdc0] [c000000008138d88] kthread+0x1a8/0x1b0 [ 41.854533] [c00000016feefe30] [c00000000800b528] ret_from_kernel_thread+0x5c/0xb4 [ 41.854642] Instruction dump: [ 41.854691] 7faa3670 7d4a0194 57a706be 7d4a07b4 794a1f24 7d28502a 7d293c36 71290001 [ 41.854797] 4082fe80 60000000 60000000 60420000 <0fe00000> 4bfffe6c 60000000 60420000 [ 41.854927] ---[ end trace c9f2cb34f3ff824e ]--- [ 497.382416495,0] OPAL: Reboot requested due to Platform error. [ 497.382470331,3] OPAL: Reboot requested due to Platform error. --== Welcome to Hostboot hostboot-7050d0a/hbicore.bin ==-- 3.99470|secure|SecureROM valid - enabling functionality 3.99474|secure|Booting in non-secure mode. 5.18577|ERRL|Dumping errors reported prior to registration 5.20139|================================================ 5.20139|Error reported by ipmi (0x2500) PLID 0x90033DC5 5.20949| Requested sensor is not present. 5.20949| ModuleId 0x03 IPMI::MOD_IPMISENSOR 5.20949| ReasonCode 0x2507 IPMI::RC_SENSOR_NOT_PRESENT 5.20950| UserData1 BMC IPMI Completion code. : 0x00000000000000cb 5.20951| UserData2 bytes [0-1]sensor name bytes [2-3]sensor number bytes [4-7]HUID of target. : 0xca22000b00010000 5.20952|------------------------------------------------ 5.20952| Callout type : Procedure Callout 5.20952| Procedure : EPUB_PRC_HB_CODE 5.20953| Priority : SRCI_PRIORITY_HIGH 5.20953|------------------------------------------------ 5.20954| Hostboot Build ID: hostboot-7050d0a/hbicore.bin 5.20954|================================================ 6.20986|ISTEP 6. 5 - host_init_fsi 6.34649|ISTEP 6. 6 - host_set_ipl_parms 6.36715|ISTEP 6. 7 - host_discover_targets 6.84607|HWAS|PRESENT> DIMM[03]=AAAA000000000000 6.84608|HWAS|PRESENT> Proc[05]=8800000000000000 6.84609|HWAS|PRESENT> Core[07]=CCFC3FFF03F30000 6.86942|ISTEP 6. 8 - host_update_master_tpm == Comment: #9 - NAVEED A. UPPINANGADY SALIH <naveed...@in.ibm.com> - 2018-02-15 23:55:28 == root@ltciofvtr-spoon4:/var/crash# ls -lrht 201802160038 total 251M -rw------- 1 root root 448M Feb 16 00:38 dump-incomplete == Comment: #14 - Hari Krishna Bathini <hbath...@in.ibm.com> - 2018-02-22 01:40:19 == The following 4 patches are missing in kexec-tools package leading to HMIs commit 69431282f075ab723c4886f20aa248976920aaae Author: Hari Bathini <hbath...@linux.vnet.ibm.com> Date: Tue Aug 29 23:08:02 2017 +0530 kexec-tools: ppc64: fix leak while checking for coherent device memory Signed-off-by: Hari Bathini <hbath...@linux.vnet.ibm.com> Signed-off-by: Simon Horman <ho...@verge.net.au> commit aec4d0f7a2502a13fc21e90ff32dc306b0ad1190 Author: Hari Bathini <hbath...@linux.vnet.ibm.com> Date: Thu Aug 17 18:01:51 2017 +0530 kexec-tools: ppc64: avoid adding coherent memory regions to crash memory ranges Accelerator devices like GPU and FPGA cards contain onboard memory. This onboard memory is represented as a memory only NUMA node, integrating it with core memory subsystem. Since, the link through which these devices are integrated to core memory goes down after a system crash and they are meant for user workloads, avoid adding coherent device memory regions to crash memory ranges. Without this change, makedumpfile tool tries to save unaccessible coherent device memory regions, crashing the system. Signed-off-by: Hari Bathini <hbath...@linux.vnet.ibm.com> Tested-by: Pingfan Liu <pi...@redhat.com> Signed-off-by: Simon Horman <ho...@verge.net.au> commit 21eb397a5fc9227cd95d23e8c74a49cf6a293e57 Author: Hari Bathini <hbath...@linux.vnet.ibm.com> Date: Wed Aug 9 23:47:42 2017 +0530 kexec-tools: powerpc: fix command line overflow error Since kernel commit a5980d064fe2 ("powerpc: Bump COMMAND_LINE_SIZE to 2048"), powerpc bumped command line size to 2048 but the size used here is still the default value of 512. Bump it to 2048 to fix command line overflow errors observed when command line length is above 512 bytes. Also, get rid of the multiple definitions of COMMAND_LINE_SIZE macro in ppc architecture. Signed-off-by: Hari Bathini <hbath...@linux.vnet.ibm.com> Signed-off-by: Simon Horman <ho...@verge.net.au> commit 47478ea66d4301b12a07862aebc8447a2932f0ed Author: Hari Bathini <hbath...@linux.vnet.ibm.com> Date: Wed Jul 26 22:49:41 2017 +0530 kexec-tools: ppc64: fix how RMA top is deduced Hang was observed, in purgatory, on a machine configured with single LPAR. This was because one of the segments was loaded outside the actual Real Memory Area (RMA) due to wrongly deduced RMA top value. Currently, top of real memory area, which is crucial for loading kexec/kdump kernel, is obtained by iterating through mem nodes and setting its value based on the base and size values of the last mem node in the iteration. That can't always be correct as the order of iteration may not be same and RMA base & size are always based on the first memory property. Fix this by setting RMA top value based on the base and size values of the memory node that has the smallest base value (first memory property) among all the memory nodes. Also, correct the misnomers rmo_base and rmo_top to rma_base and rma_top respectively. While how RMA top is deduced was broken for sometime, the issue may not have been seen so far, for couple of possible reasons: 1. Only one mem node was available. 2. First memory property has been the last node in iteration when multiple mem nodes were present. Fixes: 02f4088ffded ("kexec fix ppc64 device-tree mem node") Reported-by: Ankit Kumar <an...@linux.vnet.ibm.com> Cc: Michael Ellerman <m...@ellerman.id.au> Cc: Geoff Levand <ge...@infradead.org> Signed-off-by: Hari Bathini <hbath...@linux.vnet.ibm.com> Signed-off-by: Simon Horman <ho...@verge.net.au> -- I would recommend upgrading to the latest kexec-tools verison 2.0.16 (which includes this patches) instead of cherry-picking these patches.. Thanks Hari To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1751835/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp