------- Comment From hbath...@in.ibm.com 2019-06-24 07:49 EDT------- Thanks for the change. With it, try-restart is being triggered for kdump-tools service after CPU add operation but systemd reported failure with below logs:
Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service. Jun 24 06:47:06 ubuntu systemd[1]: Starting Kernel crash dump capture service... Jun 24 06:47:06 ubuntu kdump-tools[2023]: Starting kdump-tools: * Creating symlink /var/lib/kdump/vmlinuz Jun 24 06:47:06 ubuntu kdump-tools[2023]: * Creating symlink /var/lib/kdump/initrd.img Jun 24 06:47:06 ubuntu kdump-tools[2023]: Modified cmdline:BOOT_IMAGE=/vmlinux-5.0.0-17-generic root=/dev/mapper/ubuntu--vg-root ro systemd.unit=kdump-tools-dump.service maxcpus=1 irqpo Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Main process exited, code=killed, status=15/TERM Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Failed with result 'signal'. Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service. Jun 24 06:47:06 ubuntu systemd[1]: Starting Kernel crash dump capture service... Jun 24 06:47:06 ubuntu kdump-tools[2071]: Starting kdump-tools: * Creating symlink /var/lib/kdump/vmlinuz Jun 24 06:47:06 ubuntu kdump-tools[2071]: * Creating symlink /var/lib/kdump/initrd.img Jun 24 06:47:06 ubuntu kdump-tools[2071]: Modified cmdline:BOOT_IMAGE=/vmlinux-5.0.0-17-generic root=/dev/mapper/ubuntu--vg-root ro systemd.unit=kdump-tools-dump.service maxcpus=1 irqpo Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Main process exited, code=killed, status=15/TERM Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Failed with result 'signal'. Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service. Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Start request repeated too quickly. Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Failed with result 'signal'. Jun 24 06:47:06 ubuntu systemd[1]: Failed to start Kernel crash dump capture service. --- Looks like a ratelimit issue with systemd. Is there some systemd option to workaround it? I am running the below command on a PowerVM machine: # drmgr -c cpu -r -q 1 (to remove a core) # drmgr -c cpu -a -q 1 (to add it back -> this triggers 8 CPU online udev events as SMT is 8) To conclude, udev rule alone is not sufficient. Need a way to address the multiple requests at once.. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Committed Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Committed Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
