[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Changed in: ubuntu-power-systems Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Fix Released Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Fix Released Status in makedumpfile source package in Bionic: Fix Released Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Released Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
This bug was fixed in the package makedumpfile - 1:1.6.5-1ubuntu1~18.04.4 --- makedumpfile (1:1.6.5-1ubuntu1~18.04.4) bionic; urgency=medium [ Thadeu Lima de Souza Cascardo ] * Fixes for DLPAR cpu add operation (LP: #1828596) - d/kdump-config.in: Add a reload command. - d/kdump-config.in: implement try-reload. - d/50-kdump-tools.rules: Use kdump-config reload after cpu or memory hotplug - d/50-kdump-tools.rules: use try-reload instead. * d/rules: Use reset_devices as a cmdline parameter. (LP: #1800566) [ Guilherme G. Piccoli ] * d/kdump-tools-dump.service: Add a systemd-resolved service dependency in order to make kdump-tool able to resolve DNS when in kdump boot. (LP: #1856323) * Fix an error due to makedumpfile being out-of-sync with recent kernels. (LP: #1857616) - d/p/0004-x86_64-fix-get_kaslr_offset_x86_64-to-return-kaslr_offset-correctly.patch - d/p/0005-Increase-SECTION_MAP_LAST_BIT-to-4.patch -- gpicc...@canonical.com (Guilherme G. Piccoli) Fri, 03 Jan 2020 13:14:39 -0300 ** Changed in: makedumpfile (Ubuntu Bionic) Status: Fix Committed => Fix Released ** Changed in: makedumpfile (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Fix Committed Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Fix Released Status in makedumpfile source package in Bionic: Fix Released Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Released Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
This bug was fixed in the package makedumpfile - 1:1.6.3-2~16.04.2 --- makedumpfile (1:1.6.3-2~16.04.2) xenial; urgency=medium * Let the kernel decide the crashkernel offset for ppc64el (LP: #1741860) * Reload kdump after memory/CPU hotplug. (LP: #1655280) * Use a different service for vmcore dump. (LP: #1811692) * Reload kdump when CPU is brought online. (LP: #1828596) * Add a reload command. (LP: #1828596) * kdump-config: implement try-reload (LP: #1828596) * udev: hotplug: use try-reload (LP: #1828596) * Use reset_devices as a cmdline parameter. (LP: #1800566) -- Thadeu Lima de Souza Cascardo Wed, 18 Dec 2019 16:06:16 -0300 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Fix Committed Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Fix Released Status in makedumpfile source package in Bionic: Fix Released Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Released Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
This bug was fixed in the package makedumpfile - 1:1.6.6-2ubuntu2 --- makedumpfile (1:1.6.6-2ubuntu2) eoan; urgency=medium [ Thadeu Lima de Souza Cascardo ] * Fixes for DLPAR cpu add operation (LP: #1828596) - d/kdump-config.in: Add a reload command. - d/kdump-config.in: implement try-reload. - d/50-kdump-tools.rules: Use kdump-config reload after cpu or memory hotplug - d/50-kdump-tools.rules: use try-reload instead. * d/rules: Use reset_devices as a cmdline parameter. (LP: #1800566) [ Guilherme G. Piccoli ] * d/kdump-tools-dump.service: Add a systemd-resolved service dependency in order to make kdump-tool able to resolve DNS when in kdump boot. (LP: #1856323) * d/p/0003-Increase-SECTION_MAP_LAST_BIT-to-4.patch: x86_64: Fix an error due to makedumpfile being out-of-sync with recent kernels. (LP: #1857616) -- gpicc...@canonical.com (Guilherme G. Piccoli) Fri, 03 Jan 2020 16:10:19 -0300 ** Changed in: makedumpfile (Ubuntu Eoan) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Fix Committed Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Fix Committed Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Released Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Tags removed: targetmilestone-inin--- ** Tags added: targetmilestone-inin18043 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Fix Committed Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Fix Committed Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Committed Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hello Hari, Glad to see you around on bugs again. :) Could you please confirm whether the fix for this bug also addresses LP bug 1655280? (i.e., the verification for xenial would be equivalent/also validate that other bug.) Thanks, Mauricio -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Fix Committed Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Fix Committed Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Committed Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Yes, that's fine - disco was already set to Won't Fix on the Launchpad bug side, since it will reach it's EOL on Jan the 23rd. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Fix Committed Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Fix Committed Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Committed Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
--- Comment From hbath...@in.ibm.com 2020-01-15 15:44 EDT--- With EOL for disco this month, marking as verification done as this is verified successfully on bionic, eoan & xenial. ** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Fix Committed Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Fix Committed Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Committed Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
--- Comment From hbath...@in.ibm.com 2020-01-15 15:34 EDT--- Verified successfully on eoan with kdump-tools version 1:1.6.6-2ubuntu2 --- Comment From hbath...@in.ibm.com 2020-01-15 15:36 EDT--- Verified successfully on xenial with kdump-tools version 1:1.6.3-2~16.04.2 --- Comment From hbath...@in.ibm.com 2020-01-15 15:38 EDT--- Verified successfully on bionic with kdump-tools version 1:1.6.5-1ubuntu1~18.04.4 ** Tags removed: verification-needed-eoan verification-needed-xenial ** Tags added: verification-done-eoan verification-done-xenial -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Fix Committed Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Fix Committed Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Committed Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Changed in: ubuntu-power-systems Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Fix Committed Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Fix Committed Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Committed Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hello bugproxy, or anyone else affected, Accepted makedumpfile into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.3-2~16.04.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: makedumpfile (Ubuntu Xenial) Status: In Progress => Fix Committed ** Tags added: verification-needed-xenial -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Fix Committed Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Committed Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Thanks for the verification! Can you please verify the Eoan version too? Cheers, Guilherme -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Committed Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Tags removed: verification-needed-bionic ** Tags added: verification-done-bionic -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Committed Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hello bugproxy, or anyone else affected, Accepted makedumpfile into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1~18.04.4 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: makedumpfile (Ubuntu Bionic) Status: In Progress => Fix Committed ** Tags removed: verification-failed-bionic ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Committed Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help :
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hello bugproxy, or anyone else affected, Accepted makedumpfile into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.6-2ubuntu2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: makedumpfile (Ubuntu Eoan) Status: In Progress => Fix Committed ** Tags removed: verification-failed ** Tags added: verification-needed verification-needed-eoan -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: Fix Committed Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hi Guilherme, Thadeu, The re-spinned packages have been uploaded to Bionic and Eoan. Xenial didn't change, and is still in the upload queue (which I'll go check about on next week.) Disco didn't change, and has been removed from the upload queue. (After discussions with other people, it turns out uploading to Disco is not necessary given the short span until its EOL date.) cheers, Mauricio ** Changed in: makedumpfile (Ubuntu Disco) Status: In Progress => Won't Fix ** Changed in: makedumpfile (Ubuntu Disco) Assignee: Thadeu Lima de Souza Cascardo (cascardo) => (unassigned) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: In Progress Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
This bug was fixed in the package makedumpfile - 1:1.6.6-4ubuntu1 --- makedumpfile (1:1.6.6-4ubuntu1) focal; urgency=medium [ Thadeu Lima de Souza Cascardo ] * Merge from Debian unstable. Remaining changes: - Bump amd64 crashkernel from 384M-:128M to 512M-:192M. * Use reset_devices as a cmdline parameter. (LP: #1800566) * Use kdump-config reload after cpu or memory hotplug. (LP: #1828596) [ Guilherme G. Piccoli ] * Add a systemd-resolved service dependency in order kdump-tools is able to resolve DNS when in kdump boot. (LP: #1856323) makedumpfile (1:1.6.6-4) unstable; urgency=medium * Let the kernel decide the crashkernel offset for ppc64el (LP: #1741860) * kdump-config: implement try-reload * udev: hotplug: use try-reload * Set Rules-Requires-Root to no makedumpfile (1:1.6.6-3) unstable; urgency=medium * Add a reload command. * Use kdump-config reload after cpu or memory hotplug. * Use reset_devices as a cmdline parameter. -- Thadeu Lima de Souza Cascardo Wed, 18 Dec 2019 14:38:51 -0300 ** Changed in: makedumpfile (Ubuntu Focal) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Status in makedumpfile source package in Focal: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "eoan_V3.debdiff" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+attachment/5317481/+files/eoan_V3.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Status in makedumpfile source package in Focal: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "focal_new.debdiff" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+attachment/5317480/+files/focal_new.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Status in makedumpfile source package in Focal: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
A recent issue on makedumpfile was reported and investigated on LP#1857616. Due to the relevance of that, it makes sense to respin the debdiffs. So, I've reworked debdiffs for Bionic and Eoan, that should replace the ones above. Disco is not affected. Since those packages are not in -proposed, the respin is on top of the same version, to replace the old upload. Now, Focal is present in -proposed for some time, so the respin is on top of -proposed version (which I expect to be released to -updates soon), with version increment. Thanks, Guilherme -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Status in makedumpfile source package in Focal: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
I'm re-attaching the untouched disco debdiff, just for organization/completeness. ** Patch added: "disco_reattachment.debdiff" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+attachment/5317482/+files/disco_reattachment.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Status in makedumpfile source package in Focal: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
I'm re-attaching the untouched Xenial debdiff, just for organization/completeness. ** Patch added: "xenial_reattachment.debdiff" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+attachment/5317484/+files/xenial_reattachment.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Status in makedumpfile source package in Focal: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "bionic_V3.debdiff" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+attachment/5317483/+files/bionic_V3.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Status in makedumpfile source package in Focal: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Focal was uploaded but it's failing in ppc64el again (see LP #1851663). I've submitted a merge request[0] in Britney to skip this test once more. Cheers, Guilherme [0] https://code.launchpad.net/~gpiccoli/britney/hints- ubuntu/+merge/377160 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Status in makedumpfile source package in Focal: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Changed in: makedumpfile (Ubuntu Disco) Status: Won't Fix => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Status in makedumpfile source package in Focal: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "disco debdiff" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+attachment/5314322/+files/makedumpfile_1.6.5-1ubuntu1.4.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: In Progress Status in makedumpfile source package in Focal: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Also affects: makedumpfile (Ubuntu Focal) Importance: Undecided Assignee: Thadeu Lima de Souza Cascardo (cascardo) Status: In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: In Progress Status in makedumpfile source package in Focal: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "eoan debdiff" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+attachment/5314049/+files/makedumpfile_1.6.6-2ubuntu2.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "xenial debdiff" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+attachment/5314051/+files/makedumpfile_1.6.3-2~16.04.2.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "bionic debdiff" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+attachment/5314050/+files/makedumpfile_1.6.5-1ubuntu1~18.04.4.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Changed in: makedumpfile (Ubuntu Disco) Status: In Progress => Won't Fix ** Changed in: makedumpfile (Ubuntu Xenial) Status: New => In Progress ** Patch added: "focal debdiff" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+attachment/5314048/+files/makedumpfile_1.6.6-4ubuntu1.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Won't Fix Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
New version on ppa:cascardo/ppa for bionic. That should handle fadump and lock in the case of try-reload/condreload. Hari, can you give it a try? Thanks. Cascardo. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: New Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hi, Hari. makedumpfile 1:1.6.5-1ubuntu1~18.04.2+cascardo2 on ppa:cascardo/ppa uses a try-reload instead. Can you test it, please? Thanks. Cascardo. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: New Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hi Hari, did you have a chance to re-test using the latest version Cascardo pointed in his last comment? We wait on your testing to be sure all is working now, and we can re-upload the package to -proposed pocket. Thanks in advance, Guilherme -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: New Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
A fixed grep is already in bionic-updates. Either that or the one on my ppa must be installed in order for the makedumpfile version in my ppa to work. I will wait for testing feeback before I get this fix uploaded to eoan, disco and bionic. Thanks. Cascardo. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: New Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Changed in: makedumpfile (Ubuntu Eoan) Status: Fix Released => In Progress ** Changed in: makedumpfile (Ubuntu Disco) Status: Fix Committed => In Progress ** Changed in: makedumpfile (Ubuntu Bionic) Status: Won't Fix => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: New Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
https://bugs.launchpad.net/ubuntu/+source/grep/+bug/1844524 This should be fixed by rebuilding grep. I have uploaded grep to my ppa, so if you install kdump-tools and grep from ppa:cascardo/ppa, you will be able to test this. Can you please do it, so we reduce the risk of the next upload being remove from -proposed again? Thanks. Cascardo. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: New Status in makedumpfile source package in Bionic: Won't Fix Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Committed Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
There is some interaction between systemd MemoryDenyWriteExecute=yes setting on udevd and how grep has been built (possibly pointing out at the toolchain), so the new solution on the ppa isn't working on bionic. We will work on this bug, and see how this behaves on disco and eoan. In case either of those is fine, we will ask IBM to test it there, while we move forward with this systemd/toolchain interaction bug. Thanks. Cascardo. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: New Status in makedumpfile source package in Bionic: Won't Fix Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Committed Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hi, can you try the package in ppa:cascardo/ppa ? Thanks. Cascardo. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: New Status in makedumpfile source package in Bionic: Won't Fix Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Committed Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** No longer affects: kexec-tools (Ubuntu Eoan) ** No longer affects: kexec-tools (Ubuntu Disco) ** No longer affects: kexec-tools (Ubuntu Cosmic) ** No longer affects: kexec-tools (Ubuntu Bionic) ** No longer affects: kexec-tools (Ubuntu Xenial) ** No longer affects: kexec-tools (Ubuntu) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: New Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Committed Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Tags removed: verification-needed verification-needed-disco ** Tags added: verification-failed verification-failed-disco -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: Fix Committed Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: Won't Fix Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: Fix Committed Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
--- Comment on attachment From hbath...@in.ibm.com 2019-08-29 08:34 EDT--- udev rules are not triggering kdump-tools service restart after hot adding CPU or hot adding/removing memory with kdump-tools package version 1.6.5-1ubuntu1~18.04.2 ** Tags removed: verification-needed-bionic ** Tags added: verification-failed-bionic ** Attachment added: "kdump tools service is not restarting with the latest change to udev rules" https://bugs.launchpad.net/bugs/1828596/+attachment/5285633/+files/no-kdump-tools-restart-with-new-udev-rules.log -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: Fix Committed Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: Won't Fix Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: Fix Committed Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Changed in: ubuntu-power-systems Status: Triaged => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: In Progress Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: Fix Committed Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: Won't Fix Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: Fix Committed Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hello bugproxy, or anyone else affected, Accepted makedumpfile into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1~18.04.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: makedumpfile (Ubuntu Bionic) Status: In Progress => Fix Committed ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Triaged Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: Fix Committed Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: Won't Fix Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: Fix Committed Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to:
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hello bugproxy, or anyone else affected, Accepted makedumpfile into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: makedumpfile (Ubuntu Disco) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-disco -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Triaged Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: Fix Committed Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: Won't Fix Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: Fix Committed Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to:
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "backport to xenial" https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/1828596/+attachment/5285400/+files/makedumpfile_1.6.3-2~16.04.2.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Triaged Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: In Progress Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: Won't Fix Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: In Progress Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Changed in: makedumpfile (Ubuntu Disco) Status: New => In Progress ** Changed in: makedumpfile (Ubuntu Disco) Assignee: (unassigned) => Thadeu Lima de Souza Cascardo (cascardo) ** Changed in: makedumpfile (Ubuntu Bionic) Status: New => In Progress ** Changed in: makedumpfile (Ubuntu Bionic) Assignee: (unassigned) => Thadeu Lima de Souza Cascardo (cascardo) ** Changed in: makedumpfile (Ubuntu Cosmic) Status: New => Won't Fix -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Triaged Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: In Progress Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: Won't Fix Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: In Progress Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Changed in: ubuntu-power-systems Status: Incomplete => Triaged -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Triaged Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
It was implemented, but the upload did not build on that ppa, because I used different versions. I am still catching up after vacation time, so will post some updates as soon as I have them. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Incomplete Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hi Hari, thanks for clarifying! I can now understand, seems we need to wait for Cascardo's input, to see if he already implemented the systemd reset-failed thing or not. Cheers, Guilherme -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Incomplete Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hi Hari, did you manage to try the package in https://launchpad.net/~cascardo/+archive/ubuntu/kdump2? I've downloaded the kdump-tools deb package for ppc64 from the above PPA, and could check that it contains the udev rule: "SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" I understand by reading the latest comments that above rule is the fix for this LP, correct? Can you manually download the package from the above PPA, install it and verify that /lib/udev/rules.d/50-kdump-tools.rules contains the fixing rule? In case it has that and still fails your testing, then we need to understand why the fix is not working. Thanks, Guilherme -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Incomplete Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Disabling the ratelimit in general would break other failure modes, so I would rather just reset-failed when calling try-restart because of the hotplug events. Can you try the package in ppa:cascardo/kdump2? Packages for eoan, disco and bionic available. Thanks. Cascardo. ** Description changed: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] - Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. - + Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Incomplete Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. This has been handled by resetting the failure counter when doing such reloads. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Changed in: ubuntu-power-systems Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Incomplete Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "backport to bionic" https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/1828596/+attachment/5274730/+files/makedumpfile_1.6.5-1ubuntu2~18.04.1.debdiff ** Description changed: + [Impact] + After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. + + [Test case] + Do CPU add/hotplug, trigger a crash, and check for a successful kdump. + + [Regression potential] + Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. + + == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. - - Machine Type = na - + + Machine Type = na + ---System Hang--- - Crashed in early boot process of kdump kernel after crash + Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim - + ---Steps to Reproduce--- - 1. Configure kdump. + 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory - --- STRUCK HERE --- - + --- STRUCK HERE --- + ---uname output--- na - + ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: - - SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" + SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart + kdump-tools.service" -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "backport to cosmic" https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/1828596/+attachment/5274729/+files/makedumpfile_1.6.5-1ubuntu2~18.10.1.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "backport to disco" https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/1828596/+attachment/5274728/+files/makedumpfile_1.6.5-1ubuntu2~19.04.1.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] After a CPU add/hotplug operation on Power systems, kdump will fail after a crash. The kdump kernel needs to be reloaded after a CPU add/hotplug. [Test case] Do CPU add/hotplug, trigger a crash, and check for a successful kdump. [Regression potential] Multiple reloads caused by multiple sequential CPU adds may cause spurious log results, and systemd may fail to properly reload the kdump kernel. == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try- restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "backport for bionic" https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/1828596/+attachment/5274692/+files/makedumpfile_1.6.5-1ubuntu2~18.04.1.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "backport for cosmic" https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/1828596/+attachment/5274691/+files/makedumpfile_1.6.5-1ubuntu2~18.10.1.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Patch added: "backport for disco" https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/1828596/+attachment/5274690/+files/makedumpfile_1.6.5-1ubuntu2~19.04.1.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
This bug was fixed in the package makedumpfile - 1:1.6.5-1ubuntu2 --- makedumpfile (1:1.6.5-1ubuntu2) eoan; urgency=medium [ Thadeu Lima de Souza Cascardo ] * Use maxcpus instead of nr_cpus on ppc64el. (LP: #1828597) * Reload kdump when CPU is brought online. (LP: #1828596) -- Thadeu Lima de Souza Cascardo Fri, 14 Jun 2019 10:58:40 -0300 ** Changed in: makedumpfile (Ubuntu Eoan) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Released Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
This is now in eoan-proposed. Please verify. I will start the backport process when it hits eoan. Thanks. Cascardo. ** Also affects: kexec-tools (Ubuntu Eoan) Importance: Undecided Assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) Status: Invalid ** Also affects: makedumpfile (Ubuntu Eoan) Importance: Undecided Assignee: Thadeu Lima de Souza Cascardo (cascardo) Status: Confirmed ** Also affects: kexec-tools (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: makedumpfile (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: kexec-tools (Ubuntu Cosmic) Importance: Undecided Status: New ** Also affects: makedumpfile (Ubuntu Cosmic) Importance: Undecided Status: New ** Also affects: kexec-tools (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: makedumpfile (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: kexec-tools (Ubuntu Disco) Importance: Undecided Status: New ** Also affects: makedumpfile (Ubuntu Disco) Importance: Undecided Status: New ** Changed in: makedumpfile (Ubuntu Eoan) Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Committed Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Bionic: New Status in makedumpfile source package in Bionic: New Status in kexec-tools source package in Cosmic: New Status in makedumpfile source package in Cosmic: New Status in kexec-tools source package in Disco: New Status in makedumpfile source package in Disco: New Status in kexec-tools source package in Eoan: Invalid Status in makedumpfile source package in Eoan: Fix Committed Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Based on the last comment, it looks like IBM's testing was successful and this patch is ready for SRU. Thanks. ** Changed in: ubuntu-power-systems Status: Incomplete => Confirmed ** Changed in: makedumpfile (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Confirmed Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Hi, Hari. So, as you said, other architectures will use add/remove instead of online, and we want to support them too. Any reason not to do it that you are thinking of? Thanks. Cascardo. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Incomplete Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Incomplete Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
Marking as "incomplete" while awaiting test results from Thadeu's PPA kernel. ** Changed in: ubuntu-power-systems Status: Triaged => Incomplete ** Changed in: makedumpfile (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Incomplete Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Incomplete Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
At my ppa, there is a version with the change. Can you please test? The package is available for bionic, cosmic, disco and eoan. ppa:cascardo/kdump2 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Triaged Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: New Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Tags added: powervm -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Triaged Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: New Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Changed in: ubuntu-power-systems Status: New => Triaged -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: Triaged Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: New Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
I will start working on an upload to eoan by next week. I should have something for you to test early in the week. ** Also affects: makedumpfile (Ubuntu) Importance: Undecided Status: New ** Changed in: kexec-tools (Ubuntu) Status: New => Invalid ** Changed in: makedumpfile (Ubuntu) Assignee: (unassigned) => Thadeu Lima de Souza Cascardo (cascardo) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: New Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: New Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1828596] Re: kdump fails when crash is triggered after DLPAR cpu add operation
** Also affects: ubuntu-power-systems Importance: Undecided Status: New ** Changed in: ubuntu-power-systems Assignee: (unassigned) => Canonical Kernel Team (canonical-kernel-team) ** Changed in: ubuntu-power-systems Importance: Undecided => High -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation Status in The Ubuntu-power-systems project: New Status in kexec-tools package in Ubuntu: New Bug description: == Comment: #0 - Hari Krishna Bathini - 2019-05-10 05:55:40 == ---Problem Description--- kdump fails when crash is triggered after CPU add operation. Machine Type = na ---System Hang--- Crashed in early boot process of kdump kernel after crash Had to issue system reset from HMC to reclaim ---Steps to Reproduce--- 1. Configure kdump. 2. Add cpu from HMC. 3. Trigger crash. 4. Machine hangs after crash as below: --- [169250.213166] IPI complete [169250.234331] kexec: Starting switchover sequence. I'm in purgatory --- STRUCK HERE --- ---uname output--- na ---Debugger--- A debugger is not configured == Comment: #1 - Hari Krishna Bathini - 2019-05-10 05:56:46 == The problem is, kexec udev rule to restart kdump-tools service - when a core is added, is not being triggered. The old DT created by kexec (before the core is added) is being used by KDump Kernel. So, when system crashes on a thread from the added core(s), KDump kernel is failing to get the 'boot_cpuid' and eventually failing to boot.. == Comment: #2 - Hari Krishna Bathini - 2019-05-10 06:02:27 == The udev rule when CPU is added is not triggered because ppc64 does not eject add/remove event when a CPU is hot added/removed. It only ejects online/offline event to user space when CPU is hot added/removed. So, the below udev rules are never triggered when needed: SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Also, with how CPU hot add & remove are handled in ppc64, a udev trigger to reload kdump after CPU is hot removed is NOT necessary. So, fix the CPU hot add case by updating the udev rule and drop the udev rule meant for CPU hot remove in the kdump udev rules file: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
