[Kernel-packages] [Bug 1658733] Comment bridged from LTC Bugzilla
*** This bug is a duplicate of bug 1635597 *** https://bugs.launchpad.net/bugs/1635597 --- Comment From lagar...@br.ibm.com 2017-08-03 17:10 EDT--- Hi Canonical, Any updates here? --- Comment From lagar...@br.ibm.com 2017-08-03 17:17 EDT--- Hi Canonical, Sorry for my last comment. Disregard it. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1658733 Title: Ubuntu 16.04.2KVM:kdump fails to mount root file system on multipath root device Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: In Progress Status in kexec-tools source package in Trusty: New Status in makedumpfile source package in Trusty: New Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: New Status in kexec-tools source package in Zesty: New Status in makedumpfile source package in Zesty: New Bug description: == Comment: #0 - Richard M. Scheller - 2016-12-14 16:50:26 == ---Problem Description--- On a KVM guest installed to a multipath root device, the kdump kernel fails to mount the root file system. This error does not occur in a similar guest installed to a single path device. Full console output of the kdump failure is attached. These messages from the output may be relevant: Begin: Loading multipath modules ... Success: loaded module dm-multipath. done. Begin: Loading multipath hardware handlers ... Failure: failed to load module sc si_dh_alua. Failure: failed to load module scsi_dh_rdac. Failure: failed to load module scsi_dh_emc. done. Begin: Starting multipathd ... done. ---uname output--- Linux dotg9 4.8.0-32-generic #34~16.04.1-Ubuntu SMP Tue Dec 13 17:01:57 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux Machine Type = 8247-22L Ubuntu 16.04.1 KVM guest ---Steps to Reproduce--- - Install Ubuntu 16.04.1 to a muiltpath target disk - Install kdump-tools package - Configure kexec-tools to reserve sufficient RAM for the kdump kernel to load (I use 512MB) in /etc/default/grub.d/kexec-tools.cfg - Run update-grub - Reboot - Initiate a system crash using "echo c > /proc/sysrq-trigger" == Comment: #12 - Richard M. Scheller - 2016-12-20 20:37:45 == Here is the log level 8 kdump console log requested in comment 10. == Comment: #21 - Richard M. Scheller - 2017-01-06 11:04:17 == (In reply to comment #19) > Hi, I logged in dotkvm and I couldn't find the guest dotg9. Also, although I > found a dotg9.xml in /kte/xml/ it doesn't look like it uses multipath (it > uses .img files which I didn't found as disks). > > Could you please recreate the guest for further debug? Yes, I recreated the guest with its correct multipath lun configuration. I have also attached the guest XML to this bug. > Besides that could you please let us know: > - is the multipath the system's root? I mean / is installed/mounted on the > multipath device? Yes, the guest has only one disk. That disk is actually a LUN from a fiber channel storage device with two paths on the host side. I have passed through both paths to the guest, so the multipath nature of the target disk is known to the guest. In other words, the guest sees a multipath device and is using it as a multipath device. The root file system is called /dev/mapper/mpatha- part2 on the guest. > - how did you attach the device to the guest? Each FC LUN path on the host is mapped to a virtio-scsi controller on the guest using LUN passthrough. (See the guest XML for details on this.) == Comment: #22 - Mauro Sergio Martins Rodrigues - 2017-01-11 09:31:38 == I managed to get kdump to mount rootfs and perform its tasks by setting KDUMP_CMDLINE_APPEND="nr_cpus=4" parameter in /etc/default/kdump-tools see http://pastebin.hursley.ibm.com/8239 I'm still investigating to figure out what is the reason behind this behavior. Thanks, -- maurosr == Comment: #23 - Mauricio Faria De Oliveira - 2017-01-11 11:56:40 == Mauro, (In reply to comment #22) > I managed to get kdump to mount rootfs and perform its tasks by setting > KDUMP_CMDLINE_APPEND="nr_cpus=4" parameter in /etc/default/kdump-tools see > http://pastebin.hursley.ibm.com/8239 > > I'm still investigating to figure out what is the reason behind this > behavior. > > Thanks, > > -- > maurosr That would smell like an out of memory condition that is alleviated with a smaller number of CPUs allowed for the kernel (so the amount of memory associated with per-CPU stuff is less in total). Per the bug description, the memory reserved for the crashkernel is 512MB: (In reply to comment #23) > - Configure kexec-tools to reserve sufficient RAM for the kdump kernel to > load (I use
[Kernel-packages] [Bug 1658733] Comment bridged from LTC Bugzilla
--- Comment From lagar...@br.ibm.com 2017-07-04 15:48 EDT--- Hi Canonical, Any updates here? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1658733 Title: Ubuntu 16.04.2KVM:kdump fails to mount root file system on multipath root device Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Confirmed Status in makedumpfile package in Ubuntu: Confirmed Bug description: == Comment: #0 - Richard M. Scheller - 2016-12-14 16:50:26 == ---Problem Description--- On a KVM guest installed to a multipath root device, the kdump kernel fails to mount the root file system. This error does not occur in a similar guest installed to a single path device. Full console output of the kdump failure is attached. These messages from the output may be relevant: Begin: Loading multipath modules ... Success: loaded module dm-multipath. done. Begin: Loading multipath hardware handlers ... Failure: failed to load module sc si_dh_alua. Failure: failed to load module scsi_dh_rdac. Failure: failed to load module scsi_dh_emc. done. Begin: Starting multipathd ... done. ---uname output--- Linux dotg9 4.8.0-32-generic #34~16.04.1-Ubuntu SMP Tue Dec 13 17:01:57 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux Machine Type = 8247-22L Ubuntu 16.04.1 KVM guest ---Steps to Reproduce--- - Install Ubuntu 16.04.1 to a muiltpath target disk - Install kdump-tools package - Configure kexec-tools to reserve sufficient RAM for the kdump kernel to load (I use 512MB) in /etc/default/grub.d/kexec-tools.cfg - Run update-grub - Reboot - Initiate a system crash using "echo c > /proc/sysrq-trigger" == Comment: #12 - Richard M. Scheller - 2016-12-20 20:37:45 == Here is the log level 8 kdump console log requested in comment 10. == Comment: #21 - Richard M. Scheller - 2017-01-06 11:04:17 == (In reply to comment #19) > Hi, I logged in dotkvm and I couldn't find the guest dotg9. Also, although I > found a dotg9.xml in /kte/xml/ it doesn't look like it uses multipath (it > uses .img files which I didn't found as disks). > > Could you please recreate the guest for further debug? Yes, I recreated the guest with its correct multipath lun configuration. I have also attached the guest XML to this bug. > Besides that could you please let us know: > - is the multipath the system's root? I mean / is installed/mounted on the > multipath device? Yes, the guest has only one disk. That disk is actually a LUN from a fiber channel storage device with two paths on the host side. I have passed through both paths to the guest, so the multipath nature of the target disk is known to the guest. In other words, the guest sees a multipath device and is using it as a multipath device. The root file system is called /dev/mapper/mpatha- part2 on the guest. > - how did you attach the device to the guest? Each FC LUN path on the host is mapped to a virtio-scsi controller on the guest using LUN passthrough. (See the guest XML for details on this.) == Comment: #22 - Mauro Sergio Martins Rodrigues - 2017-01-11 09:31:38 == I managed to get kdump to mount rootfs and perform its tasks by setting KDUMP_CMDLINE_APPEND="nr_cpus=4" parameter in /etc/default/kdump-tools see http://pastebin.hursley.ibm.com/8239 I'm still investigating to figure out what is the reason behind this behavior. Thanks, -- maurosr == Comment: #23 - Mauricio Faria De Oliveira - 2017-01-11 11:56:40 == Mauro, (In reply to comment #22) > I managed to get kdump to mount rootfs and perform its tasks by setting > KDUMP_CMDLINE_APPEND="nr_cpus=4" parameter in /etc/default/kdump-tools see > http://pastebin.hursley.ibm.com/8239 > > I'm still investigating to figure out what is the reason behind this > behavior. > > Thanks, > > -- > maurosr That would smell like an out of memory condition that is alleviated with a smaller number of CPUs allowed for the kernel (so the amount of memory associated with per-CPU stuff is less in total). Per the bug description, the memory reserved for the crashkernel is 512MB: (In reply to comment #23) > - Configure kexec-tools to reserve sufficient RAM for the kdump kernel to > load (I use 512MB) in /etc/default/grub.d/kexec-tools.cfg That seems low for Power guests/systems. I think it theory is doesn't seem so, but the reality is that _for some reason(s)_ we require just too much memory to load and boot a kernel/initramfs (either on boot or kdump). When working w/ kdump and Ubuntu, I usually set the crashkernel allocated size right away to 4GB to avoid problems. Since this is a smaller sized guest, obviously we'd want to use less than that, but more than 512 MB given the evidence observed. Hope this helps
[Kernel-packages] [Bug 1658733] Comment bridged from LTC Bugzilla
--- Comment From maur...@br.ibm.com 2017-03-27 13:05 EDT--- Hi Canonical, Is there any news related to this bug and the patch attached? Is it already accepted? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1658733 Title: Ubuntu 16.04.2KVM:kdump fails to mount root file system on multipath root device Status in kexec-tools package in Ubuntu: New Status in makedumpfile package in Ubuntu: New Bug description: == Comment: #0 - Richard M. Scheller - 2016-12-14 16:50:26 == ---Problem Description--- On a KVM guest installed to a multipath root device, the kdump kernel fails to mount the root file system. This error does not occur in a similar guest installed to a single path device. Full console output of the kdump failure is attached. These messages from the output may be relevant: Begin: Loading multipath modules ... Success: loaded module dm-multipath. done. Begin: Loading multipath hardware handlers ... Failure: failed to load module sc si_dh_alua. Failure: failed to load module scsi_dh_rdac. Failure: failed to load module scsi_dh_emc. done. Begin: Starting multipathd ... done. ---uname output--- Linux dotg9 4.8.0-32-generic #34~16.04.1-Ubuntu SMP Tue Dec 13 17:01:57 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux Machine Type = 8247-22L Ubuntu 16.04.1 KVM guest ---Steps to Reproduce--- - Install Ubuntu 16.04.1 to a muiltpath target disk - Install kdump-tools package - Configure kexec-tools to reserve sufficient RAM for the kdump kernel to load (I use 512MB) in /etc/default/grub.d/kexec-tools.cfg - Run update-grub - Reboot - Initiate a system crash using "echo c > /proc/sysrq-trigger" == Comment: #12 - Richard M. Scheller - 2016-12-20 20:37:45 == Here is the log level 8 kdump console log requested in comment 10. == Comment: #21 - Richard M. Scheller - 2017-01-06 11:04:17 == (In reply to comment #19) > Hi, I logged in dotkvm and I couldn't find the guest dotg9. Also, although I > found a dotg9.xml in /kte/xml/ it doesn't look like it uses multipath (it > uses .img files which I didn't found as disks). > > Could you please recreate the guest for further debug? Yes, I recreated the guest with its correct multipath lun configuration. I have also attached the guest XML to this bug. > Besides that could you please let us know: > - is the multipath the system's root? I mean / is installed/mounted on the > multipath device? Yes, the guest has only one disk. That disk is actually a LUN from a fiber channel storage device with two paths on the host side. I have passed through both paths to the guest, so the multipath nature of the target disk is known to the guest. In other words, the guest sees a multipath device and is using it as a multipath device. The root file system is called /dev/mapper/mpatha- part2 on the guest. > - how did you attach the device to the guest? Each FC LUN path on the host is mapped to a virtio-scsi controller on the guest using LUN passthrough. (See the guest XML for details on this.) == Comment: #22 - Mauro Sergio Martins Rodrigues - 2017-01-11 09:31:38 == I managed to get kdump to mount rootfs and perform its tasks by setting KDUMP_CMDLINE_APPEND="nr_cpus=4" parameter in /etc/default/kdump-tools see http://pastebin.hursley.ibm.com/8239 I'm still investigating to figure out what is the reason behind this behavior. Thanks, -- maurosr == Comment: #23 - Mauricio Faria De Oliveira - 2017-01-11 11:56:40 == Mauro, (In reply to comment #22) > I managed to get kdump to mount rootfs and perform its tasks by setting > KDUMP_CMDLINE_APPEND="nr_cpus=4" parameter in /etc/default/kdump-tools see > http://pastebin.hursley.ibm.com/8239 > > I'm still investigating to figure out what is the reason behind this > behavior. > > Thanks, > > -- > maurosr That would smell like an out of memory condition that is alleviated with a smaller number of CPUs allowed for the kernel (so the amount of memory associated with per-CPU stuff is less in total). Per the bug description, the memory reserved for the crashkernel is 512MB: (In reply to comment #23) > - Configure kexec-tools to reserve sufficient RAM for the kdump kernel to > load (I use 512MB) in /etc/default/grub.d/kexec-tools.cfg That seems low for Power guests/systems. I think it theory is doesn't seem so, but the reality is that _for some reason(s)_ we require just too much memory to load and boot a kernel/initramfs (either on boot or kdump). When working w/ kdump and Ubuntu, I usually set the crashkernel allocated size right away to 4GB to avoid problems. Since this is a smaller sized guest, obviously we'd want to use less than that, but more than 512 MB given the evidence observed. Hope this helps
[Kernel-packages] [Bug 1658733] Comment bridged from LTC Bugzilla
--- Comment From rsche...@us.ibm.com 2017-01-30 16:37 EDT--- I have tried the attached path to kdump-config, and it fixes the problem for me. Kdump to multipath root disk works correctly. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1658733 Title: Ubuntu 16.04.2KVM:kdump fails to mount root file system on multipath root device Status in kexec-tools package in Ubuntu: New Status in makedumpfile package in Ubuntu: New Bug description: == Comment: #0 - Richard M. Scheller - 2016-12-14 16:50:26 == ---Problem Description--- On a KVM guest installed to a multipath root device, the kdump kernel fails to mount the root file system. This error does not occur in a similar guest installed to a single path device. Full console output of the kdump failure is attached. These messages from the output may be relevant: Begin: Loading multipath modules ... Success: loaded module dm-multipath. done. Begin: Loading multipath hardware handlers ... Failure: failed to load module sc si_dh_alua. Failure: failed to load module scsi_dh_rdac. Failure: failed to load module scsi_dh_emc. done. Begin: Starting multipathd ... done. ---uname output--- Linux dotg9 4.8.0-32-generic #34~16.04.1-Ubuntu SMP Tue Dec 13 17:01:57 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux Machine Type = 8247-22L Ubuntu 16.04.1 KVM guest ---Steps to Reproduce--- - Install Ubuntu 16.04.1 to a muiltpath target disk - Install kdump-tools package - Configure kexec-tools to reserve sufficient RAM for the kdump kernel to load (I use 512MB) in /etc/default/grub.d/kexec-tools.cfg - Run update-grub - Reboot - Initiate a system crash using "echo c > /proc/sysrq-trigger" == Comment: #12 - Richard M. Scheller - 2016-12-20 20:37:45 == Here is the log level 8 kdump console log requested in comment 10. == Comment: #21 - Richard M. Scheller - 2017-01-06 11:04:17 == (In reply to comment #19) > Hi, I logged in dotkvm and I couldn't find the guest dotg9. Also, although I > found a dotg9.xml in /kte/xml/ it doesn't look like it uses multipath (it > uses .img files which I didn't found as disks). > > Could you please recreate the guest for further debug? Yes, I recreated the guest with its correct multipath lun configuration. I have also attached the guest XML to this bug. > Besides that could you please let us know: > - is the multipath the system's root? I mean / is installed/mounted on the > multipath device? Yes, the guest has only one disk. That disk is actually a LUN from a fiber channel storage device with two paths on the host side. I have passed through both paths to the guest, so the multipath nature of the target disk is known to the guest. In other words, the guest sees a multipath device and is using it as a multipath device. The root file system is called /dev/mapper/mpatha- part2 on the guest. > - how did you attach the device to the guest? Each FC LUN path on the host is mapped to a virtio-scsi controller on the guest using LUN passthrough. (See the guest XML for details on this.) == Comment: #22 - Mauro Sergio Martins Rodrigues - 2017-01-11 09:31:38 == I managed to get kdump to mount rootfs and perform its tasks by setting KDUMP_CMDLINE_APPEND="nr_cpus=4" parameter in /etc/default/kdump-tools see http://pastebin.hursley.ibm.com/8239 I'm still investigating to figure out what is the reason behind this behavior. Thanks, -- maurosr == Comment: #23 - Mauricio Faria De Oliveira - 2017-01-11 11:56:40 == Mauro, (In reply to comment #22) > I managed to get kdump to mount rootfs and perform its tasks by setting > KDUMP_CMDLINE_APPEND="nr_cpus=4" parameter in /etc/default/kdump-tools see > http://pastebin.hursley.ibm.com/8239 > > I'm still investigating to figure out what is the reason behind this > behavior. > > Thanks, > > -- > maurosr That would smell like an out of memory condition that is alleviated with a smaller number of CPUs allowed for the kernel (so the amount of memory associated with per-CPU stuff is less in total). Per the bug description, the memory reserved for the crashkernel is 512MB: (In reply to comment #23) > - Configure kexec-tools to reserve sufficient RAM for the kdump kernel to > load (I use 512MB) in /etc/default/grub.d/kexec-tools.cfg That seems low for Power guests/systems. I think it theory is doesn't seem so, but the reality is that _for some reason(s)_ we require just too much memory to load and boot a kernel/initramfs (either on boot or kdump). When working w/ kdump and Ubuntu, I usually set the crashkernel allocated size right away to 4GB to avoid problems. Since this is a smaller sized guest, obviously we'd want to use less than that, but more than 512 MB given the evidence
