** Description changed:

  SRU Justification:
  
  [Impact]
  Users of ppc64el hardware need the ability to use crashdumps to do kernel 
debugging.
  
  [Fix]
  Commit upstream and already in utopic:
  
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=429d2e8342954d337abe370d957e78291032d867
  
  [Test Case]
  Taken from:
  https://wiki.ubuntu.com/Kernel/CrashdumpRecipe
  https://help.ubuntu.com/14.04/serverguide/kernel-crash-dump.html
  
  1) apt-get install linux-crashdump
- 2) reboot the machine
- 3) sudo sed -i 's/USE_KDUMP=0/USE_KDUMP=1/g' /etc/default/kdump-tools
- 4) kdump-config show # should return no errors
- 5) echo 'c' | sudo tee /proc/sysrq-trigger
- 6) This should crash the machine and we should kexec into another kernel to 
dump the core, then on the next reboot we should see a crash in /var/crash/*
+ 2) increase crashdump size:
+ sudo vim /etc/default/grub.d/kexec-tools.cfg
+ set crashkernel=1024M
+ sudo update-grub
+ 3) reboot the machine
+ 4) sudo sed -i 's/USE_KDUMP=0/USE_KDUMP=1/g' /etc/default/kdump-tools
+ 5) kdump-config show # should return no errors
+ 6) echo 'c' | sudo tee /proc/sysrq-trigger
+ 7) This should crash the machine and we should kexec into another kernel to 
dump the core, then on the next reboot we should see a crash in /var/crash/*
  
  --
- 
  
  ---Problem Description---
  kdump is not producing a dump on powerKVM LE P8 Ubuntu 14.04
  
  ---uname output---
  3.13.0-30-generic
  
  ---Additional Hardware Info---
  Power8 LE configuration.
  
  ---Patches Installed---
  1324544 - kdump-config load fails with vmlinux kernel (vs. vmlinuz)
  
  Machine Type = 8247-22L
  
  ---Steps to Reproduce---
  Installed kdump-tools 1.5.5-2ubuntu1 and crash 7.0.3-3ubuntu3.
  Updated /etc/default/kdump-tools, first I updated just USE_KDUMP=1. Rebooted 
the node and see:
  root=UUID=87986483-5fec-4b4d-b22e-bf2a72096df8 ro quiet splash 
crashkernel=384M-:128M
  root@c656f2n02:~# cat /proc/sys/kernel/sysrq
  1
  root@c656f2n02:~# cat /proc/sys/kernel/sysrq
  1
  root@c656f2n02:~# ^Cnd /proc | grep sysrq
  root@c656f2n02:~# kdump-config status
  current state   : ready to kdump
  root@c656f2n02:~# kdump-config show
  USE_KDUMP:        1
  KDUMP_SYSCTL:     kernel.panic_on_oops=1
  KDUMP_COREDIR:    /var/crash
  crashkernel addr:
  current state:    ready to kdump
  
  kexec command:
    /sbin/kexec -p --args-linux 
--command-line="root=UUID=87986483-5fec-4b4d-b22e-bf2a72096df8 ro quiet splash  
irqpoll maxcpus=1 nousb" --initrd=/boot/initrd.img-3.13.0-30-generic 
/boot/vmlinux-3.13.0-30-generic
  
  root@c656f2n02:/boot/grub# cat /sys/kernel/kexec_crash_loaded
  1
  root@c656f2n02:/boot/grub# cat /sys/kernel/kexec_loaded
  0
  
  echo c > /proc/sysrq-trigger
  
  root@c656f2n02:/var/log# echo c > /proc/sysrq-trigger
  [ 1956.014243] SysRq : Trigger a crash
  [ 1956.014328] Unable to handle kernel paging request for data at address 
0x00000000
  [ 1956.014404] Faulting instruction address: 0xc000000000586c2c
  [ 1956.014468] Oops: Kernel access of bad area, sig: 11 [#1]
  [ 1956.014518] SMP NR_CPUS=2048 NUMA PowerNV
  [ 1956.014570] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 
nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT 
xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc ip6table_filter ip6_tables 
iptable_filter ip_tables ebtable_nat ebtables x_tables autofs4 rdma_ucm(OF) 
ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) 
ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) 
ib_core(OF) ib_addr(OF) mlx4_en(OF) mlx4_core(OF) compat(OF) nfsd auth_rpcgss 
nfs_acl nfs lockd sunrpc fscache rtc_generic powernv_rng ses enclosure ipr
  [ 1956.015306] CPU: 146 PID: 2522 Comm: bash Tainted: GF          O 
3.13.0-30-generic #54-Ubuntu
  [ 1956.015394] task: c000003fcabda120 ti: c000003fcac58000 task.ti: 
c000003fcac58000
  [ 1956.015469] NIP: c000000000586c2c LR: c000000000587b8c CTR: 
c000000000586c00
  [ 1956.015543] REGS: c000003fcac5b820 TRAP: 0300   Tainted: GF          O  
(3.13.0-30-generic)
  [ 1956.015617] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 42422822  
XER: 20000000
  [ 1956.015804] CFAR: c000000000009318 DAR: 0000000000000000 DSISR: 42000000 
SOFTE: 0
  GPR00: c000000000587b8c c000003fcac5baa0 c00000000162e840 0000000000000063
  GPR04: c000000002f45bd0 c000000002f564c8 0000000000015ad0 c000000001827480
  GPR08: c000000000dfe840 0000000000000000 0000000000000001 0000000000015ad0
  GPR12: 0000000042422822 c000000007e5ff00 000001002fe90648 000000001016e008
  GPR16: 000000001013ad70 000001002fe94648 000000001016fed0 000000001016e008
  GPR20: 00000000100c31e0 0000000000000000 0000000010171fc8 000000001016f840
  GPR24: 0000000000000004 0000000000000000 0000000000000001 c0000000014b7dc8
  GPR28: c000000001974c90 0000000000000063 c00000000148d9c0 c0000000014b8188
  [ 1956.016794] NIP [c000000000586c2c] .sysrq_handle_crash+0x2c/0x40
  [ 1956.016858] LR [c000000000587b8c] .__handle_sysrq+0xfc/0x260
  [ 1956.016920] Call Trace:
  [ 1956.016948] [c000003fcac5baa0] [0000000010172a34] 0x10172a34 (unreliable)
  [ 1956.017025] [c000003fcac5bb10] [c000000000587b8c] 
.__handle_sysrq+0xfc/0x260
  [ 1956.017101] [c000003fcac5bbd0] [c000000000588324] 
.write_sysrq_trigger+0x74/0x90
  [ 1956.017190] [c000003fcac5bc50] [c0000000002dff1c] 
.proc_reg_write+0xac/0x110
  [ 1956.017266] [c000003fcac5bcf0] [c000000000254c00] .vfs_write+0xe0/0x260
  [ 1956.017342] [c000003fcac5bd90] [c0000000002558f4] .SyS_write+0x64/0xe0
  [ 1956.017418] [c000003fcac5be30] [c00000000000a158] syscall_exit+0x0/0x98
  [ 1956.017492] Instruction dump:
  [ 1956.017530] 4bffffac 7c0802a6 f8010010 f821ff91 60000000 60000000 3d42001f 
392a8ca8
  [ 1956.017658] 39400001 91490000 7c0004ac 39200000 <99490000> 38210070 
e8010010 7c0803a6
  [ 1956.017894] ---[ end trace d163ff42366bde72 ]---
  [ 1956.017986]
  [ 1956.018042] Sending IPI to other CPUs
  [ 1956.019188] IPI complete
   -> smp_release_cpus()
  spinning_secondaries = 159
   <- smp_release_cpus()
   <- setup_system()
  The console stays remains at this message until I power cycle the cec. There 
is no /proc/vmcore on reboot.
  
  I recreated the hang on my victim node.
  Some CPUs are hitting the 4400's interrupt vector. I think this is due to the 
commit 429d2e834295 "powerpc: Fix kdump hang issue on p8 with relocation on 
exception enabled." from Mahesh but I need to double check that since it may 
not be only patch missing.
  
  Definitively, the patch I mentioned is fixing the hang.
  Here are the commit details :
  
  
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=429d2e8342954d337abe370d957e78291032d867
  
  powerpc: Fix kdump hang issue on p8 with relocation on exception
  enabled.
  
  On p8 systems, with relocation on exception feature enabled we are seeing
  kdump kernel hang at interrupt vector 0xc*4400. The reason is, with this
  feature enabled, exception are raised with MMU (IR=DR=1) ON with the
  default offset of 0xc*4000. Since exception is raised in virtual mode it
  requires the vector region to be executable without which it fails to
  fetch and execute instruction at 0xc*4xxx. For default kernel since kernel
  is loaded at real 0, the htab mappings sets the entire kernel text region
  executable. But for relocatable kernel (e.g. kdump case) we only copy
  interrupt vectors down to real 0 and never marked that region as
  executable because in p7 and below we always get exception in real mode.
  
  This patch fixes this issue by marking htab mapping range as executable
  that overlaps with the interrupt vector region for relocatable kernel.
  
  Thanks to Ben who helped me to debug this issue and find the root cause.
  
  Signed-off-by: Mahesh Salgaonkar <mah...@linux.vnet.ibm.com>
  Signed-off-by: Benjamin Herrenschmidt <b...@kernel.crashing.org>
  
  I think this bug should be mirrored to Ubuntu so they can include this
  patch in the 14.04 kernel, and may be also in the 14.10 kernel too.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1352056

Title:
  linux: kdump on Ubuntu 14.04 is not generating a dump.

Status in “linux” package in Ubuntu:
  Fix Released
Status in “linux” source package in Trusty:
  In Progress
Status in “linux” source package in Utopic:
  Fix Released

Bug description:
  SRU Justification:

  [Impact]
  Users of ppc64el hardware need the ability to use crashdumps to do kernel 
debugging.

  [Fix]
  Commit upstream and already in utopic:
  
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=429d2e8342954d337abe370d957e78291032d867

  [Test Case]
  Taken from:
  https://wiki.ubuntu.com/Kernel/CrashdumpRecipe
  https://help.ubuntu.com/14.04/serverguide/kernel-crash-dump.html

  1) apt-get install linux-crashdump
  2) increase crashdump size:
  sudo vim /etc/default/grub.d/kexec-tools.cfg
  set crashkernel=1024M
  sudo update-grub
  3) reboot the machine
  4) sudo sed -i 's/USE_KDUMP=0/USE_KDUMP=1/g' /etc/default/kdump-tools
  5) kdump-config show # should return no errors
  6) echo 'c' | sudo tee /proc/sysrq-trigger
  7) This should crash the machine and we should kexec into another kernel to 
dump the core, then on the next reboot we should see a crash in /var/crash/*

  --

  ---Problem Description---
  kdump is not producing a dump on powerKVM LE P8 Ubuntu 14.04

  ---uname output---
  3.13.0-30-generic

  ---Additional Hardware Info---
  Power8 LE configuration.

  ---Patches Installed---
  1324544 - kdump-config load fails with vmlinux kernel (vs. vmlinuz)

  Machine Type = 8247-22L

  ---Steps to Reproduce---
  Installed kdump-tools 1.5.5-2ubuntu1 and crash 7.0.3-3ubuntu3.
  Updated /etc/default/kdump-tools, first I updated just USE_KDUMP=1. Rebooted 
the node and see:
  root=UUID=87986483-5fec-4b4d-b22e-bf2a72096df8 ro quiet splash 
crashkernel=384M-:128M
  root@c656f2n02:~# cat /proc/sys/kernel/sysrq
  1
  root@c656f2n02:~# cat /proc/sys/kernel/sysrq
  1
  root@c656f2n02:~# ^Cnd /proc | grep sysrq
  root@c656f2n02:~# kdump-config status
  current state   : ready to kdump
  root@c656f2n02:~# kdump-config show
  USE_KDUMP:        1
  KDUMP_SYSCTL:     kernel.panic_on_oops=1
  KDUMP_COREDIR:    /var/crash
  crashkernel addr:
  current state:    ready to kdump

  kexec command:
    /sbin/kexec -p --args-linux 
--command-line="root=UUID=87986483-5fec-4b4d-b22e-bf2a72096df8 ro quiet splash  
irqpoll maxcpus=1 nousb" --initrd=/boot/initrd.img-3.13.0-30-generic 
/boot/vmlinux-3.13.0-30-generic

  root@c656f2n02:/boot/grub# cat /sys/kernel/kexec_crash_loaded
  1
  root@c656f2n02:/boot/grub# cat /sys/kernel/kexec_loaded
  0

  echo c > /proc/sysrq-trigger

  root@c656f2n02:/var/log# echo c > /proc/sysrq-trigger
  [ 1956.014243] SysRq : Trigger a crash
  [ 1956.014328] Unable to handle kernel paging request for data at address 
0x00000000
  [ 1956.014404] Faulting instruction address: 0xc000000000586c2c
  [ 1956.014468] Oops: Kernel access of bad area, sig: 11 [#1]
  [ 1956.014518] SMP NR_CPUS=2048 NUMA PowerNV
  [ 1956.014570] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 
nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT 
xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc ip6table_filter ip6_tables 
iptable_filter ip_tables ebtable_nat ebtables x_tables autofs4 rdma_ucm(OF) 
ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) 
ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) 
ib_core(OF) ib_addr(OF) mlx4_en(OF) mlx4_core(OF) compat(OF) nfsd auth_rpcgss 
nfs_acl nfs lockd sunrpc fscache rtc_generic powernv_rng ses enclosure ipr
  [ 1956.015306] CPU: 146 PID: 2522 Comm: bash Tainted: GF          O 
3.13.0-30-generic #54-Ubuntu
  [ 1956.015394] task: c000003fcabda120 ti: c000003fcac58000 task.ti: 
c000003fcac58000
  [ 1956.015469] NIP: c000000000586c2c LR: c000000000587b8c CTR: 
c000000000586c00
  [ 1956.015543] REGS: c000003fcac5b820 TRAP: 0300   Tainted: GF          O  
(3.13.0-30-generic)
  [ 1956.015617] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 42422822  
XER: 20000000
  [ 1956.015804] CFAR: c000000000009318 DAR: 0000000000000000 DSISR: 42000000 
SOFTE: 0
  GPR00: c000000000587b8c c000003fcac5baa0 c00000000162e840 0000000000000063
  GPR04: c000000002f45bd0 c000000002f564c8 0000000000015ad0 c000000001827480
  GPR08: c000000000dfe840 0000000000000000 0000000000000001 0000000000015ad0
  GPR12: 0000000042422822 c000000007e5ff00 000001002fe90648 000000001016e008
  GPR16: 000000001013ad70 000001002fe94648 000000001016fed0 000000001016e008
  GPR20: 00000000100c31e0 0000000000000000 0000000010171fc8 000000001016f840
  GPR24: 0000000000000004 0000000000000000 0000000000000001 c0000000014b7dc8
  GPR28: c000000001974c90 0000000000000063 c00000000148d9c0 c0000000014b8188
  [ 1956.016794] NIP [c000000000586c2c] .sysrq_handle_crash+0x2c/0x40
  [ 1956.016858] LR [c000000000587b8c] .__handle_sysrq+0xfc/0x260
  [ 1956.016920] Call Trace:
  [ 1956.016948] [c000003fcac5baa0] [0000000010172a34] 0x10172a34 (unreliable)
  [ 1956.017025] [c000003fcac5bb10] [c000000000587b8c] 
.__handle_sysrq+0xfc/0x260
  [ 1956.017101] [c000003fcac5bbd0] [c000000000588324] 
.write_sysrq_trigger+0x74/0x90
  [ 1956.017190] [c000003fcac5bc50] [c0000000002dff1c] 
.proc_reg_write+0xac/0x110
  [ 1956.017266] [c000003fcac5bcf0] [c000000000254c00] .vfs_write+0xe0/0x260
  [ 1956.017342] [c000003fcac5bd90] [c0000000002558f4] .SyS_write+0x64/0xe0
  [ 1956.017418] [c000003fcac5be30] [c00000000000a158] syscall_exit+0x0/0x98
  [ 1956.017492] Instruction dump:
  [ 1956.017530] 4bffffac 7c0802a6 f8010010 f821ff91 60000000 60000000 3d42001f 
392a8ca8
  [ 1956.017658] 39400001 91490000 7c0004ac 39200000 <99490000> 38210070 
e8010010 7c0803a6
  [ 1956.017894] ---[ end trace d163ff42366bde72 ]---
  [ 1956.017986]
  [ 1956.018042] Sending IPI to other CPUs
  [ 1956.019188] IPI complete
   -> smp_release_cpus()
  spinning_secondaries = 159
   <- smp_release_cpus()
   <- setup_system()
  The console stays remains at this message until I power cycle the cec. There 
is no /proc/vmcore on reboot.

  I recreated the hang on my victim node.
  Some CPUs are hitting the 4400's interrupt vector. I think this is due to the 
commit 429d2e834295 "powerpc: Fix kdump hang issue on p8 with relocation on 
exception enabled." from Mahesh but I need to double check that since it may 
not be only patch missing.

  Definitively, the patch I mentioned is fixing the hang.
  Here are the commit details :

  
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=429d2e8342954d337abe370d957e78291032d867

  powerpc: Fix kdump hang issue on p8 with relocation on exception
  enabled.

  On p8 systems, with relocation on exception feature enabled we are seeing
  kdump kernel hang at interrupt vector 0xc*4400. The reason is, with this
  feature enabled, exception are raised with MMU (IR=DR=1) ON with the
  default offset of 0xc*4000. Since exception is raised in virtual mode it
  requires the vector region to be executable without which it fails to
  fetch and execute instruction at 0xc*4xxx. For default kernel since kernel
  is loaded at real 0, the htab mappings sets the entire kernel text region
  executable. But for relocatable kernel (e.g. kdump case) we only copy
  interrupt vectors down to real 0 and never marked that region as
  executable because in p7 and below we always get exception in real mode.

  This patch fixes this issue by marking htab mapping range as executable
  that overlaps with the interrupt vector region for relocatable kernel.

  Thanks to Ben who helped me to debug this issue and find the root
  cause.

  Signed-off-by: Mahesh Salgaonkar <mah...@linux.vnet.ibm.com>
  Signed-off-by: Benjamin Herrenschmidt <b...@kernel.crashing.org>

  I think this bug should be mirrored to Ubuntu so they can include this
  patch in the 14.04 kernel, and may be also in the 14.10 kernel too.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1352056/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to