[Kernel-packages] [Bug 1651199] Re: 16.04 - Kernel panic on docker server
Has this been fixed in any of the newer linux-image packages? I, unfortunately, can't offer to test since I was only seeing this on my production database server... For now I've just frozen my kernel version to 4.4.0-38. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1651199 Title: 16.04 - Kernel panic on docker server Status in Linux: New Status in linux package in Ubuntu: Confirmed Status in linux source package in Xenial: Confirmed Bug description: [31016.057405] BUG: unable to handle kernel paging request at 82c4801e6bda [31016.061249] IP: [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] PGD 0 [31016.061380] Oops: 0010 [#1] SMP [31016.061380] Modules linked in: binfmt_misc xt_REDIRECT nf_nat_redirect veth xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay xt_nat xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables isofs ppdev serio_raw parport_pc parport ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ixgbevf psmouse floppy [31016.061380] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.0-53-generic #74-Ubuntu [31016.061380] Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016 [31016.061380] task: 880406496040 ti: 8804064e task.ti: 8804064e [31016.061380] RIP: 0010:[] [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] RSP: 0018:88040fc43f78 EFLAGS: 00010082 [31016.061380] RAX: RBX: 88040fc4b840 RCX: 0001 [31016.061380] RDX: fffe RSI: 000123a0 RDI: 88040fc43f30 [31016.061380] RBP: 88040fc43f90 R08: f24a R09: 0001 [31016.061380] R10: 0001 R11: R12: 0001 [31016.061380] R13: 0003 R14: R15: 8804064e [31016.178419] FS: () GS:88040fc4() knlGS: [31016.178419] CS: 0010 DS: ES: CR0: 80050033 [31016.178419] CR2: 82c4801e6bda CR3: 000204f2f000 CR4: 001406e0 [31016.178419] DR0: DR1: DR2: [31016.178419] DR3: DR6: fffe0ff0 DR7: 0400 [31016.178419] Stack: [31016.178419] 0003 88040fc43fa8 [31016.178419] 814d6bc0 81f36a00 8804064e3e90 81837f32 [31016.178419] 8804064e3de8 8804064e [31016.178419] Call Trace: [31016.178419] [31016.178419] [] xen_evtchn_do_upcall+0x30/0x40 [31016.178419] [] xen_hvm_callback_vector+0x82/0x90 [31016.178419] [31016.178419] [] ? native_safe_halt+0x6/0x10 [31016.178419] [] default_idle+0x1e/0xe0 [31016.239315] [] arch_cpu_idle+0xf/0x20 [31016.239899] [] default_idle_call+0x2a/0x40 [31016.239899] [] cpu_startup_entry+0x2f1/0x350 [31016.239899] [] start_secondary+0x154/0x190 [31016.239899] Code: 01 00 00 00 65 44 8b 2d dc 56 b3 7e c6 03 00 44 89 e0 65 0f c1 05 2e d8 b3 7e 85 c0 75 35 48 8b 05 63 ce d0 00 44 89 ef ff 50 50 <9c> 58 0f 1f 44 00 00 f6 c4 02 75 23 65 8b 05 0a d8 b3 7e 65 c7 [31016.239899] RIP [] __xen_evtchn_do_upcall+0x43/0x80 [31016.239899] RSP [31016.239899] CR2: 82c4801e6bda [31016.239899] ---[ end trace 5b3e8ea32013e327 ]--- [31016.239899] Kernel panic - not syncing: Fatal exception in interrupt [31016.239899] Kernel Offset: disabled We believe this appeared in the last 1-2mo of releases, since this started happening after we did a `apt-get upgrade` on our machines after a bit of a pause. These are EC2 m4.xlarge servers running Ubuntu 16.04.1. They are Kubernetes minions, so I assume docker is likely the trigger. Unfortunately there aren't really any interesting logs in journalctl from the previous boot prior to the panic. Let me know what I can do to debug further. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-53-generic 4.4.0-53.74 ProcVersionSignature: Ubuntu 4.4.0-53.74-generic 4.4.30 Uname: Linux 4.4.0-53-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Dec 19 15:56 seq crw-rw 1 root audio 116, 33 Dec 19 15:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such
[Kernel-packages] [Bug 1651199] Re: 16.04 - Kernel panic on docker server
BTW, I'm seeing this same bug (same symptom at least; the exact same call stack in the kernel panic) and I'm *not* using docker. The main workload on the machine where I'm seeing this is Postgres. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1651199 Title: 16.04 - Kernel panic on docker server Status in Linux: New Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Incomplete Bug description: [31016.057405] BUG: unable to handle kernel paging request at 82c4801e6bda [31016.061249] IP: [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] PGD 0 [31016.061380] Oops: 0010 [#1] SMP [31016.061380] Modules linked in: binfmt_misc xt_REDIRECT nf_nat_redirect veth xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay xt_nat xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables isofs ppdev serio_raw parport_pc parport ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ixgbevf psmouse floppy [31016.061380] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.0-53-generic #74-Ubuntu [31016.061380] Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016 [31016.061380] task: 880406496040 ti: 8804064e task.ti: 8804064e [31016.061380] RIP: 0010:[] [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] RSP: 0018:88040fc43f78 EFLAGS: 00010082 [31016.061380] RAX: RBX: 88040fc4b840 RCX: 0001 [31016.061380] RDX: fffe RSI: 000123a0 RDI: 88040fc43f30 [31016.061380] RBP: 88040fc43f90 R08: f24a R09: 0001 [31016.061380] R10: 0001 R11: R12: 0001 [31016.061380] R13: 0003 R14: R15: 8804064e [31016.178419] FS: () GS:88040fc4() knlGS: [31016.178419] CS: 0010 DS: ES: CR0: 80050033 [31016.178419] CR2: 82c4801e6bda CR3: 000204f2f000 CR4: 001406e0 [31016.178419] DR0: DR1: DR2: [31016.178419] DR3: DR6: fffe0ff0 DR7: 0400 [31016.178419] Stack: [31016.178419] 0003 88040fc43fa8 [31016.178419] 814d6bc0 81f36a00 8804064e3e90 81837f32 [31016.178419] 8804064e3de8 8804064e [31016.178419] Call Trace: [31016.178419] [31016.178419] [] xen_evtchn_do_upcall+0x30/0x40 [31016.178419] [] xen_hvm_callback_vector+0x82/0x90 [31016.178419] [31016.178419] [] ? native_safe_halt+0x6/0x10 [31016.178419] [] default_idle+0x1e/0xe0 [31016.239315] [] arch_cpu_idle+0xf/0x20 [31016.239899] [] default_idle_call+0x2a/0x40 [31016.239899] [] cpu_startup_entry+0x2f1/0x350 [31016.239899] [] start_secondary+0x154/0x190 [31016.239899] Code: 01 00 00 00 65 44 8b 2d dc 56 b3 7e c6 03 00 44 89 e0 65 0f c1 05 2e d8 b3 7e 85 c0 75 35 48 8b 05 63 ce d0 00 44 89 ef ff 50 50 <9c> 58 0f 1f 44 00 00 f6 c4 02 75 23 65 8b 05 0a d8 b3 7e 65 c7 [31016.239899] RIP [] __xen_evtchn_do_upcall+0x43/0x80 [31016.239899] RSP [31016.239899] CR2: 82c4801e6bda [31016.239899] ---[ end trace 5b3e8ea32013e327 ]--- [31016.239899] Kernel panic - not syncing: Fatal exception in interrupt [31016.239899] Kernel Offset: disabled We believe this appeared in the last 1-2mo of releases, since this started happening after we did a `apt-get upgrade` on our machines after a bit of a pause. These are EC2 m4.xlarge servers running Ubuntu 16.04.1. They are Kubernetes minions, so I assume docker is likely the trigger. Unfortunately there aren't really any interesting logs in journalctl from the previous boot prior to the panic. Let me know what I can do to debug further. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-53-generic 4.4.0-53.74 ProcVersionSignature: Ubuntu 4.4.0-53.74-generic 4.4.30 Uname: Linux 4.4.0-53-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Dec 19 15:56 seq crw-rw 1 root audio 116, 33 Dec 19 15:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory:
[Kernel-packages] [Bug 1651199] Re: 16.04 - Kernel panic on docker server
I'm seeing the same thing on an m4.16xlarge EC2 instance. It has crashed twice in the past 24 hours. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1651199 Title: 16.04 - Kernel panic on docker server Status in Linux: New Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Incomplete Bug description: [31016.057405] BUG: unable to handle kernel paging request at 82c4801e6bda [31016.061249] IP: [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] PGD 0 [31016.061380] Oops: 0010 [#1] SMP [31016.061380] Modules linked in: binfmt_misc xt_REDIRECT nf_nat_redirect veth xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay xt_nat xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables isofs ppdev serio_raw parport_pc parport ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ixgbevf psmouse floppy [31016.061380] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.0-53-generic #74-Ubuntu [31016.061380] Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016 [31016.061380] task: 880406496040 ti: 8804064e task.ti: 8804064e [31016.061380] RIP: 0010:[] [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] RSP: 0018:88040fc43f78 EFLAGS: 00010082 [31016.061380] RAX: RBX: 88040fc4b840 RCX: 0001 [31016.061380] RDX: fffe RSI: 000123a0 RDI: 88040fc43f30 [31016.061380] RBP: 88040fc43f90 R08: f24a R09: 0001 [31016.061380] R10: 0001 R11: R12: 0001 [31016.061380] R13: 0003 R14: R15: 8804064e [31016.178419] FS: () GS:88040fc4() knlGS: [31016.178419] CS: 0010 DS: ES: CR0: 80050033 [31016.178419] CR2: 82c4801e6bda CR3: 000204f2f000 CR4: 001406e0 [31016.178419] DR0: DR1: DR2: [31016.178419] DR3: DR6: fffe0ff0 DR7: 0400 [31016.178419] Stack: [31016.178419] 0003 88040fc43fa8 [31016.178419] 814d6bc0 81f36a00 8804064e3e90 81837f32 [31016.178419] 8804064e3de8 8804064e [31016.178419] Call Trace: [31016.178419] [31016.178419] [] xen_evtchn_do_upcall+0x30/0x40 [31016.178419] [] xen_hvm_callback_vector+0x82/0x90 [31016.178419] [31016.178419] [] ? native_safe_halt+0x6/0x10 [31016.178419] [] default_idle+0x1e/0xe0 [31016.239315] [] arch_cpu_idle+0xf/0x20 [31016.239899] [] default_idle_call+0x2a/0x40 [31016.239899] [] cpu_startup_entry+0x2f1/0x350 [31016.239899] [] start_secondary+0x154/0x190 [31016.239899] Code: 01 00 00 00 65 44 8b 2d dc 56 b3 7e c6 03 00 44 89 e0 65 0f c1 05 2e d8 b3 7e 85 c0 75 35 48 8b 05 63 ce d0 00 44 89 ef ff 50 50 <9c> 58 0f 1f 44 00 00 f6 c4 02 75 23 65 8b 05 0a d8 b3 7e 65 c7 [31016.239899] RIP [] __xen_evtchn_do_upcall+0x43/0x80 [31016.239899] RSP [31016.239899] CR2: 82c4801e6bda [31016.239899] ---[ end trace 5b3e8ea32013e327 ]--- [31016.239899] Kernel panic - not syncing: Fatal exception in interrupt [31016.239899] Kernel Offset: disabled We believe this appeared in the last 1-2mo of releases, since this started happening after we did a `apt-get upgrade` on our machines after a bit of a pause. These are EC2 m4.xlarge servers running Ubuntu 16.04.1. They are Kubernetes minions, so I assume docker is likely the trigger. Unfortunately there aren't really any interesting logs in journalctl from the previous boot prior to the panic. Let me know what I can do to debug further. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-53-generic 4.4.0-53.74 ProcVersionSignature: Ubuntu 4.4.0-53.74-generic 4.4.30 Uname: Linux 4.4.0-53-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Dec 19 15:56 seq crw-rw 1 root audio 116, 33 Dec 19 15:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed
