[Kernel-packages] [Bug 1573231] Re: Kernel Panic on EC2 After Upgrading from 14.04 to 16.04 via do-release-upgrade -d
The Ubuntu 14.04 cloud images remain broken on m3.large and c3.large. Is there a way to expedite building them with the fix? It seems that the Ubuntu release process is broken, if bugs like this make it into the LTS images and then don't get a clear timeline or priority for getting fixed. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1573231 Title: Kernel Panic on EC2 After Upgrading from 14.04 to 16.04 via do- release-upgrade -d Status in linux package in Ubuntu: Fix Released Status in linux source package in Xenial: Fix Committed Status in linux source package in Yakkety: Fix Released Bug description: [0.00] Initializing cgroup subsys cpuset [0.00] Initializing cgroup subsys cpu [0.00] Initializing cgroup subsys cpuacct [0.00] Linux version 4.4.0-21-generic (buildd@lgw01-21) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2) ) #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 (Ubuntu 4.4.0-21.37-generic 4.4.6) [0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-21-generic root=UUID=8ea401db-b84b-4cd6-a628-d72f30bbf1e5 ro console=tty1 console=ttyS0 [0.00] KERNEL supported cpus: [0.00] Intel GenuineIntel [0.00] AMD AuthenticAMD [0.00] Centaur CentaurHauls [0.00] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [0.00] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating point registers' [0.00] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers' [0.00] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers' [0.00] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [0.00] x86/fpu: Using 'eager' FPU context switches. [0.00] e820: BIOS-provided physical RAM map: [0.00] BIOS-e820: [mem 0x-0x0009dfff] usable [0.00] BIOS-e820: [mem 0x0009e000-0x0009] reserved [0.00] BIOS-e820: [mem 0x000e-0x000f] reserved [0.00] BIOS-e820: [mem 0x0010-0xefff] usable [0.00] BIOS-e820: [mem 0xfc00-0x] reserved [0.00] BIOS-e820: [mem 0x0001-0x0001efff] usable [0.00] NX (Execute Disable) protection: active [0.00] SMBIOS 2.4 present. [0.00] Hypervisor detected: Xen [0.00] Xen version 4.2. [0.00] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs. [0.00] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks. [0.00] You might have to change the root device [0.00] from /dev/hd[a-d] to /dev/xvd[a-d] [0.00] in your root= kernel command line option [0.00] e820: last_pfn = 0x1f max_arch_pfn = 0x4 [0.00] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT [0.00] e820: last_pfn = 0xf max_arch_pfn = 0x4 [0.00] found SMP MP-table at [mem 0x000fbba0-0x000fbbaf] mapped at [880fbba0] [0.00] Scanning 1 areas for low memory corruption [0.00] RAMDISK: [mem 0x3407e000-0x36036fff] [0.00] ACPI: Early table checksum verification disabled [0.00] ACPI: RSDP 0x000EA020 24 (v02 Xen ) [0.00] ACPI: XSDT 0xFC00F5A0 54 (v01 XenHVM HVML ) [0.00] ACPI: FACP 0xFC00F260 F4 (v04 XenHVM HVML ) [0.00] ACPI: DSDT 0xFC0035E0 00BBF6 (v02 XenHVM INTL 20090123) [0.00] ACPI: FACS 0xFC0035A0 40 [0.00] ACPI: FACS 0xFC0035A0 40 [0.00] ACPI: APIC 0xFC00F360 D8 (v02 XenHVM HVML ) [0.00] ACPI: HPET 0xFC00F4B0 38 (v01 XenHVM HVML ) [0.00] ACPI: WAET 0xFC00F4F0 28 (v01 XenHVM HVML ) [0.00] ACPI: SSDT 0xFC00F520 31 (v02 XenHVM INTL 20090123) [0.00] ACPI: SSDT 0xFC00F560 31 (v02 XenHVM INTL 20090123) [0.00] No NUMA configuration found [0.00] Faking a node at [mem 0x-0x0001efff] [0.00] NODE_DATA(0) allocated [mem 0x1efff7000-0x1efffbfff] [0.00] Zone ranges: [0.00] DMA [mem 0x1000-0x00ff] [0.00] DMA32[mem 0x0100-0x] [0.00] Normal [mem 0x0001-0x0001efff] [0.00] Device empty [0.00] Movable zone start for each node [0.00] Early
[Kernel-packages] [Bug 1350373] Re: Kernel BUG in paravirt_enter_lazy_mmu when running as a Xen PV guest
Is there an estimate for when this will hit release and cloud images? We are blocked by this, and I don't see a reference to it in https://launchpad.net/ubuntu/+source/linux. Thanks. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1350373 Title: Kernel BUG in paravirt_enter_lazy_mmu when running as a Xen PV guest Status in “linux” package in Ubuntu: Triaged Status in “linux” source package in Trusty: Triaged Bug description: Xen PV guests may crash during boot in paravirt_enter_lazy_mmu() while expanding the grant table (usually when requested by blkfront, when booting). See an example trace below. This is caused by calling functions that are unsafe in atomic context. The fix (which has been submitted to 3.16) is available here (also attached): https://git.kernel.org/cgit/linux/kernel/git/xen/tip.git/commit/?h=stable /for-linus-3.16id=b7dd0e350e0bd4c0fddcc9b8958342700b00b168 The fix is applicable to all kernel since 2.6.39 but only appears to trigger with the 3.13 kernel in 14.04. [2.577876] [ cut here ] [2.577896] kernel BUG at /build/buildd/linux-3.13.0/arch/x86/kernel/paravirt.c:239! [2.577910] invalid opcode: [#1] SMP [2.577922] Modules linked in: [2.577933] CPU: 0 PID: 1 Comm: init Not tainted 3.13.0-24-generic #46-Ubuntu [2.577946] task: ec058000 ti: ec09 task.ti: ec09 [2.577955] EIP: 0061:[c1645ebc] EFLAGS: 00010002 CPU: 0 [2.577973] EIP is at enter_lazy.part.1+0x3/0x5 [2.577982] EAX: 0001 EBX: ec0cc000 ECX: 00581980 EDX: [2.577992] ESI: edc0 EDI: edc0 EBP: ec091a50 ESP: ec091a50 [2.578001] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069 [2.578014] CR0: 8005003b CR2: bfca2fa4 CR3: 2c392000 CR4: 2660 [2.578027] Stack: [2.578032] ec091a58 c1046564 ec091ab4 c1146fd3 fa83b2da edc00fff edc01000 c1a93018 [2.578052] edc00fff c193ce80 edc01000 [2.578076] ed3ef588 ed3ef588 c1b87b70 ec091ad0 edc01000 c1b65310 1000 [2.578096] Call Trace: [2.578111] [c1046564] paravirt_enter_lazy_mmu+0x24/0x30 [2.578127] [c1146fd3] apply_to_page_range+0x1a3/0x310 [2.578141] [c1008eb8] arch_gnttab_map_status+0x38/0x60 [2.578152] [c1008d70] ? map_pte_fn+0x70/0x70 [2.578166] [c13ab020] gnttab_map_frames_v2+0xb0/0x100 [2.578182] [c13ab205] gnttab_map+0x95/0x120 [2.578198] [c12c7ff0] ? blk_update_request+0x190/0x340 [2.578209] [c13ab363] get_free_entries+0xd3/0x280 [2.578221] [c13ab5d3] gnttab_alloc_grant_references+0x13/0x30 [2.578238] [c1424be5] do_blkif_request+0x535/0x6f0 [2.578253] [c16523dc] ? _raw_spin_unlock_irqrestore+0x1c/0x40 [2.578269] [c12c57ee] __blk_run_queue+0x2e/0x40 [2.578280] [c12c5825] blk_start_queue+0x25/0x40 [2.578291] [c1424dbe] kick_pending_request_queues+0x1e/0x30 [2.578304] [c142546f] blkif_interrupt+0x69f/0x740 [2.578318] [c100654f] ? xen_set_pte_at+0xbf/0xf0 [2.578335] [c10a5ba5] handle_irq_event_percpu+0x35/0x1a0 [2.578351] [c12f136a] ? radix_tree_lookup+0xa/0x10 [2.578364] [c10a5d41] handle_irq_event+0x31/0x50 [2.578376] [c10a8036] handle_edge_irq+0x66/0x110 [2.578389] [c13ac246] __xen_evtchn_do_upcall+0x1c6/0x2c0 [2.578402] [c13ae100] xen_evtchn_do_upcall+0x20/0x40 [2.578415] [c165a087] xen_do_upcall+0x7/0xc [2.578427] [c1001227] ? xen_hypercall_xen_version+0x7/0x20 [2.578441] [c10083cf] ? xen_force_evtchn_callback+0xf/0x20 [2.578454] [c1008c50] check_events+0x8/0xc [2.578464] [c1008c47] ? xen_restore_fl_direct_reloc+0x4/0x4 [2.578480] [c1006373] ? xen_batched_set_pte+0xb3/0x160 [2.578493] [c10064b8] xen_set_pte_at+0x28/0xf0 [2.578505] [c10048e6] ? __raw_callee_save_xen_pte_val+0x6/0x8 [2.578521] [c11447a8] copy_pte_range+0x258/0x4c0 [2.578534] [c1146d27] copy_page_range+0x1d7/0x2e0 [2.578549] [c105462e] dup_mm+0x28e/0x4f0 [2.578561] [c1055866] copy_process.part.33+0xfa6/0x10d0 [2.578574] [c1055b41] do_fork+0xc1/0x2c0 [2.578591] [c1067996] ? SyS_rt_sigprocmask+0x76/0xa0 [2.578604] [c1055e05] SyS_clone+0x25/0x30 [2.578615] [c1659b4d] sysenter_do_call+0x12/0x28 [2.578626] Code: c4 1c 5b 5e 5f 5d c3 55 89 e5 f3 0f b8 c0 90 5d c3 55 ba a0 2c aa c1 89 e5 b9 25 00 00 00 57 31 c0 89 d7 f3 ab 5f 5d c3 55 89 e5 0f 0b 55 89 e5 66 66 66 66 90 0f 0b 8b 15 28 d9 91 c1 55 89 e5 [2.578745] EIP: [c1645ebc] enter_lazy.part.1+0x3/0x5 SS:ESP 0069:ec091a50 [2.578765] ---[ end trace ab5b5344be71ca3d ]--- [2.578775] Kernel panic - not syncing: Fatal exception in interrupt To manage notifications about this bug go to:
[Kernel-packages] [Bug 1350373] Re: Kernel BUG in paravirt_enter_lazy_mmu when running as a Xen PV guest
Attaching a typical trace on one of our affected machines. This shows the crash happening roughly 33 seconds after boot, which is where we observe most of these. Some also happen much later after boot. ** Attachment added: xen_bugcheck.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1350373/+attachment/4169803/+files/xen_bugcheck.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1350373 Title: Kernel BUG in paravirt_enter_lazy_mmu when running as a Xen PV guest Status in “linux” package in Ubuntu: Triaged Status in “linux” source package in Trusty: Triaged Bug description: Xen PV guests may crash during boot in paravirt_enter_lazy_mmu() while expanding the grant table (usually when requested by blkfront, when booting). See an example trace below. This is caused by calling functions that are unsafe in atomic context. The fix (which has been submitted to 3.16) is available here (also attached): https://git.kernel.org/cgit/linux/kernel/git/xen/tip.git/commit/?h=stable /for-linus-3.16id=b7dd0e350e0bd4c0fddcc9b8958342700b00b168 The fix is applicable to all kernel since 2.6.39 but only appears to trigger with the 3.13 kernel in 14.04. [2.577876] [ cut here ] [2.577896] kernel BUG at /build/buildd/linux-3.13.0/arch/x86/kernel/paravirt.c:239! [2.577910] invalid opcode: [#1] SMP [2.577922] Modules linked in: [2.577933] CPU: 0 PID: 1 Comm: init Not tainted 3.13.0-24-generic #46-Ubuntu [2.577946] task: ec058000 ti: ec09 task.ti: ec09 [2.577955] EIP: 0061:[c1645ebc] EFLAGS: 00010002 CPU: 0 [2.577973] EIP is at enter_lazy.part.1+0x3/0x5 [2.577982] EAX: 0001 EBX: ec0cc000 ECX: 00581980 EDX: [2.577992] ESI: edc0 EDI: edc0 EBP: ec091a50 ESP: ec091a50 [2.578001] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069 [2.578014] CR0: 8005003b CR2: bfca2fa4 CR3: 2c392000 CR4: 2660 [2.578027] Stack: [2.578032] ec091a58 c1046564 ec091ab4 c1146fd3 fa83b2da edc00fff edc01000 c1a93018 [2.578052] edc00fff c193ce80 edc01000 [2.578076] ed3ef588 ed3ef588 c1b87b70 ec091ad0 edc01000 c1b65310 1000 [2.578096] Call Trace: [2.578111] [c1046564] paravirt_enter_lazy_mmu+0x24/0x30 [2.578127] [c1146fd3] apply_to_page_range+0x1a3/0x310 [2.578141] [c1008eb8] arch_gnttab_map_status+0x38/0x60 [2.578152] [c1008d70] ? map_pte_fn+0x70/0x70 [2.578166] [c13ab020] gnttab_map_frames_v2+0xb0/0x100 [2.578182] [c13ab205] gnttab_map+0x95/0x120 [2.578198] [c12c7ff0] ? blk_update_request+0x190/0x340 [2.578209] [c13ab363] get_free_entries+0xd3/0x280 [2.578221] [c13ab5d3] gnttab_alloc_grant_references+0x13/0x30 [2.578238] [c1424be5] do_blkif_request+0x535/0x6f0 [2.578253] [c16523dc] ? _raw_spin_unlock_irqrestore+0x1c/0x40 [2.578269] [c12c57ee] __blk_run_queue+0x2e/0x40 [2.578280] [c12c5825] blk_start_queue+0x25/0x40 [2.578291] [c1424dbe] kick_pending_request_queues+0x1e/0x30 [2.578304] [c142546f] blkif_interrupt+0x69f/0x740 [2.578318] [c100654f] ? xen_set_pte_at+0xbf/0xf0 [2.578335] [c10a5ba5] handle_irq_event_percpu+0x35/0x1a0 [2.578351] [c12f136a] ? radix_tree_lookup+0xa/0x10 [2.578364] [c10a5d41] handle_irq_event+0x31/0x50 [2.578376] [c10a8036] handle_edge_irq+0x66/0x110 [2.578389] [c13ac246] __xen_evtchn_do_upcall+0x1c6/0x2c0 [2.578402] [c13ae100] xen_evtchn_do_upcall+0x20/0x40 [2.578415] [c165a087] xen_do_upcall+0x7/0xc [2.578427] [c1001227] ? xen_hypercall_xen_version+0x7/0x20 [2.578441] [c10083cf] ? xen_force_evtchn_callback+0xf/0x20 [2.578454] [c1008c50] check_events+0x8/0xc [2.578464] [c1008c47] ? xen_restore_fl_direct_reloc+0x4/0x4 [2.578480] [c1006373] ? xen_batched_set_pte+0xb3/0x160 [2.578493] [c10064b8] xen_set_pte_at+0x28/0xf0 [2.578505] [c10048e6] ? __raw_callee_save_xen_pte_val+0x6/0x8 [2.578521] [c11447a8] copy_pte_range+0x258/0x4c0 [2.578534] [c1146d27] copy_page_range+0x1d7/0x2e0 [2.578549] [c105462e] dup_mm+0x28e/0x4f0 [2.578561] [c1055866] copy_process.part.33+0xfa6/0x10d0 [2.578574] [c1055b41] do_fork+0xc1/0x2c0 [2.578591] [c1067996] ? SyS_rt_sigprocmask+0x76/0xa0 [2.578604] [c1055e05] SyS_clone+0x25/0x30 [2.578615] [c1659b4d] sysenter_do_call+0x12/0x28 [2.578626] Code: c4 1c 5b 5e 5f 5d c3 55 89 e5 f3 0f b8 c0 90 5d c3 55 ba a0 2c aa c1 89 e5 b9 25 00 00 00 57 31 c0 89 d7 f3 ab 5f 5d c3 55 89 e5 0f 0b 55 89 e5 66 66 66 66 90 0f 0b 8b 15 28 d9 91 c1 55 89 e5 [2.578745] EIP: [c1645ebc] enter_lazy.part.1+0x3/0x5 SS:ESP 0069:ec091a50 [2.578765] ---[ end trace ab5b5344be71ca3d ]--- [
[Kernel-packages] [Bug 1350373] Re: Kernel BUG in paravirt_enter_lazy_mmu when running as a Xen PV guest
We can confirm that we see a large fraction of our 14.04 Trusty Xen PV instances exhibit a stack trace similar to the one referenced above in paravirt_enter_lazy_mmu, followed by a crash. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1350373 Title: Kernel BUG in paravirt_enter_lazy_mmu when running as a Xen PV guest Status in “linux” package in Ubuntu: Triaged Status in “linux” source package in Trusty: Triaged Bug description: Xen PV guests may crash during boot in paravirt_enter_lazy_mmu() while expanding the grant table (usually when requested by blkfront, when booting). See an example trace below. This is caused by calling functions that are unsafe in atomic context. The fix (which has been submitted to 3.16) is available here (also attached): https://git.kernel.org/cgit/linux/kernel/git/xen/tip.git/commit/?h=stable /for-linus-3.16id=b7dd0e350e0bd4c0fddcc9b8958342700b00b168 The fix is applicable to all kernel since 2.6.39 but only appears to trigger with the 3.13 kernel in 14.04. [2.577876] [ cut here ] [2.577896] kernel BUG at /build/buildd/linux-3.13.0/arch/x86/kernel/paravirt.c:239! [2.577910] invalid opcode: [#1] SMP [2.577922] Modules linked in: [2.577933] CPU: 0 PID: 1 Comm: init Not tainted 3.13.0-24-generic #46-Ubuntu [2.577946] task: ec058000 ti: ec09 task.ti: ec09 [2.577955] EIP: 0061:[c1645ebc] EFLAGS: 00010002 CPU: 0 [2.577973] EIP is at enter_lazy.part.1+0x3/0x5 [2.577982] EAX: 0001 EBX: ec0cc000 ECX: 00581980 EDX: [2.577992] ESI: edc0 EDI: edc0 EBP: ec091a50 ESP: ec091a50 [2.578001] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069 [2.578014] CR0: 8005003b CR2: bfca2fa4 CR3: 2c392000 CR4: 2660 [2.578027] Stack: [2.578032] ec091a58 c1046564 ec091ab4 c1146fd3 fa83b2da edc00fff edc01000 c1a93018 [2.578052] edc00fff c193ce80 edc01000 [2.578076] ed3ef588 ed3ef588 c1b87b70 ec091ad0 edc01000 c1b65310 1000 [2.578096] Call Trace: [2.578111] [c1046564] paravirt_enter_lazy_mmu+0x24/0x30 [2.578127] [c1146fd3] apply_to_page_range+0x1a3/0x310 [2.578141] [c1008eb8] arch_gnttab_map_status+0x38/0x60 [2.578152] [c1008d70] ? map_pte_fn+0x70/0x70 [2.578166] [c13ab020] gnttab_map_frames_v2+0xb0/0x100 [2.578182] [c13ab205] gnttab_map+0x95/0x120 [2.578198] [c12c7ff0] ? blk_update_request+0x190/0x340 [2.578209] [c13ab363] get_free_entries+0xd3/0x280 [2.578221] [c13ab5d3] gnttab_alloc_grant_references+0x13/0x30 [2.578238] [c1424be5] do_blkif_request+0x535/0x6f0 [2.578253] [c16523dc] ? _raw_spin_unlock_irqrestore+0x1c/0x40 [2.578269] [c12c57ee] __blk_run_queue+0x2e/0x40 [2.578280] [c12c5825] blk_start_queue+0x25/0x40 [2.578291] [c1424dbe] kick_pending_request_queues+0x1e/0x30 [2.578304] [c142546f] blkif_interrupt+0x69f/0x740 [2.578318] [c100654f] ? xen_set_pte_at+0xbf/0xf0 [2.578335] [c10a5ba5] handle_irq_event_percpu+0x35/0x1a0 [2.578351] [c12f136a] ? radix_tree_lookup+0xa/0x10 [2.578364] [c10a5d41] handle_irq_event+0x31/0x50 [2.578376] [c10a8036] handle_edge_irq+0x66/0x110 [2.578389] [c13ac246] __xen_evtchn_do_upcall+0x1c6/0x2c0 [2.578402] [c13ae100] xen_evtchn_do_upcall+0x20/0x40 [2.578415] [c165a087] xen_do_upcall+0x7/0xc [2.578427] [c1001227] ? xen_hypercall_xen_version+0x7/0x20 [2.578441] [c10083cf] ? xen_force_evtchn_callback+0xf/0x20 [2.578454] [c1008c50] check_events+0x8/0xc [2.578464] [c1008c47] ? xen_restore_fl_direct_reloc+0x4/0x4 [2.578480] [c1006373] ? xen_batched_set_pte+0xb3/0x160 [2.578493] [c10064b8] xen_set_pte_at+0x28/0xf0 [2.578505] [c10048e6] ? __raw_callee_save_xen_pte_val+0x6/0x8 [2.578521] [c11447a8] copy_pte_range+0x258/0x4c0 [2.578534] [c1146d27] copy_page_range+0x1d7/0x2e0 [2.578549] [c105462e] dup_mm+0x28e/0x4f0 [2.578561] [c1055866] copy_process.part.33+0xfa6/0x10d0 [2.578574] [c1055b41] do_fork+0xc1/0x2c0 [2.578591] [c1067996] ? SyS_rt_sigprocmask+0x76/0xa0 [2.578604] [c1055e05] SyS_clone+0x25/0x30 [2.578615] [c1659b4d] sysenter_do_call+0x12/0x28 [2.578626] Code: c4 1c 5b 5e 5f 5d c3 55 89 e5 f3 0f b8 c0 90 5d c3 55 ba a0 2c aa c1 89 e5 b9 25 00 00 00 57 31 c0 89 d7 f3 ab 5f 5d c3 55 89 e5 0f 0b 55 89 e5 66 66 66 66 90 0f 0b 8b 15 28 d9 91 c1 55 89 e5 [2.578745] EIP: [c1645ebc] enter_lazy.part.1+0x3/0x5 SS:ESP 0069:ec091a50 [2.578765] ---[ end trace ab5b5344be71ca3d ]--- [2.578775] Kernel panic - not syncing: Fatal exception in interrupt To manage notifications about this bug go to:
