Possibly relevant: the issue does *not* occur if I run my VM with a single core.
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1567622 Title: Kernel Oops - unable to handle kernel NULL pointer dereference at 0000000000000910 in update_blocked_averages Status in linux package in Ubuntu: Confirmed Bug description: Running in Virtualbox as a guest in a Macbook Pro, I'm running into a reliably-reproducible kernel oops on Ubuntu 16.04 with Linux 4.4.0-17-generic. The stack trace isn't always the same, although the error always happens in the same `update_blocked_averages` function. Here' one oops: ``` [ 56.972407] BUG: unable to handle kernel NULL pointer dereference at 0000000000000910 [ 57.076894] IP: [<ffffffff810b403d>] update_blocked_averages+0x8d/0x520 [ 57.173751] PGD 0 [ 57.193537] Oops: 0000 [#1] SMP [ 57.245472] Modules linked in: veth xt_addrtype xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables x_tables nf_nat nf_conntrack bridge stp llc aufs ppdev crct10dif_pclmul crc32_pclmul snd_intel8x0 snd_ac97_codec aesni_intel ac97_bus aes_x86_64 snd_pcm lrw gf128mul snd_timer input_leds glue_helper ablk_helper snd cryptd joydev serio_raw soundcore i2c_piix4 8250_fintek mac_hid parport_pc lp parport autofs4 hid_generic usbhid ahci psmouse hid libahci pata_acpi e1000 video fjes [ 57.937239] CPU: 1 PID: 269 Comm: systemd-journal Not tainted 4.4.0-17-generic #33-Ubuntu [ 58.049743] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 58.189828] task: ffff8800d8f90dc0 ti: ffff8800da9a8000 task.ti: ffff8800da9a8000 [ 58.299087] RIP: 0010:[<ffffffff810b403d>] [<ffffffff810b403d>] update_blocked_averages+0x8d/0x520 [ 58.320327] RSP: 0018:ffff8800da9abc70 EFLAGS: 00010046 [ 58.366603] RAX: 0000000000000000 RBX: ffff88020c3bae00 RCX: 0000000000000000 [ 58.514856] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001 [ 58.612464] RBP: ffff8800da9abcc8 R08: afb504000afb5041 R09: 0000000000000000 [ 58.738629] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000045 [ 58.885338] R13: 0000000000000000 R14: ffff8802198975c0 R15: 0000000000000001 [ 59.001563] FS: 00007f7db7f6e840(0000) GS:ffff880219880000(0000) knlGS:0000000000000000 [ 59.160653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 59.296218] CR2: 0000000000000910 CR3: 000000020c3fd000 CR4: 00000000000406e0 [ 59.418514] Stack: [ 59.458306] 00000000000205f5 ffff880219896d00 0000000000000086 afb504000afb5041 [ 59.580604] 0000000000000000 0000ad60ffff01fe 0000000000000001 00000000ffff02f8 [ 59.720268] 0000000000016d00 ffff880219896d00 ffff8800d8f91310 ffff8800da9abd38 [ 59.848040] Call Trace: [ 59.902635] [<ffffffff810bcbe7>] pick_next_task_fair+0x1e7/0x4f0 [ 59.992238] [<ffffffff8181e175>] __schedule+0x125/0xa10 [ 60.030794] [<ffffffff8181ea95>] schedule+0x35/0x80 [ 60.074193] [<ffffffff81822053>] schedule_hrtimeout_range_clock+0x193/0x1b0 [ 60.134427] [<ffffffff81254137>] ? ep_scan_ready_list+0x1e7/0x1f0 [ 60.195234] [<ffffffff810c9591>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20 [ 60.289354] [<ffffffff81822083>] schedule_hrtimeout_range+0x13/0x20 [ 60.325540] [<ffffffff81254430>] ep_poll+0x2c0/0x3d0 [ 60.328490] [<ffffffff810aaca0>] ? wake_up_q+0x70/0x70 [ 60.387888] [<ffffffff81255718>] SyS_epoll_wait+0xb8/0xd0 [ 60.502960] [<ffffffff81822b72>] entry_SYSCALL_64_fastpath+0x16/0x71 [ 60.610213] Code: 24 08 49 b8 41 50 fb 0a 00 04 b5 af 4d 89 fe 0f 1f 44 00 00 44 8b 9b 24 01 00 00 45 85 db 0f 85 a0 03 00 00 48 8b 83 c8 00 00 00 <48> 8b 80 10 09 00 00 48 2b 83 18 01 00 00 48 8b 93 a0 00 00 00 [ 60.941287] RIP [<ffffffff810b403d>] update_blocked_averages+0x8d/0x520 [ 61.111617] RSP <ffff8800da9abc70> [ 61.201406] CR2: 0000000000000910 [ 61.258351] ---[ end trace fe3df8ee7b476828 ]--- ``` Here's another one: ``` [ 154.454951] BUG: unable to handle kernel [ 154.455154] device vethe5e9554 entered promiscuous mode [ 154.455202] IPv6: ADDRCONF(NETDEV_UP): vethe5e9554: link is not ready [ 154.455203] docker0: port 1(vethe5e9554) entered forwarding state [ 154.455210] docker0: port 1(vethe5e9554) entered forwarding state [ 154.744305] NULL pointer dereference at 0000000000000910 [ 154.783574] IP: [<ffffffff810b403d>] update_blocked_averages+0x8d/0x520 [ 154.784877] PGD 0 [ 154.785372] Oops: 0000 [#1] SMP [ 154.790654] Modules linked in: veth xt_addrtype xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables x_tables nf_nat nf_conntrack bridge stp llc aufs vboxsf(OE) ppdev crct10dif_pclmul crc32_pclmul aesni_intel snd_intel8x0 aes_x86_64 input_leds lrw snd_ac97_codec gf128mul glue_helper ac97_bus ablk_helper cryptd serio_raw vboxvideo(OE) joydev snd_pcm i2c_piix4 snd_timer drm 8250_fintek snd soundcore vboxguest(OE) parport_pc mac_hid lp parport autofs4 hid_generic usbhid ahci hid psmouse libahci e1000 pata_acpi video fjes [ 155.387419] CPU: 0 PID: 2043 Comm: systemd-udevd Tainted: G OE 4.4.0-17-generic #33-Ubuntu [ 155.546119] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 155.654775] task: ffff8800dbb3b700 ti: ffff88020da8c000 task.ti: ffff88020da8c000 [ 155.753906] RIP: 0010:[<ffffffff810b403d>] [<ffffffff810b403d>] update_blocked_averages+0x8d/0x520 [ 155.775805] RSP: 0000:ffff880219803df0 EFLAGS: 00010046 [ 155.811224] RAX: 0000000000000000 RBX: ffff8800db891200 RCX: 0000000000000001 [ 155.857524] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: 0000000000000002 [ 155.861832] RBP: ffff880219803e50 R08: afb504000afb5041 R09: 0000000000000100 [ 155.980061] R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000291 [ 156.059221] R13: 0000000000000000 R14: ffff8802198175c0 R15: 0000000000000001 [ 156.167527] FS: 00007f06cde8c8c0(0000) GS:ffff880219800000(0000) knlGS:0000000000000000 [ 156.302318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 156.386563] CR2: 0000000000000910 CR3: 000000020c6a7000 CR4: 00000000000406f0 [ 156.507153] Stack: [ 156.559351] 0000000000445496 ffff880219816d00 0000000000000286 afb504000afb5041 [ 156.590109] 0000000000000000 0043430019816d00 0000000000016d00 0000000000000000 [ 156.684722] 00000000ffff61a3 0000000000000001 ffff880219816d00 ffff8800db5097f8 [ 156.798239] Call Trace: [ 156.837119] <IRQ> [ 156.863469] [<ffffffff810bcf3b>] rebalance_domains+0x4b/0x2d0 [ 156.997302] [<ffffffff810ed7b9>] ? update_process_times+0x59/0x60 [ 157.111072] [<ffffffff810bd399>] run_rebalance_domains+0x1d9/0x210 [ 157.183338] [<ffffffff81051a4d>] ? lapic_next_event+0x1d/0x30 [ 157.271222] [<ffffffff81084851>] __do_softirq+0x101/0x290 [ 157.359428] [<ffffffff81084b53>] irq_exit+0xa3/0xb0 [ 157.453079] [<ffffffff81825622>] smp_apic_timer_interrupt+0x42/0x50 [ 157.535290] [<ffffffff818238e2>] apic_timer_interrupt+0x82/0x90 [ 157.656699] <EOI> [ 157.688002] [<ffffffff8118c88c>] ? filemap_map_pages+0x8c/0x230 [ 157.752691] [<ffffffff810c9591>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20 [ 157.806798] [<ffffffff811bec61>] handle_mm_fault+0x1131/0x1820 [ 157.860940] [<ffffffff8122be24>] ? mntput+0x24/0x40 [ 157.904327] [<ffffffff8106a537>] __do_page_fault+0x197/0x400 [ 157.984122] [<ffffffff8106a7c2>] do_page_fault+0x22/0x30 [ 158.076895] [<ffffffff81824cf8>] page_fault+0x28/0x30 [ 158.138672] Code: 24 08 49 b8 41 50 fb 0a 00 04 b5 af 4d 89 fe 0f 1f 44 00 00 44 8b 9b 24 01 00 00 45 85 db 0f 85 a0 03 00 00 48 8b 83 c8 00 00 00 <48> 8b 80 10 09 00 00 48 2b 83 18 01 00 00 48 8b 93 a0 00 00 00 [ 158.290783] RIP [<ffffffff810b403d>] update_blocked_averages+0x8d/0x520 [ 158.383333] RSP <ffff880219803df0> [ 158.414588] CR2: 0000000000000910 [ 158.467115] ---[ end trace 27ceb18bfd8d94ca ]--- [ 158.505862] Kernel panic - not syncing: Fatal exception in interrupt [ 159.629144] Shutting down cpus with NMI [ 159.745790] Kernel Offset: disabled [ 159.859365] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ``` And a third one: ``` [ 139.661389] BUG: unable to handle kernel NULL pointer dereference at 0000000000000910 [ 139.807547] IP: [<ffffffff810b403d>] update_blocked_averages+0x8d/0x520 [ 139.946882] PGD 0 [ 139.984609] Oops: 0000 [#1] SMP [ 140.048650] Modules linked in: veth xt_addrtype xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables x_tables nf_nat nf_conntrack bridge stp llc aufs vboxsf ppdev crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_intel8x0 snd_ac97_codec vboxvideo ac97_bus input_leds joydev snd_pcm drm snd_timer serio_raw snd i2c_piix4 soundcore 8250_fintek vboxguest parport_pc mac_hid lp parport autofs4 hid_generic usbhid ahci psmouse libahci hid e1000 pata_acpi fjes video [ 141.115576] CPU: 0 PID: 2038 Comm: systemd-udevd Not tainted 4.4.0-17-generic #33-Ubuntu [ 141.310652] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 141.415336] task: ffff88020c2b9b80 ti: ffff8800dadf0000 task.ti: ffff8800dadf0000 [ 141.519708] RIP: 0010:[<ffffffff810b403d>] [<ffffffff810b403d>] update_blocked_averages+0x8d/0x520 [ 141.707701] RSP: 0018:ffff8800dadf3c70 EFLAGS: 00010046 [ 141.824270] RAX: 0000000000000000 RBX: ffff88020c1a9000 RCX: 0000000000000001 [ 141.999630] RDX: 0000000000000485 RSI: 00000000cd4629c5 RDI: ffff880219bf1000 [ 142.180519] RBP: ffff8800dadf3cc8 R08: afb504000afb5041 R09: 0000000000000000 [ 142.369893] R10: 00000000ffff518b R11: 0000000000000000 R12: 00000000ffff5285 [ 142.525548] R13: 0000000000016d00 R14: ffff8802198175c0 R15: ffff8802198175c0 [ 142.675206] FS: 00007ff7980cf8c0(0000) GS:ffff880219800000(0000) knlGS:0000000000000000 [ 142.871842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 143.019994] CR2: 0000000000000910 CR3: 00000000d8e0e000 CR4: 00000000000406f0 [ 143.181043] Stack: [ 143.225953] ffffffff813fcd45 ffff880219816d00 0000000000000086 ffffffff810c9591 [ 143.308932] ffff88020ee31c00 00000000ffff518b 0000000000000000 00000000ffff5285 [ 143.470616] 0000000000016d00 ffff880219816d00 ffff88020c2ba0d0 ffff8800dadf3d38 [ 143.593228] Call Trace: [ 143.695835] [<ffffffff813fcd45>] ? find_next_bit+0x15/0x20 [ 143.812410] [<ffffffff810c9591>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20 [ 143.999872] [<ffffffff810bcbe7>] pick_next_task_fair+0x1e7/0x4f0 [ 144.104135] [<ffffffff8181e175>] __schedule+0x125/0xa10 [ 144.120508] [<ffffffff8181ea95>] schedule+0x35/0x80 [ 144.226139] [<ffffffff81821f85>] schedule_hrtimeout_range_clock+0xc5/0x1b0 [ 144.372190] [<ffffffff810edbf0>] ? __hrtimer_init+0x90/0x90 [ 144.491899] [<ffffffff81821f79>] ? schedule_hrtimeout_range_clock+0xb9/0x1b0 [ 144.646067] [<ffffffff81822083>] schedule_hrtimeout_range+0x13/0x20 [ 144.687266] [<ffffffff81254430>] ep_poll+0x2c0/0x3d0 [ 144.689511] [<ffffffff810aaca0>] ? wake_up_q+0x70/0x70 [ 144.785095] [<ffffffff81255718>] SyS_epoll_wait+0xb8/0xd0 [ 144.843879] [<ffffffff81822b72>] entry_SYSCALL_64_fastpath+0x16/0x71 [ 144.859757] Code: 24 08 49 b8 41 50 fb 0a 00 04 b5 af 4d 89 fe 0f 1f 44 00 00 44 8b 9b 24 01 00 00 45 85 db 0f 85 a0 03 00 00 48 8b 83 c8 00 00 00 <48> 8b 80 10 09 00 00 48 2b 83 18 01 00 00 48 8b 93 a0 00 00 00 [ 144.982177] RIP [<ffffffff810b403d>] update_blocked_averages+0x8d/0x520 [ 145.024132] RSP <ffff8800dadf3c70> [ 145.069952] CR2: 0000000000000910 [ 145.118873] ---[ end trace 5324671710cba237 ]--- ``` I can reproduce this reliably using the following script (which relies on Docker). This starts a container, then adds another task into its namespace via `docker exec`, and then kills PID 1 in the container, which tears down the PID namespace and should kill both tasks. If I remove the `docker exec` line, there is no crash. ``` #!/bin/bash set -exu cid=$(docker run -it -d ubuntu sleep 1000) docker exec "$cid" sleep 1000 & sleep 1 docker kill -s KILL "$cid" docker run -it --rm ubuntu echo "Alive 1?" docker run -it --rm ubuntu echo "Alive 2?" docker run -it --rm ubuntu echo "Alive 3?" ``` In the vast majority of cases, the crash will happen before "Alive 1?" is printed to the screen. In some cases, it'll happen at "Alive 2?". I tried reproducing the issue on an AWS instance, with no success so far. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-17-generic 4.4.0-17.33 ProcVersionSignature: Ubuntu 4.4.0-17.33-generic 4.4.6 Uname: Linux 4.4.0-17-generic x86_64 AlsaVersion: Advanced Linux Sound Architecture Driver Version k4.4.0-17-generic. AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/pcmC0D1c', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer' Card0.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer' Date: Thu Apr 7 20:53:43 2016 HibernationDevice: RESUME=/dev/mapper/ubuntu--build--vg-swap_1 InstallationDate: Installed on 2016-02-03 (63 days ago) InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805) IwConfig: eth0 no wireless extensions. eth1 no wireless extensions. lo no wireless extensions. Lsusb: Bus 001 Device 002: ID 80ee:0021 VirtualBox USB Tablet Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub MachineType: innotek GmbH VirtualBox ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-17-generic root=/dev/mapper/ubuntu--build--vg-root ro console=ttyS0,9600 console=tty0 RelatedPackageVersions: linux-restricted-modules-4.4.0-17-generic N/A linux-backports-modules-4.4.0-17-generic N/A linux-firmware 1.157 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2016-04-06 (1 days ago) dmi.bios.date: 12/01/2006 dmi.bios.vendor: innotek GmbH dmi.bios.version: VirtualBox dmi.board.name: VirtualBox dmi.board.vendor: Oracle Corporation dmi.board.version: 1.2 dmi.chassis.type: 1 dmi.chassis.vendor: Oracle Corporation dmi.modalias: dmi:bvninnotekGmbH:bvrVirtualBox:bd12/01/2006:svninnotekGmbH:pnVirtualBox:pvr1.2:rvnOracleCorporation:rnVirtualBox:rvr1.2:cvnOracleCorporation:ct1:cvr: dmi.product.name: VirtualBox dmi.product.version: 1.2 dmi.sys.vendor: innotek GmbH To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1567622/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
