Cut from Bug Description. ** Summary changed:
- Kernel Oops - BUG: unable to handle kernel paging request at 0000000041d0f980 + Kernel Oops - BUG: unable to handle kernel paging request at 0000000041d0f980; RIP: 0010:[<ffffffff8105287d>] [<ffffffff8105287d>] task_rq_lock+0x4d/0xa0 ** Tags removed: bug kernel-upstream-testing-not-needed kvm oops ** Tags added: lucid needs-upstream-testing ** Attachment added: "log" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/732811/+attachment/3238570/+files/log ** Description changed: + 1) lsb_release -rd + Description: Ubuntu 10.04.2 LTS + Release: 10.04 + + 2) The version of the package you are using: Don't know it's probably + related to the xfs kernel module looking at the stacktrace + + 3) What I expected to happen is that the instance doesn't crash on IO + operations. + + 4) What happened instead is the machine freezes and can only be restored + by killing it from the hostmachine issuing the command virsh destroy + instance_name. + The server on which this happens is actually a KVM instance running on a Ubuntu Lucid 64bit hostmachine, the instance is also a 64 bit lucid installation. It seems to be correlated between some network traffic and IO on the hostmachine (the hostmachine rsyncs snapshots to a backupmachine limited to 100 mbit/s ) The stacktrace on the instance is here below: - - [2844391.352629] BUG: unable to handle kernel paging request at 0000000041d0f980 - [2844391.362351] IP: [<ffffffff8105287d>] task_rq_lock+0x4d/0xa0 - [2844391.362351] PGD 0 - [2844391.362351] Oops: 0000 [#1] SMP - [2844391.362351] last sysfs file: /sys/devices/virtio-pci/virtio2/block/vda/stat - [2844391.362351] CPU 2 - [2844391.362351] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs reiserfs xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables sha1_generic arc4 ppp_mppe ppp_async crc_ccitt fbcon tileblit font bitblit lp softcursor vga16fb parport vgastate psmouse serio_raw xfs exportfs floppy - [2844391.362351] Pid: 0, comm: swapper Not tainted 2.6.32-28-server #55-Ubuntu Bochs - [2844391.362351] RIP: 0010:[<ffffffff8105287d>] [<ffffffff8105287d>] task_rq_lock+0x4d/0xa0 - [2844391.362351] RSP: 0018:ffff880018103c28 EFLAGS: 00010086 - [2844391.362351] RAX: 0000000018095bc0 RBX: 0000000000015bc0 RCX: 0000000000000000 - [2844391.362351] RDX: 0000000000000086 RSI: ffff880018103c70 RDI: ffff8804008a2de0 - [2844391.362351] RBP: ffff880018103c48 R08: 0000000000000000 R09: 2200000000000000 - [2844391.362351] R10: ffff88040dc97200 R11: 0000000000000000 R12: ffff8804008a2de0 - [2844391.362351] R13: ffff880018103c70 R14: 0000000000015bc0 R15: 0000000000000002 - [2844391.362351] FS: 0000000000000000(0000) GS:ffff880018100000(0000) knlGS:0000000000000000 - [2844391.362351] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b - [2844391.362351] CR2: 0000000041d0f980 CR3: 00000002059e1000 CR4: 00000000000006e0 - [2844391.362351] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 - [2844391.362351] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 - [2844391.362351] Process swapper (pid: 0, threadinfo ffff88040e508000, task ffff88040e502de0) - [2844391.362351] Stack: - [2844391.362351] ffff8804008a2de0 0000000000000001 0000000000000003 0000000000000000 - [2844391.362351] <0> ffff880018103ca8 ffffffff81059761 0000000000015bc0 0000000000015bc0 - [2844391.362351] <0> 0000000100000082 0000000000000086 ffff88001810fed0 ffff88009dc76e60 - [2844391.362351] Call Trace: - [2844391.362351] <IRQ> - [2844391.362351] [<ffffffff81059761>] try_to_wake_up+0x41/0x470 - [2844391.362351] [<ffffffff81059ba2>] default_wake_function+0x12/0x20 - [2844391.362351] [<ffffffff8104ae99>] __wake_up_common+0x59/0x90 - [2844391.362351] [<ffffffff810526b7>] complete+0x47/0x60 - [2844391.362351] [<ffffffffa007e3a5>] xfs_buf_ioend+0xa5/0xb0 [xfs] - [2844391.362351] [<ffffffffa007e49d>] xfs_buf_bio_end_io+0xed/0x150 [xfs] - [2844391.362351] [<ffffffff811711ed>] bio_endio+0x1d/0x40 - [2844391.362351] [<ffffffff8129e31b>] req_bio_endio+0xab/0x100 - [2844391.362351] [<ffffffff8129ed27>] blk_update_request+0x107/0x450 - [2844391.362351] [<ffffffff8129f097>] blk_update_bidi_request+0x27/0x70 - [2844391.362351] [<ffffffff8129fc5e>] __blk_end_request_all+0x2e/0x60 - [2844391.362351] [<ffffffff81381185>] blk_done+0x35/0x100 - [2844391.362351] [<ffffffff8131d2cd>] vring_interrupt+0x2d/0x50 - [2844391.362351] [<ffffffff810c4160>] handle_IRQ_event+0x60/0x170 - [2844391.362351] [<ffffffff810c65a2>] handle_edge_irq+0xd2/0x170 - [2844391.362351] [<ffffffff81014d12>] handle_irq+0x22/0x30 - [2844391.362351] [<ffffffff815607fc>] do_IRQ+0x6c/0xf0 - [2844391.362351] [<ffffffff81012b13>] ret_from_intr+0x0/0x11 - [2844391.362351] <EOI> - [2844391.362351] [<ffffffff81037adb>] ? native_safe_halt+0xb/0x10 - [2844391.362351] [<ffffffff8101a77d>] ? default_idle+0x3d/0x90 - [2844391.362351] [<ffffffff81010e63>] ? cpu_idle+0xb3/0x110 - [2844391.362351] [<ffffffff81553463>] ? start_secondary+0xa8/0xaa - [2844391.362351] Code: c3 c0 5b 01 00 49 89 fc 49 89 f5 9c 58 0f 1f 44 00 00 48 89 c2 fa 66 0f 1f 44 00 00 49 89 55 00 49 8b 44 24 08 49 89 de 8b 40 18 <4c> 03 34 c5 80 1b 86 81 4c 89 f7 e8 33 8d 50 00 49 8b 44 24 08 - [2844391.362351] RIP [<ffffffff8105287d>] task_rq_lock+0x4d/0xa0 - [2844391.362351] RSP <ffff880018103c28> - [2844391.362351] CR2: 0000000041d0f980 - [2844391.362351] ---[ end trace 82df0364670fc9b4 ]--- - [2844391.361752] BUG: unable to handle kernel paging request at ffffffffeefcfb40 - [2844391.361752] IP: [<ffffffff81052f9e>] hrtick_start_fair+0x3e/0x190 - [2844391.361752] PGD 1003067 PUD 1004067 PMD 0 - [2844391.361752] Thread overran stack, or stack corrupted - [2844391.361752] Oops: 0000 [#2] SMP - [2844391.361752] last sysfs file: /sys/devices/virtio-pci/virtio2/block/vda/stat - [2844391.361752] CPU 1 - [2844391.361752] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs reiserfs xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables sha1_generic arc4 ppp_mppe ppp_async crc_ccitt fbcon tileblit font bitblit lp softcursor vga16fb parport vgastate psmouse serio_raw xfs exportfs floppy - [2844391.361752] Pid: 633, comm: flush-252:16 Tainted: G D 2.6.32-28-server #55-Ubuntu Bochs - [2844391.361752] RIP: 0010:[<ffffffff81052f9e>] [<ffffffff81052f9e>] hrtick_start_fair+0x3e/0x190 - [2844391.361752] RSP: 0018:ffff880401dea050 EFLAGS: 00010086 - [2844391.361752] RAX: 0000000000015bc0 RBX: ffff880018095bc0 RCX: 00029a7be7860131 - [2844391.361752] RDX: 000000000daedbf8 RSI: ffff8804008a2de0 RDI: ffff880018095bc0 - [2844391.361752] RBP: ffff880401dea070 R08: 0000000000000000 R09: 0000000000000001 - [2844391.361752] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8804008a2de0 - [2844391.361752] R13: 0000000000015bc0 R14: ffff880018095c28 R15: 0000000000000206 - [2844391.361752] FS: 0000000000000000(0000) GS:ffff880018080000(0000) knlGS:0000000000000000 - [2844391.361752] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b - [2844391.361752] CR2: ffffffffeefcfb40 CR3: 0000000273bb7000 CR4: 00000000000006e0 - [2844391.361752] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 - [2844391.361752] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 - [2844391.361752] Process flush-252:16 (pid: 633, threadinfo ffff880401dea000, task ffff8804008a2de0) - [2844391.361752] Stack: - [2844391.361752] 0000000000000000 ffff880018095bc0 0000000000000001 ffff880401dea1e0 - [2844391.361752] <0> ffff880401dea090 ffffffff8105e10b ffff88040daedbc0 ffff880018095bc0 - [2844391.361752] <0> ffff880401dea0b0 ffffffff8104a4cf 0000000000040c00 ffff880018095bc0 - [2844391.361752] Call Trace: - [2844391.361752] [<ffffffff8105e10b>] enqueue_task_fair+0x9b/0xa0 - [2844391.361752] general protection fault: 0000 [#3] SMP - [2844391.361752] last sysfs file: /sys/devices/virtio-pci/virtio2/block/vda/stat - [2844391.361752] CPU 1 - [2844391.361752] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs reiserfs xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables sha1_generic arc4 ppp_mppe ppp_async crc_ccitt fbcon tileblit font bitblit lp softcursor vga16fb parport vgastate psmouse serio_raw xfs exportfs floppy - [2844391.361752] Pid: 633, comm: flush-252:16 Tainted: G D 2.6.32-28-server #55-Ubuntu Bochs - [2844391.361752] RIP: 0010:[<ffffffff81016361>] [<ffffffff81016361>] print_context_stack+0xb1/0x140 - [2844391.361752] RSP: 0018:ffff880401de9c78 EFLAGS: 00010002 - [2844391.361752] RAX: 00029a7be7860131 RBX: ffff880401dea078 RCX: 00000000000009cf - [2844391.361752] RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0000000000000046 - [2844391.361752] RBP: ffff880401de9cd8 R08: 00000000fffffffe R09: 0000000000000000 - [2844391.361752] R10: 0000000000000005 R11: 0000000000000000 R12: ffff880401dea090 - [2844391.361752] R13: ffff880401dea000 R14: ffffffff81568360 R15: 0000000000000000 - [2844391.361752] FS: 0000000000000000(0000) GS:ffff880018080000(0000) knlGS:0000000000000000 - [2844391.361752] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b - [2844391.361752] CR2: ffffffffeefcfb40 CR3: 0000000273bb7000 CR4: 00000000000006e0 - [2844391.361752] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 - [2844391.361752] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 - [2844391.361752] Process flush-252:16 (pid: 633, threadinfo ffff880401dea000, task ffff8804008a2de0) - [2844391.361752] Stack: - [2844391.361752] ffff880401de9ca8 ffff880401debff8 ffff880401dea090 ffffffffffffe000 - [2844391.361752] <0> ffffffff816e95d3 ffffffff8105e10b ffff880401de0020 ffff880401dea050 - [2844391.361752] <0> 000000000000cbe0 ffffffff81568360 ffff880401dea070 ffffffff816e95d3 - [2844391.361752] Call Trace: - [2844391.361752] Code: c0 74 30 49 8d 44 24 08 48 39 c3 0f 84 81 00 00 00 31 d2 48 8b 75 c8 48 8b 7d c0 41 ff 56 10 48 81 7d c8 81 1f 01 81 49 8b 45 00 <8b> 90 b8 16 00 00 74 1f 48 83 c3 08 4d 85 ff 75 8e 4c 39 eb 76 - [2844391.361752] RIP [<ffffffff81016361>] print_context_stack+0xb1/0x140 - [2844391.361752] RSP <ffff880401de9c78> - [2844391.361752] ---[ end trace 82df0364670fc9b5 ]--- - [2844391.361752] Kernel panic - not syncing: Fatal exception in interrupt - [2844391.361752] Pid: 633, comm: flush-252:16 Tainted: G D 2.6.32-28-server #55-Ubuntu - [2844391.361752] Call Trace: - [2844391.362351] Kernel panic - not syncing: Fatal exception in interrupt - [2844391.362351] Pid: 0, comm: swapper Tainted: G D 2.6.32-28-server #55-Ubuntu - [2844391.362351] Call Trace: - [2844391.362351] <IRQ> [<ffffffff815587e6>] panic+0x78/0x139 - [2844391.362351] [<ffffffff8155c75a>] oops_end+0xea/0xf0 - [2844391.362351] [<ffffffff8103fb63>] no_context+0xf3/0x190 - [2844391.362351] [<ffffffff8103fd25>] __bad_area_nosemaphore+0x125/0x1e0 - [2844391.362351] [<ffffffff81052d48>] ? enqueue_sleeper+0x158/0x220 - [2844391.362351] [<ffffffff8103fdf3>] bad_area_nosemaphore+0x13/0x20 - [2844391.362351] [<ffffffff8155e304>] do_page_fault+0x2e4/0x3b0 - [2844391.362351] [<ffffffff8155ba95>] page_fault+0x25/0x30 - [2844391.362351] [<ffffffff8105287d>] ? task_rq_lock+0x4d/0xa0 - [2844391.362351] [<ffffffff81059761>] try_to_wake_up+0x41/0x470 - [2844391.362351] [<ffffffff81059ba2>] default_wake_function+0x12/0x20 - [2844391.362351] [<ffffffff8104ae99>] __wake_up_common+0x59/0x90 - [2844391.362351] [<ffffffff810526b7>] complete+0x47/0x60 - [2844391.362351] [<ffffffffa007e3a5>] xfs_buf_ioend+0xa5/0xb0 [xfs] - [2844391.362351] [<ffffffffa007e49d>] xfs_buf_bio_end_io+0xed/0x150 [xfs] - [2844391.362351] [<ffffffff811711ed>] bio_endio+0x1d/0x40 - [2844391.362351] [<ffffffff8129e31b>] req_bio_endio+0xab/0x100 - [2844391.362351] [<ffffffff8129ed27>] blk_update_request+0x107/0x450 - [2844391.362351] [<ffffffff8129f097>] blk_update_bidi_request+0x27/0x70 - [2844391.362351] [<ffffffff8129fc5e>] __blk_end_request_all+0x2e/0x60 - [2844391.362351] [<ffffffff81381185>] blk_done+0x35/0x100 - [2844391.362351] [<ffffffff8131d2cd>] vring_interrupt+0x2d/0x50 - [2844391.362351] [<ffffffff810c4160>] handle_IRQ_event+0x60/0x170 - [2844391.362351] [<ffffffff810c65a2>] handle_edge_irq+0xd2/0x170 - [2844391.362351] [<ffffffff81014d12>] handle_irq+0x22/0x30 - [2844391.362351] [<ffffffff815607fc>] do_IRQ+0x6c/0xf0 - [2844391.362351] [<ffffffff81012b13>] ret_from_intr+0x0/0x11 - [2844391.362351] <EOI> [<ffffffff81037adb>] ? native_safe_halt+0xb/0x10 - [2844391.362351] [<ffffffff8101a77d>] ? default_idle+0x3d/0x90 - [2844391.362351] [<ffffffff81010e63>] ? cpu_idle+0xb3/0x110 - [2844391.362351] [<ffffffff81553463>] ? start_secondary+0xa8/0xaa - - 1) The release of Ubuntu you are using - lsb_release -rd - Description: Ubuntu 10.04.2 LTS - Release: 10.04 - - 2) The version of the package you are using - Don't know it's probably related to the xfs kernel module looking at the stacktrace - - 3) What you expected to happen - That the instance doesn't crash on IO operations - - 4) What happened instead - The machine freezes and can only be restored by killing it from the hostmachine issuing the command virsh destroy instance_name. + https://bugs.launchpad.net/ubuntu/+source/linux/+bug/732811/+attachment/3238570/+files/log The crashes seem to be pretty much random but consistent, the logfiles on the instance are empty and offer no clue prior to what might be happening that leads to the crash/freeze, the above stacktrace was only possible because we enabled serial logging from the instance to the hostmachine. There are no cronjobs running on the instance that could account for the crash, the only process doing some IO and network traffic is the actual rsync process on the host. Some more details: - Hostmachine * Dell R610 * 2 x Intel X5670 HT enabled * 64GB RAM - * 6 x 600 GB 10K RPM in raid6 (h700 controller with 512 NV RAM) + * 6 x 600 GB 10K RPM in raid6 (h700 controller with 512 NV RAM) * Ubuntu Lucid 64bit with kvm-qemu from the default ubuntu repositories. - Instance * 4 x vCPU (QEMU Virtual CPU version 0.12.3) * 16 GB RAM * one diskimage of 500 GB (LVM volume XFS formatted) * Virtio drivers loaded - * Ubuntu Lucid 64bit + * Ubuntu Lucid 64bit KVM start options for this instance: usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 16384 -smp 4 -name instance-123 -uuid 4916d022-c2e7-4f62-82ea-a8496d246610 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/instance-123.monitor,server,nowait -monitor chardev:monitor -boot dc -drive file=/dev/xx/disk-318,if=virtio,index=0,boot=on,format=raw,cache=none,if=virtio,index=1,format=raw,cache=none ,if=virtio,index=2,format=raw,cache=none -drive if=ide,media=cdrom,index=2,format=raw -net nic,macaddr=56:16:7a:07:61:bd,vlan=0,model=virtio,name=virtio.0 -net tap,fd=45,vlan=0,name=tap.0 -net nic,macaddr=56:16:18:c0:78:1b,vlan=1,model=virtio,name=virtio.1 -net tap,fd=46,vlan=1,name=tap.1 -chardev file,id=serial0,path=/var/log/libvirt/instance-123.serial.log -serial chardev:serial0 -parallel none -usb -vnc 127.0.0.1:0 -vga std -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/732811 Title: Kernel Oops - BUG: unable to handle kernel paging request at 0000000041d0f980; RIP: 0010:[<ffffffff8105287d>] [<ffffffff8105287d>] task_rq_lock+0x4d/0xa0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/732811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
