[Bug 1322441] Re: QEMU Windows guest unstable after random amount of time
Tested with kernel - Linux 3.15.0-031500rc7-generic After 24 hours there's no sign of the problem. After watching numastat for a while I noticed that with the 3.13.0 kernel the allocated memory for qemu does a really sudden drop from 6GB down to 2GB. QEMU then freezes with 100% CPU until the allocation goes back up to 6GB again. On 3.15 the memory stays almost constant and is migrated towards one numa node as you'd expect. Just speculating but there must be some kind of page invalidation bug in the numa balancer. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: QEMU Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] Re: QEMU Windows guest unstable after random amount of time
Not this exact problem but one with similar symptoms. From kernel 3.5 onwards there was a problem with guests not receiving RTC ticks which was reported here - http://lists.gnu.org/archive/html/qemu- devel/2013-02/msg03827.html The 3.2 kernel from 12.04.1 never had any issues but the 3.5 kernel from 12.04.2 onwards hit the RTC bug whoch was fixed in later releases of QEMU. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: QEMU Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1322441/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1322441] Re: QEMU Windows guest unstable after random amount of time
Based on the fact that flush_tlb_page() appears to be taking up most of the CPU time when QEMU is 'frozen' I started playing with memory settings. Disabling automatic NUMA balancing appears to have solved the problem. echo 0 /proc/sys/kernel/numa_balancing The guest has been running 4+ hours now without an issue. I currently have KSM and transparent huge pages enabled. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: QEMU Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1322441/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1322441] Re: QEMU Windows guest unstable after random amount of time
BIOS has now been updated # sudo dmidecode -s bios-version sudo dmidecode -s bios-release-date 2.1c 04/22/2014 I've been able to reproduce the problem using a fresh Windows 2008R2 server image. The guest functions correctly for a period of time then starts to 'freeze' for around 10 seconds every 5-20 seconds. While frozen the CPU usage of QEMU jumps to around 105%. Attached is the updated perf record. 11.74% qemu-system-x86 [kernel.kallsyms][k] flush_tlb_page 6.27% qemu-system-x86 [kernel.kallsyms][k] kvm_handle_hva_range 4.98% qemu-system-x86 [kernel.kallsyms][k] __srcu_read_lock 4.76% qemu-system-x86 [kernel.kallsyms][k] try_to_unmap_ksm 4.62% qemu-system-x86 [kernel.kallsyms][k] _raw_spin_unlock 4.30% qemu-system-x86 [kernel.kallsyms][k] up_read 4.05% qemu-system-x86 [kernel.kallsyms][k] down_read 3.97% qemu-system-x86 [kernel.kallsyms][k] try_to_unmap_one 3.85% qemu-system-x86 [kernel.kallsyms][k] generic_exec_single 3.72% qemu-system-x86 [kernel.kallsyms][k] _raw_spin_lock 3.45% qemu-system-x86 [kernel.kallsyms][k] __srcu_read_unlock 3.18% qemu-system-x86 [kernel.kallsyms][k] mm_find_pmd 3.00% qemu-system-x86 [kernel.kallsyms][k] rmap_walk_ksm 2.90% qemu-system-x86 [kernel.kallsyms][k] cpumask_any_but 2.87% qemu-system-x86 [kernel.kallsyms][k] remove_migration_pte 2.25% qemu-system-x86 [kernel.kallsyms][k] __anon_vma_interval_tree_subtree_search 1.64% qemu-system-x86 [kernel.kallsyms][k] kvm_unmap_rmapp ** Attachment added: perf.data https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+attachment/4119371/+files/perf.data -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: QEMU Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] Re: QEMU Windows guest unstable after random amount of time
Not this exact problem but one with similar symptoms. From kernel 3.5 onwards there was a problem with guests not receiving RTC ticks which was reported here - http://lists.gnu.org/archive/html/qemu- devel/2013-02/msg03827.html The 3.2 kernel from 12.04.1 never had any issues but the 3.5 kernel from 12.04.2 onwards hit the RTC bug whoch was fixed in later releases of QEMU. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: QEMU Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] Re: QEMU Windows guest unstable after random amount of time
Based on the fact that flush_tlb_page() appears to be taking up most of the CPU time when QEMU is 'frozen' I started playing with memory settings. Disabling automatic NUMA balancing appears to have solved the problem. echo 0 /proc/sys/kernel/numa_balancing The guest has been running 4+ hours now without an issue. I currently have KSM and transparent huge pages enabled. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: QEMU Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] [NEW] Windows guest unstable after random amount of time
Public bug reported: Ubuntu 14.04, all updates done as of 23/5/2014 Kernel : Linux 3.13.0-24-generic Qemu : 1.7.1 and 2.0.0 tested Tested using both Xeon 5620 and 5520 processors, 48GB RAM. Anywhere from 20 minutes for 3+ hours after booting Windows guests become unstable. Guest appear to freeze intermittently (VNC console unresponsive, network pings dropped to guest, frozen IO) for 20-60 seconds at a time which repeats constantly every few minutes and guests do not recover unless QEMU is killed and the guest is started again. Whilst the guest is frozen CPU usage of the QEMU process jumps to 100-150%. ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Attachment added: lspci-vnvn.log https://bugs.launchpad.net/bugs/1322441/+attachment/4118035/+files/lspci-vnvn.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] Re: Windows guest unstable after random amount of time
The bug occurred when upgrading from 12.04.1 (3.2 kernel) to 14.04 (3.13 kernel). Booting with the old 3.2 kernel resolves the problem. Guest command line - qemu-system-x86_64 -enable-kvm -name test -S-machine pc-i440fx-1.4,accel=kvm,usb=off -cpu qemu64,hv_relaxed -m 4096 -realtimemlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid e00d6bde-ca3e-8e6c-2c92-d5b5ec632b9a -no-user-config -nodefaults -chardevsocket,id=charmonitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait -monchardev=charmonitor,id=monitor,mode=contro l-rtcbase=localtime,driftfix=slew -no-hpet -no-shutdown -bootstrict=on -devicepiix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=rbd:ssd/test:auth_supported=none,if=none,id=drive-virtio-disk0-devicevirtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw-deviceide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdevtap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -devicevirtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:81:8a,bus=pci.0,addr=0x3-chardevpty,id=charserial0 -device isa-serial,chardev=charse rial0,id=serial0-vnc127.0.0.1:0 -devicecirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 Attached is a perf record of the qemu process whilst the fault was occuring. ** Attachment added: perf.data https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+attachment/4118045/+files/perf.data -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] Re: Windows guest unstable after random amount of time
apport information ** Tags added: apport-collected trusty ** Description changed: Ubuntu 14.04, all updates done as of 23/5/2014 Kernel : Linux 3.13.0-24-generic Qemu : 1.7.1 and 2.0.0 tested Tested using both Xeon 5620 and 5520 processors, 48GB RAM. - Anywhere from 20 minutes for 3+ hours after booting Windows guests - become unstable. Guest appear to freeze intermittently (VNC console - unresponsive, network pings dropped to guest, frozen IO) for 20-60 - seconds at a time which repeats constantly every few minutes and guests - do not recover unless QEMU is killed and the guest is started again. - Whilst the guest is frozen CPU usage of the QEMU process jumps to - 100-150%. + Anywhere from 20 minutes for 3+ hours after booting Windows guests become unstable. Guest appear to freeze intermittently (VNC console unresponsive, network pings dropped to guest, frozen IO) for 20-60 seconds at a time which repeats constantly every few minutes and guests do not recover unless QEMU is killed and the guest is started again. Whilst the guest is frozen CPU usage of the QEMU process jumps to 100-150%. + --- + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 May 24 01:25 seq + crw-rw 1 root audio 116, 33 May 24 01:25 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.14.1-0ubuntu3.1 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + CRDA: Error: [Errno 2] No such file or directory + DistroRelease: Ubuntu 14.04 + HibernationDevice: RESUME=UUID=38dc08b6-55f7-482d-8f82-f048b3dbad56 + InstallationDate: Installed on 2013-03-21 (428 days ago) + InstallationMedia: Ubuntu-Server 12.04.1 LTS Precise Pangolin - Release amd64 (20120817.3) + MachineType: Supermicro X8DTT + Package: linux (not installed) + PciMultimedia: + + ProcFB: 0 VESA VGA + ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-24-generic root=UUID=386ad1f0-7ced-414f-9a09-4c078c36977c ro nomdmonddf nomdmonisw nomdmonddf nomdmonisw + ProcVersionSignature: Ubuntu 3.13.0-24.47-generic 3.13.9 + RelatedPackageVersions: + linux-restricted-modules-3.13.0-24-generic N/A + linux-backports-modules-3.13.0-24-generic N/A + linux-firmware 1.127.2 + RfKill: Error: [Errno 2] No such file or directory + Tags: trusty + Uname: Linux 3.13.0-24-generic x86_64 + UpgradeStatus: Upgraded to trusty on 2014-05-21 (2 days ago) + UserGroups: + + _MarkForUpload: True + dmi.bios.date: 05/20/2010 + dmi.bios.vendor: American Megatrends Inc. + dmi.bios.version: 080016 + dmi.board.asset.tag: 1234567890 + dmi.board.name: X8DTT + dmi.board.vendor: Supermicro + dmi.board.version: 2.0 + dmi.chassis.asset.tag: To Be Filled By O.E.M. + dmi.chassis.type: 17 + dmi.chassis.vendor: Supermicro + dmi.chassis.version: 1234567890 + dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr080016:bd05/20/2010:svnSupermicro:pnX8DTT:pvr1234567890:rvnSupermicro:rnX8DTT:rvr2.0:cvnSupermicro:ct17:cvr1234567890: + dmi.product.name: X8DTT + dmi.product.version: 1234567890 + dmi.sys.vendor: Supermicro ** Attachment added: BootDmesg.txt https://bugs.launchpad.net/bugs/1322441/+attachment/4118528/+files/BootDmesg.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] CurrentDmesg.txt
apport information ** Attachment added: CurrentDmesg.txt https://bugs.launchpad.net/bugs/1322441/+attachment/4118529/+files/CurrentDmesg.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] ProcInterrupts.txt
apport information ** Attachment added: ProcInterrupts.txt https://bugs.launchpad.net/bugs/1322441/+attachment/4118535/+files/ProcInterrupts.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] Lspci.txt
apport information ** Attachment added: Lspci.txt https://bugs.launchpad.net/bugs/1322441/+attachment/4118531/+files/Lspci.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] ProcEnviron.txt
apport information ** Attachment added: ProcEnviron.txt https://bugs.launchpad.net/bugs/1322441/+attachment/4118534/+files/ProcEnviron.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] Lsusb.txt
apport information ** Attachment added: Lsusb.txt https://bugs.launchpad.net/bugs/1322441/+attachment/4118532/+files/Lsusb.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] UdevDb.txt
apport information ** Attachment added: UdevDb.txt https://bugs.launchpad.net/bugs/1322441/+attachment/4118537/+files/UdevDb.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] IwConfig.txt
apport information ** Attachment added: IwConfig.txt https://bugs.launchpad.net/bugs/1322441/+attachment/4118530/+files/IwConfig.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] WifiSyslog.txt
apport information ** Attachment added: WifiSyslog.txt https://bugs.launchpad.net/bugs/1322441/+attachment/4118539/+files/WifiSyslog.txt ** Changed in: linux (Ubuntu) Status: Incomplete = Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] UdevLog.txt
apport information ** Attachment added: UdevLog.txt https://bugs.launchpad.net/bugs/1322441/+attachment/4118538/+files/UdevLog.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] ProcCpuinfo.txt
apport information ** Attachment added: ProcCpuinfo.txt https://bugs.launchpad.net/bugs/1322441/+attachment/4118533/+files/ProcCpuinfo.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1322441] ProcModules.txt
apport information ** Attachment added: ProcModules.txt https://bugs.launchpad.net/bugs/1322441/+attachment/4118536/+files/ProcModules.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1322441 Title: Windows guest unstable after random amount of time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1322441/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs