Re: [Qemu-devel] [RFH] Qemu main thread is blocked in g_poll in windows guest
On 10/15/2013 04:18 PM, Xiexiangyou wrote: Thanks for your reply :-) The QEMU version is 1.5.1,and the KVM version is 3.6 QEMU command: /usr/bin/qemu-kvm -name win2008_dc_5 -S -machine pc-i440fx-1.5,accel=kvm,usb=off -m 2048 -realtime mlock=off -smp 4,maxcpus=64,sockets=16,cores=4,threads=1 -uuid 13e08e3e-cd23-4450-8bd3-60e7c220316d -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win2008_dc_5.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -no-hpet -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/dev/vmdisk/win2008_dc_5,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none,aio=native -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:16:49:23,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/run/libvirt/qe m u/win2008_dc_5.extend,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1 -chardev socket,id=charchannel1,path=/var/run/libvirt/qemu/win2008_dc_5.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -vnc 0.0.0.0:4 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 (gdb) bt #0 0x7f9ba661a423 in poll () from /lib64/libc.so.6 #1 0x0059460f in os_host_main_loop_wait (timeout=4294967295) at main-loop.c:226 #2 0x005946a4 in main_loop_wait (nonblocking=0) at main-loop.c:464 #3 0x00619309 in main_loop () at vl.c:2182 #4 0x0061fb5e in main (argc=54, argv=0x7fff879830c8, envp=0x7fff87983280) at vl.c:4611 Main thread's strace message: # strace -p 6386 Process 6386 attached - interrupt to quit restart_syscall(... resuming interrupted call ... cpu thread's strace message: # strace -p 6389 Process 6389 attached - interrupt to quit rt_sigtimedwait([BUS USR1], 0x7f9ba36fbc00) = -1 EAGAIN (Resource temporarily unavailable) rt_sigpending([]) = 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ... Thanks! --xie -Original Message- From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo Bonzini Sent: Tuesday, October 15, 2013 7:52 PM To: Xiexiangyou Cc: qemu-devel@nongnu.org; qemu-devel-requ...@nongnu.org; k...@vger.kernel.org; Huangpeng (Peter); Luonengjun Subject: Re: [RFH] Qemu main thread is blocked in g_poll in windows guest Il 15/10/2013 12:21, Xiexiangyou ha scritto: Hi all: Windows2008 Guest run without pressure for long time. Sometimes, it stop and looks like hanging. But when I connect to it with VNC, It resume to run, but VM's time is delayed . When the vm is hanging, I check the main thread of QEMU. I find that the thread is blocked in g_poll function. it is waiting for a SIG, However, there is no SIG . I tried the clock with hpet and no hpet, but came out the same problem. Then I upgrade the glibc to newer, it didn't work too. I'm confused. Is the reason that VM in sleep state and doesn't emit the signal. I set the windows 's power option, enable/disable the allow the wake timers, I didn't work. Is anybody have met the same problem before, or know the reason. Your reply will be very helpful. This post is missing a few pieces of information: * What version of QEMU is this? * What is the command line? * How do you know g_poll is waiting for a signal and not for a file descriptor? * What is the backtrace of the main thread? What is the backtrace of the VCPU thread? etc. Paolo -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello, To revive this thread - I have exactly the same problem on freshly migrated virtual machines. The guest operating system is almost always Linux, bug impact ratio is very low, about one per tens of migrations. VM
Re: [Qemu-devel] [RFH] Qemu main thread is blocked in g_poll in windows guest
We have the exactly same problem on our NUMA systems. The clock of windows 2008 guests sometimes just stops and it becomes slow when you are connected via rdp. After connecting to qemu's vnc server everything runs fine again. The problem seems to occour more often when the host is under pressure. What really is weird is that this only happens to the windows 2008 guest. Linux and windows 2003 guests run just fine. We have also only seen the problem on our NUMA hosts, even with cpu and memory pinning. The singel socket servers run the exactly same software without that problem. We have also tried many different configurations on the host(CONFIG_HZ, CONFIG_PREEMTION, CONFIG_NUMA_BALANCING), qemu (different timers, different catchup policies, different spinlock settings) and on the guest side (use platformclock) but none did solve the problem Qemu version: 1.5.3 Linux: 3.11.4 glibc: 2.15 gcc: 4.7.3 # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 24 On-line CPU(s) list: 0-23 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 44 Stepping: 2 CPU MHz: 2401.000 BogoMIPS: 4800.18 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 12288K NUMA node0 CPU(s): 0-5,12-17 NUMA node1 CPU(s): 6-11,18-23 # lspci 00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22) 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22) 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22) 00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22) 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22) 00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22) 00:0d.0 Host bridge: Intel Corporation Device 343a (rev 22) 00:0d.1 Host bridge: Intel Corporation Device 343b (rev 22) 00:0d.2 Host bridge: Intel Corporation Device 343c (rev 22) 00:0d.3 Host bridge: Intel Corporation Device 343d (rev 22) 00:0d.4 Host bridge: Intel Corporation 7500/5520/5500/X58 Physical Layer Port 0 (rev 22) 00:0d.5 Host bridge: Intel Corporation 7500/5520/5500 Physical Layer Port 1 (rev 22) 00:0d.6 Host bridge: Intel Corporation Device 341a (rev 22) 00:0e.0 Host bridge: Intel Corporation Device 341c (rev 22) 00:0e.1 Host bridge: Intel Corporation Device 341d (rev 22) 00:0e.2 Host bridge: Intel Corporation Device 341e (rev 22) 00:0e.4 Host bridge: Intel Corporation Device 3439 (rev 22) 00:13.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 22) 00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 22) 00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22) 00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22) 00:14.3 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers (rev 22) 00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) 00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) 00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) 00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) 00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) 00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) 00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) 00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) 00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5 00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6 00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90) 00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R)
[Qemu-devel] [RFH] Qemu main thread is blocked in g_poll in windows guest
Hi all: Windows2008 Guest run without pressure for long time. Sometimes, it stop and looks like hanging. But when I connect to it with VNC, It resume to run, but VM's time is delayed . When the vm is hanging, I check the main thread of QEMU. I find that the thread is blocked in g_poll function. it is waiting for a SIG, However, there is no SIG . I tried the clock with hpet and no hpet, but came out the same problem. Then I upgrade the glibc to newer, it didn't work too. I'm confused. Is the reason that VM in sleep state and doesn't emit the signal. I set the windows 's power option, enable/disable the allow the wake timers, I didn't work. Is anybody have met the same problem before, or know the reason. Your reply will be very helpful. Thanks --xiexiangyou
Re: [Qemu-devel] [RFH] Qemu main thread is blocked in g_poll in windows guest
Il 15/10/2013 12:21, Xiexiangyou ha scritto: Hi all: Windows2008 Guest run without pressure for long time. Sometimes, it stop and looks like hanging. But when I connect to it with VNC, It resume to run, but VM's time is delayed . When the vm is hanging, I check the main thread of QEMU. I find that the thread is blocked in g_poll function. it is waiting for a SIG, However, there is no SIG . I tried the clock with hpet and no hpet, but came out the same problem. Then I upgrade the glibc to newer, it didn't work too. I'm confused. Is the reason that VM in sleep state and doesn't emit the signal. I set the windows 's power option, enable/disable the allow the wake timers, I didn't work. Is anybody have met the same problem before, or know the reason. Your reply will be very helpful. This post is missing a few pieces of information: * What version of QEMU is this? * What is the command line? * How do you know g_poll is waiting for a signal and not for a file descriptor? * What is the backtrace of the main thread? What is the backtrace of the VCPU thread? etc. Paolo
Re: [Qemu-devel] [RFH] Qemu main thread is blocked in g_poll in windows guest
Thanks for your reply :-) The QEMU version is 1.5.1,and the KVM version is 3.6 QEMU command: /usr/bin/qemu-kvm -name win2008_dc_5 -S -machine pc-i440fx-1.5,accel=kvm,usb=off -m 2048 -realtime mlock=off -smp 4,maxcpus=64,sockets=16,cores=4,threads=1 -uuid 13e08e3e-cd23-4450-8bd3-60e7c220316d -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win2008_dc_5.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -no-hpet -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/dev/vmdisk/win2008_dc_5,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none,aio=native -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:16:49:23,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/run/libvirt/qemu/win2008_dc_5.extend,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1 -chardev socket,id=charchannel1,path=/var/run/libvirt/qemu/win2008_dc_5.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -vnc 0.0.0.0:4 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 (gdb) bt #0 0x7f9ba661a423 in poll () from /lib64/libc.so.6 #1 0x0059460f in os_host_main_loop_wait (timeout=4294967295) at main-loop.c:226 #2 0x005946a4 in main_loop_wait (nonblocking=0) at main-loop.c:464 #3 0x00619309 in main_loop () at vl.c:2182 #4 0x0061fb5e in main (argc=54, argv=0x7fff879830c8, envp=0x7fff87983280) at vl.c:4611 Main thread's strace message: # strace -p 6386 Process 6386 attached - interrupt to quit restart_syscall(... resuming interrupted call ... cpu thread's strace message: # strace -p 6389 Process 6389 attached - interrupt to quit rt_sigtimedwait([BUS USR1], 0x7f9ba36fbc00) = -1 EAGAIN (Resource temporarily unavailable) rt_sigpending([]) = 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 ... Thanks! --xie -Original Message- From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo Bonzini Sent: Tuesday, October 15, 2013 7:52 PM To: Xiexiangyou Cc: qemu-devel@nongnu.org; qemu-devel-requ...@nongnu.org; k...@vger.kernel.org; Huangpeng (Peter); Luonengjun Subject: Re: [RFH] Qemu main thread is blocked in g_poll in windows guest Il 15/10/2013 12:21, Xiexiangyou ha scritto: Hi all: Windows2008 Guest run without pressure for long time. Sometimes, it stop and looks like hanging. But when I connect to it with VNC, It resume to run, but VM's time is delayed . When the vm is hanging, I check the main thread of QEMU. I find that the thread is blocked in g_poll function. it is waiting for a SIG, However, there is no SIG . I tried the clock with hpet and no hpet, but came out the same problem. Then I upgrade the glibc to newer, it didn't work too. I'm confused. Is the reason that VM in sleep state and doesn't emit the signal. I set the windows 's power option, enable/disable the allow the wake timers, I didn't work. Is anybody have met the same problem before, or know the reason. Your reply will be very helpful. This post is missing a few pieces of information: * What version of QEMU is this? * What is the command line? * How do you know g_poll is waiting for a signal and not for a file descriptor? * What is the backtrace of the main thread? What is the backtrace of the VCPU thread? etc. Paolo
Re: [Qemu-devel] [RFH] Qemu main thread is blocked in g_poll in windows guest
Il 15/10/2013 14:18, Xiexiangyou ha scritto: (gdb) bt #0 0x7f9ba661a423 in poll () from /lib64/libc.so.6 #1 0x0059460f in os_host_main_loop_wait (timeout=4294967295) at main-loop.c:226 #2 0x005946a4 in main_loop_wait (nonblocking=0) at main-loop.c:464 #3 0x00619309 in main_loop () at vl.c:2182 #4 0x0061fb5e in main (argc=54, argv=0x7fff879830c8, envp=0x7fff87983280) at vl.c:4611 This backtrace looks okay. poll is not holding the iothread lock and thus the VCPU thread can run QEMU code. It also looks like the VCPU thread is doing nothing special besides running KVM_RUN. Did you take the backtrace during one of these hangups? Paolo
Re: [Qemu-devel] [RFH] Qemu main thread is blocked in g_poll in windows guest
The backtrace of VCPU in hangup state is: (gdb) bt #0 0x7f9ba661bc67 in ioctl () from /lib64/libc.so.6 #1 0x0069631d in kvm_vcpu_ioctl (cpu=0x169f880, type=44672) at /home/qemu-kvm-1.5.1/kvm-all.c:1735 #2 0x00695bc6 in kvm_cpu_exec (env=0x169f990) at /home/qemu-kvm-1.5.1/kvm-all.c:1620 #3 0x00625feb in qemu_kvm_cpu_thread_fn (arg=0x169f990) at /home/qemu-kvm-1.5.1/cpus.c:759 #4 0x7f9ba68e0f05 in start_thread () from /lib64/libpthread.so.0 #5 0x7f9ba662353d in clone () from /lib64/libc.so.6 There is no problem can be found from this. However, trace the VCPU thread's syscall, we can find that it only run KVM_RUN ioctl , the strace message always is: ioctl(17, 0xae80, 0)= 0 The normal VCPU thread's strace message like this: ioctl(17, 0xae80, 0)= 0 ioctl(11, 0xc008ae67, 0x7fb157ffeab0) = 0 ioctl(17, 0xae80, 0)= 0 ioctl(11, 0xc008ae67, 0x7fb157ffeab0) = 0 ioctl(17, 0xae80, 0)= 0 futex(0xf9d260, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) futex(0xf9d260, FUTEX_WAKE_PRIVATE, 1) = 0 ioctl(17, 0xae80, 0)= 0 ioctl(17, 0xae80, 0)= 0 The issue is hard to be reproduced, except when I let the VM in idle state, it recur in several hours. Thanks --xie -Original Message- From: Paolo Bonzini [mailto:pbonz...@redhat.com] Sent: Tuesday, October 15, 2013 8:30 PM To: Xiexiangyou Cc: qemu-devel@nongnu.org; qemu-devel-requ...@nongnu.org; k...@vger.kernel.org; Huangpeng (Peter); Luonengjun Subject: Re: [RFH] Qemu main thread is blocked in g_poll in windows guest Il 15/10/2013 14:18, Xiexiangyou ha scritto: (gdb) bt #0 0x7f9ba661a423 in poll () from /lib64/libc.so.6 #1 0x0059460f in os_host_main_loop_wait (timeout=4294967295) at main-loop.c:226 #2 0x005946a4 in main_loop_wait (nonblocking=0) at main-loop.c:464 #3 0x00619309 in main_loop () at vl.c:2182 #4 0x0061fb5e in main (argc=54, argv=0x7fff879830c8, envp=0x7fff87983280) at vl.c:4611 This backtrace looks okay. poll is not holding the iothread lock and thus the VCPU thread can run QEMU code. It also looks like the VCPU thread is doing nothing special besides running KVM_RUN. Did you take the backtrace during one of these hangups? Paolo
