[Qemu-devel] Is it necessary to add a Monitor event of QEVENT_DEVICE_ADDED

[xiexiangyou]

Hi all,

   As we know, when hot-add a device, we don't know whether the device is 
present in VM from outside(like libvirt).
However, unplugging device, when complete remove the device, Qemu will generate 
a Monitor event,QEVENT_DEVICE_DELETED.
Libvirt can receive the event and do something.
   So is it necessary to add a Monitor event of completing add device such as 
QEVENT_DEVICE_ADDED.

Regards,
-xie

Re: [Qemu-devel] BUG: RTC issue when Windows guest is idle

[Xiexiangyou]

Hi Anderson,

RTC timer may stop after live migration if you set the rtc_clock = host_clock. 
Because the different hosts have different system time.
Rtc is waiting for the next_periodic_time after migrating, it may wait more 
longer. During the time, VM will lose one tick with great possibility,
and the RTC periodic timer interrupt is blocked by "ioapic->irr==1" in KVM 
module.
To solve the problem, you can try "rtc_clock = vm_clock"(HPET is ok because it 
using vm_clock). Like this:
  


  

I hope it will help you.

Best wishes
---
Xiexiangyou

-Original Message-
From: Matthew Anderson [mailto:matth...@base3.com.au] 
Sent: Monday, October 28, 2013 2:59 PM
To: 'Alex Bligh'; Xiexiangyou
Cc: Stefan Hajnoczi; qemu-devel@nongnu.org
Subject: RE: [Qemu-devel] BUG: RTC issue when Windows guest is idle

Hi Alex,

I've been doing some testing with the latest git version and so far I haven't 
seen a guest freeze in the same circumstances as before. 

A weird thing that has been happening is the RTC timer stopping after a live 
migration. This happened in both 1.6.1 and the 1.6.50 git build. To replicate 
the issue I was migrating to/from the same machine and anywhere between 1 and 3 
migrations the guest clock would stop. Connecting to the VNC console would not 
get it running again. I've tried to replicate the issue with the HPET enabled 
but the guest clock works flawlessly with it enabled.

Guest VM is Windows 2008R2. Host is Ubuntu 13.04 (kernel 3.8.0-25-generic)
Command line -
/usr/local/bin/qemu-system-x86_64 -enable-kvm -nodefconfig -nodefaults 
-daemonize -usb -chardev 
socket,id=charmonitor,path=/var/run/based1/monitor/525ce3d009c437d67802.monitor,server,nowait
 -mon chardev=charmonitor,id=monitor,mode=control -chardev 
socket,path=/var/run/based1/ga/525ce3d009c437d67802.guestagent,server,nowait,id=qga0
 -device virtio-serial -device 
virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -pidfile 
/var/run/based1/pid/525ce3d009c437d67802.pid -vga cirrus -vnc 0.0.0.0:1 -M 
pc-i440fx-1.6 -m 1024 -smp sockets=1,cores=1,threads=1 -cpu 
qemu64,+vme,+dts,+acpi,+dtes64,+vmx,+smx,+ssse3,+sse4_1,+sse4_2,+tpr_shadow,+vnmi,+flexpriority,+ept,+vpid,hv_relaxed,hv_spinlocks=0x,hv_vapic
 -rtc base=utc,driftfix=slew --no-hpet -drive 
aio=native,file=rbd:sata/525ce3d009c437d67803,if=virtio,id=disk-525ce3d009c437d67805,format=raw,cache=writeback,media=disk,index=0,addr=0xa
 -netdev 
tap,id=netdev-5264b6d46e53c81719000236,vhost=off,ifname=tap2,script=no,downscript=no
 -device 
virtio-net-pci,netdev=netdev-5264b6d46e53c81719000236,id=interface-5264b6d46e53c81719000236,mac=9a:a5:63:64:6f:76,bus=pci.0,addr=0xb
 -incoming tcp:0:3004

Thanks
-Matt


-Original Message-
From: Alex Bligh [mailto:a...@alex.org.uk] 
Sent: Tuesday, 22 October 2013 5:36 PM
To: Xiexiangyou; Matthew Anderson
Cc: Stefan Hajnoczi; qemu-devel@nongnu.org; Alex Bligh
Subject: Re: [Qemu-devel] BUG: RTC issue when Windows guest is idle



--On 22 October 2013 08:28:08 + Xiexiangyou 
wrote:

> Hi:
>
> I have run windows2008r2 guest with qemu-1.5.1/1.6(I have not test the 
> the newer  version) for long time, the issue (guest in hangup state) 
> will come out.When guest  in hangup state, QEMU main thread is blocked 
> in g_poll loop.
>> 398c398,399
>> < uint32_t timeout = UINT32_MAX;
>> ---
>>> /* uint32_t timeout = UINT32_MAX; */
>>> uint32_t timeout = 1000;
>>
> It seems can fix the problem, and rtc/hpet interrupt can inject into 
> guest again  because of the "timeout", and guest will wake up . But 
> maybe the issue is also exist,  because during the time before timeout 
> , guest also will lose rtc/hpet ticks.

I do not think that is the correct fix for 1.5.1/1.6; what you are basically 
doing is limiting the wait in the mainloop to one second (1.5.1/1.6 are in 
milliseconds); however, I believe there may be other code that looks for 
infinite timeouts. Either there is some other bug that this is masking (in 
which case it may or may not be fixed in master / 1.7), or its a bug in the 
timer stuff in 1.5.1/1.6 (which would not surprise me) which is likely to have 
been fixed in master / 1.7.

--
Alex Bligh

Re: [Qemu-devel] BUG: RTC issue when Windows guest is idle

[Xiexiangyou]

Hi Alex：
I have been test with the QEMU Master version for several days, and the 
issue is not appear too. 
I think you have fixed it using the timeout instead of alarm timer! 
It' great!
But I want to know why alarm timer will make the problem, is the reason that 
losing the alarm event?

Thanks
---
xiexiangyou  

-Original Message-
From: Alex Bligh [mailto:a...@alex.org.uk] 
Sent: Monday, October 28, 2013 3:45 PM
To: Matthew Anderson
Cc: Alex Bligh; Xiexiangyou; Stefan Hajnoczi; qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] BUG: RTC issue when Windows guest is idle


On 28 Oct 2013, at 06:58, Matthew Anderson wrote:

> I've been doing some testing with the latest git version and so far I haven't 
> seen a guest freeze in the same circumstances as before. 

That's good news.

> A weird thing that has been happening is the RTC timer stopping after a live 
> migration. This happened in both 1.6.1 and the 1.6.50 git build. To replicate 
> the issue I was migrating to/from the same machine and anywhere between 1 and 
> 3 migrations the guest clock would stop. Connecting to the VNC console would 
> not get it running again. I've tried to replicate the issue with the HPET 
> enabled but the guest clock works flawlessly with it enabled.

Does that one happen on master/1.7 as well? Oddly we saw this one or something 
like it on Xen+qemu.

-- 
Alex Bligh

Re: [Qemu-devel] BUG: RTC issue when Windows guest is idle

[Xiexiangyou]

Hi:

I have run windows2008r2 guest with qemu-1.5.1/1.6(I have not test the the newer
 version) for long time, the issue (guest in hangup state) will come out.When 
guest 
in hangup state, QEMU main thread is blocked in g_poll loop.
> 398c398,399
> < uint32_t timeout = UINT32_MAX;
> ---
>> /* uint32_t timeout = UINT32_MAX; */
>> uint32_t timeout = 1000;
>
It seems can fix the problem, and rtc/hpet interrupt can inject into guest 
again 
because of the "timeout", and guest will wake up . But maybe the issue is also 
exist, 
because during the time before timeout , guest also will lose rtc/hpet ticks. 

Regards
xiexiangyou

-Original Message-
From: Alex Bligh [mailto:a...@alex.org.uk] 
Sent: Monday, October 21, 2013 10:56 PM
To: Matthew Anderson; Xiexiangyou
Cc: qemu-devel@nongnu.org; Alex Bligh; Stefan Hajnoczi
Subject: Re: [Qemu-devel] BUG: RTC issue when Windows guest is idle



--On 16 October 2013 14:56:12 + Matthew Anderson 
 wrote:

> Hi Xiangyouxie,
>
> I personally haven't tried to solve the problem as yet but I've been in
> contact with Anders Fudali who was able to find the issue with the help
> of one of his developers.
>
> See below for his comments. I've love to hear from any of the devs that
> can explain the issue further because as far as I know it's still an
> issue although I haven't with tried any releases past 1.4.1.

It would be interesting to know whether this happens on a current qemu
(i.e. master/1.7/whatever, not 1.6).

The timeout code in main-loop.c has been changed. For context, in
1.4.2, this does:

395 int main_loop_wait(int nonblocking)
396 {
397 int ret;
398 uint32_t timeout = UINT32_MAX;
399
400 if (nonblocking) {
401 timeout = 0;
402 }
403

and I can promise that code has been changed. This might have fixed
things or (in other cases) made an existing bug more obvious.

I'm not sure whether the RTC is meant to be generating some sort of
qemu_notify() here, or using a timer, or what, but it should be doing
something to break out of the select() loop. Stefan - any idea?

Alex

> 
>
> We use QEMU-1.4.2 and Kernel-3.8.8 on our host machines. During the
> latest "RTC-freeze" we managed to strace the affected QEMU-process and
> one of our developers figured out that a file descriptor selector loops
> with a NULL value for timeout... we adjusted the source code for the
> file: main-loop.c and replaced UINT32_MAX with the value: 1000, see the
> diff below:
>
> 398c398,399
> < uint32_t timeout = UINT32_MAX;
> ---
>> /* uint32_t timeout = UINT32_MAX; */
>> uint32_t timeout = 1000;
>
> I'm not sure IF this might have some other undesirable effects or
> consequence and why only Windows guests seems to be affected, but we've
> been running with this "fix" for > 3 days now and haven't seen the
> problem since.
>
> Thought to share this with you.
>
> Regards
>
> Anders Fudali
>
> --
>
> From: Xiexiangyou [mailto:xiexiang...@huawei.com]
> Sent: Tuesday, 8 October 2013 5:39 PM
> To: Matthew Anderson
> Cc: qemu-devel@nongnu.org
> Subject: Re: [Qemu-devel] BUG: RTC issue when Windows guest is idle
>
> Hi:
>  I have met the same bug that windows2008 guest stop
> receive the RTC ticks when it in idle status by fortuitous. When vnc
> connect, guest will resume to receive RTC ticks and the time run fast
> because of the coalesced timer
>
> HPET is diabled, and RTC is set "catchup", as following:
>   
>     
>     
>   
>
> My kvm module is version3.6. Should I upgrade it to latest version. Any
> suggestion is welcome !
>
> Thanks!
>
> --xiangyouxie  
>   
>
> On Thu, Feb 21, 2013 at 06:16:10PM +, Matthew Anderson wrote:
>> If this isn't the correct list just let me know,
>>
>> I've run into a bug whereby a Windows guest (tested on Server 2008R2 and
>> 2012) no longer receives RTC ticks when it has been idle for a random
>> amount  of time. HPET is disabled and the guest is running Hyper-V
>> relaxed timers  (same situation without hv_relaxed). The guest clock
>> stands still and the  qemu process uses very little CPU (<0.5%, normally
>> it's >5% when the guest is  idle) . Eventually the guest stops
>> responding to network requests but if you  open the guest console via
>> VNC and move the mouse around it comes back to  life and QEMU replays
>> the lost RTC ticks and the guest recovers. I've also  been able to make
>> it recover by querying the clock over the network via the  ne

Re: [Qemu-devel] [RFH] Qemu main thread is blocked in g_poll in windows guest

[Xiexiangyou]

The backtrace of VCPU in "hangup" state is:
(gdb) bt
#0  0x7f9ba661bc67 in ioctl () from /lib64/libc.so.6
#1  0x0069631d in kvm_vcpu_ioctl (cpu=0x169f880, type=44672) at 
/home/qemu-kvm-1.5.1/kvm-all.c:1735
#2  0x00695bc6 in kvm_cpu_exec (env=0x169f990) at 
/home/qemu-kvm-1.5.1/kvm-all.c:1620
#3  0x00625feb in qemu_kvm_cpu_thread_fn (arg=0x169f990) at 
/home/qemu-kvm-1.5.1/cpus.c:759
#4  0x7f9ba68e0f05 in start_thread () from /lib64/libpthread.so.0
#5  0x7f9ba662353d in clone () from /lib64/libc.so.6
There is no problem can be found from this.

However, trace the VCPU thread's syscall, we can find that it only run KVM_RUN 
ioctl , the strace message always is:
ioctl(17, 0xae80, 0)= 0

The normal VCPU thread's strace message like this:
ioctl(17, 0xae80, 0)= 0
ioctl(11, 0xc008ae67, 0x7fb157ffeab0) = 0
ioctl(17, 0xae80, 0)= 0
ioctl(11, 0xc008ae67, 0x7fb157ffeab0) = 0
ioctl(17, 0xae80, 0)= 0
futex(0xf9d260, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily 
unavailable)
futex(0xf9d260, FUTEX_WAKE_PRIVATE, 1)  = 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0

The issue is hard to be reproduced, except when I let the VM in idle state, it 
recur in several hours. 

Thanks
--xie

-Original Message-
From: Paolo Bonzini [mailto:pbonz...@redhat.com] 
Sent: Tuesday, October 15, 2013 8:30 PM
To: Xiexiangyou
Cc: qemu-devel@nongnu.org; qemu-devel-requ...@nongnu.org; k...@vger.kernel.org; 
Huangpeng (Peter); Luonengjun
Subject: Re: [RFH] Qemu main thread is blocked in g_poll in windows guest

Il 15/10/2013 14:18, Xiexiangyou ha scritto:
> (gdb) bt
> #0  0x7f9ba661a423 in poll () from /lib64/libc.so.6
> #1  0x0059460f in os_host_main_loop_wait (timeout=4294967295) at 
> main-loop.c:226
> #2  0x005946a4 in main_loop_wait (nonblocking=0) at main-loop.c:464
> #3  0x00619309 in main_loop () at vl.c:2182
> #4  0x0061fb5e in main (argc=54, argv=0x7fff879830c8, 
> envp=0x7fff87983280) at vl.c:4611

This backtrace looks okay.  poll is not holding the "iothread lock" and
thus the VCPU thread can run QEMU code.

It also looks like the VCPU thread is doing nothing special besides
running KVM_RUN.  Did you take the backtrace during one of these "hangups"?

Paolo

Re: [Qemu-devel] [RFH] Qemu main thread is blocked in g_poll in windows guest

[Xiexiangyou]

Thanks for your reply :-)
The QEMU version is 1.5.1,and the KVM version is 3.6

QEMU command:
/usr/bin/qemu-kvm -name win2008_dc_5 -S -machine 
pc-i440fx-1.5,accel=kvm,usb=off -m 2048 -realtime mlock=off -smp 
4,maxcpus=64,sockets=16,cores=4,threads=1 -uuid 
13e08e3e-cd23-4450-8bd3-60e7c220316d -no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/win2008_dc_5.monitor,server,nowait
 -mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=utc,clock=vm,driftfix=slew -no-hpet -no-shutdown -device 
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device 
virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device 
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive 
file=/dev/vmdisk/win2008_dc_5,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none,aio=native
 -device 
scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:16:49:23,bus=pci.0,addr=0x3 
-chardev 
socket,id=charchannel0,path=/var/run/libvirt/qemu/win2008_dc_5.extend,server,nowait
 -device 
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
 -chardev 
socket,id=charchannel1,path=/var/run/libvirt/qemu/win2008_dc_5.agent,server,nowait
 -device 
virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
 -device usb-tablet,id=input0 -vnc 0.0.0.0:4 -device 
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

(gdb) bt
#0  0x7f9ba661a423 in poll () from /lib64/libc.so.6
#1  0x0059460f in os_host_main_loop_wait (timeout=4294967295) at 
main-loop.c:226
#2  0x005946a4 in main_loop_wait (nonblocking=0) at main-loop.c:464
#3  0x00619309 in main_loop () at vl.c:2182
#4  0x0061fb5e in main (argc=54, argv=0x7fff879830c8, 
envp=0x7fff87983280) at vl.c:4611

Main thread's strace message:
# strace -p 6386
Process 6386 attached - interrupt to quit
restart_syscall(<... resuming interrupted call ...>

cpu thread's strace message:
# strace -p 6389
Process 6389 attached - interrupt to quit
rt_sigtimedwait([BUS USR1], 0x7f9ba36fbc00) = -1 EAGAIN (Resource temporarily 
unavailable)
rt_sigpending([])   = 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
ioctl(17, 0xae80, 0)= 0
...

Thanks!
--xie

-Original Message-
From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo Bonzini
Sent: Tuesday, October 15, 2013 7:52 PM
To: Xiexiangyou
Cc: qemu-devel@nongnu.org; qemu-devel-requ...@nongnu.org; k...@vger.kernel.org; 
Huangpeng (Peter); Luonengjun
Subject: Re: [RFH] Qemu main thread is blocked in g_poll in windows guest

Il 15/10/2013 12:21, Xiexiangyou ha scritto:
> Hi all:
> 
> Windows2008 Guest run without pressure for long time. Sometimes, it
> stop and looks like hanging. But when I connect to it with VNC, It
> resume to run, but VM's time is delayed . When the vm is hanging, I
> check the main thread of QEMU. I find that the thread is blocked in
> g_poll function. it is waiting for a SIG, However, there is no SIG .
> 
> I tried the clock with "hpet" and "no hpet", but came out the same
> problem. Then I upgrade the glibc to newer, it didn't work too. I'm
> confused. Is the reason that VM in sleep state and doesn't emit the
> signal. I set the windows 's "power option", enable/disable the
> "allow the wake timers", I didn't work.
> 
> Is anybody have met the same problem before, or know the reason. Your
> reply will be very helpful.

This post is missing a few pieces of information:

* What version of QEMU is this?

* What is the command line?

* How do you know g_poll is waiting for a signal and not for a file
descriptor?

* What is the backtrace of the main thread?  What is the backtrace of
the VCPU thread?

etc.

Paolo

[Qemu-devel] [RFH] Qemu main thread is blocked in g_poll in windows guest

[Xiexiangyou]

Hi all:

Windows2008 Guest run without pressure for long time. Sometimes, it stop and 
looks like hanging. But when I connect to it with VNC, It resume to run, but 
VM's time is delayed . 
When the vm is hanging, I check the main thread of QEMU. I find that the thread 
is blocked in g_poll function. it is waiting for a SIG, However, there is no 
SIG .

I tried the clock with "hpet" and "no hpet", but came out the same problem. 
Then I upgrade the glibc to newer, it didn't work too.
I'm confused. Is the reason that VM in sleep state and doesn't emit the signal. 
I set the windows 's "power option", enable/disable the "allow the wake 
timers", I didn't work.

Is anybody have met the same problem before, or know the reason. Your reply 
will be very helpful.

Thanks

--xiexiangyou

Re: [Qemu-devel] BUG: RTC issue when Windows guest is idle

[Xiexiangyou]

Hi:

 I have met the same bug that windows2008 guest stop receive the RTC 
ticks when it in idle status by fortuitous.

When vnc connect, guest will resume to receive RTC ticks and the time run fast 
because of the coalesced timer



HPET is diabled, and RTC is set "catchup", as following:

  





  



My kvm module is version3.6. Should I upgrade it to latest version. Any 
suggestion is welcome !



Thanks!



--xiangyouxie



On Thu, Feb 21, 2013 at 06:16:10PM +, Matthew Anderson wrote:

> If this isn't the correct list just let me know,

>

> I've run into a bug whereby a Windows guest (tested on Server 2008R2 and

> 2012) no longer receives RTC ticks when it has been idle for a random amount

> of time. HPET is disabled and the guest is running Hyper-V relaxed timers

> (same situation without hv_relaxed). The guest clock stands still and the

> qemu process uses very little CPU (<0.5%, normally it's >5% when the guest is

> idle) . Eventually the guest stops responding to network requests but if you

> open the guest console via VNC and move the mouse around it comes back to

> life and QEMU replays the lost RTC ticks and the guest recovers. I've also

> been able to make it recover by querying the clock over the network via the

> net time command, you can see the clock stand still for 30 seconds then it

> replays the ticks and catches up.

>

> I've tried to reproduce the issue but it seems fairly illusive, the only way

> I've been able to reproduce it is by letting the VM's idle and waiting.

> Sometimes it's hours and sometimes minutes. Can anyone suggest a way to

> narrow the issue down?

>

> Qemu command line is-

> /usr/bin/kvm -name SQL01 -S -M pc-0.14 -cpu qemu64,hv_relaxed -enable-kvm -m

> 2048 -smp 2,sockets=2,cores=1,threads=1 -uuid

> 5f54333b-c250-aa72-c979-39d156814b85 -no-user-config -nodefaults -chardev

> socket,id=charmonitor,path=/var/lib/libvirt/qemu/iHost-SQL01.monitor,server,nowait

>  -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime

> -no-hpet -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2

> -drive

> file=/mnt/gluster1-norep/iHost/SQL01.qed,if=none,id=drive-virtio-disk0,format=qed,cache=writeback

>  -device

> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0

>  -drive

> file=/mnt/gluster1-norep/iHost/SQL01-Data.qed,if=none,id=drive-virtio-disk2,format=qed,cache=writeback

>  -device

> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk2,id=virtio-disk2

>  -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device

> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev

> tap,fd=29,id=hostnet0,vhost=on,vhostfd=39 -device

> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:2c:8d:23,bus=pci.0,addr=0x3

>  -chardev pty,id=charserial0 -device

> isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc

> 127.0.0.1:22 -vga cirrus -device

> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4

>

> Environment is -

> Mainline 3.7.5 and 3.8.0

> Qemu 1.2.2, 1.3.1 and 1.4.0