Block live migration without pausing VM
Hi all, I want to implement live-migration of a highly available VM but I cannot use shared storage. The -b option to the migrate command already allows for copying the block device (locally stored raw file) and that is exactly what I want. This worked in my experiments but the VM is unreachable (CPU halted?) while the block device is copied, which is unacceptable for my use-case. Is there a way to copy the block device while the VM is running (copy on read or such)? The LiveBlockMigration page in the qemu wiki [1] mentions that some of this is already implemented, but I cannot find any of it in the latest qemu-kvm. Any pointers? Thanks for your suggestions. [1] http://wiki.qemu.org/Features/LiveBlockMigration -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: strange guest slowness after some time
Tomasz Chmielewski wrote: Felix Leimbach schrieb: BTW, what CPU do you have? One dual core Opteron 2212 Note: I will upgrade to two Shanghai Quad-Cores in 2 weeks and test with those as well. processor : 1 vendor_id : AuthenticAMD cpu family : 15 model : 65 model name : Dual-Core AMD Opteron(tm) Processor 2212 stepping: 2 cpu MHz : 1994.996 cache size : 1024 KB It's exactly the same CPU I have. Interesting: Since two months I'm running on 2 Shanghai Quad-Cores instead and the problem is definitely gone. The rest of the hardware as well as the whole software-stack remained unchanged. That should confirm what we assumed already. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: strange guest slowness after some time
Avi Kivity wrote: Since there are now at least two reports, maybe it won't be that difficult. If you can figure out a way to reliably reproduce this, that would be most helpful. I'll see what I can do. Although I'm not too optimistic because I have not experienced the problem after upgrading to kvm-84. But hey, that's a good thing. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: strange guest slowness after some time
Avi Kivity wrote: Felix Leimbach wrote: Is tsc breakage still s.th. to watch out after I've upgraded to the Shanghai quadcores? No, should be gone. Will you have the old server around so we can test things? No, I'll be upgrading the existing server. If you have specific tests in mind I can perform them in the next two weeks before the upgrade. But I cannot restart the server because a few VMs are in production use. If a developer is interested in the old CPU (Opteron 2212) then I can have it mailed to him/her. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: strange guest slowness after some time
Avi, Avi Kivity wrote: packet counters are will within 32-bit limits. byte counters not so interesting. This night my test overflowed the *packet* counters twice without any slowness or loss of connectivity. Snippet from my log file of the sending VM (linux 2.6.27): Wed Mar 18 05:14:18 CET 2009: TX packet counter = 4292944043 Wed Mar 18 05:15:18 CET 2009: TX packet counter = 6259211 ifconfig after stress test: # ifconfig eth0 Link encap:Ethernet HWaddr 52:54:00:74:01:01 inet addr:10.75.13.1 Bcast:10.75.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:150 Metric:1 RX packets:48950340 errors:0 dropped:0 overruns:0 frame:0 TX packets:727911367 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3686511201 (3.4 GiB) TX bytes:4207842269 (3.9 GiB) I didn't create a log file on the receiving Windows VM but they must have overflowed as well. Its packet counters are currently: Sent: 47.112.780 Received: 1.515.275.693 No problem on the Windows guest either. So the problem must lie elsewhere. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: strange guest slowness after some time
Tomasz Chmielewski wrote: Tomasz Chmielewski schrieb: Avi Kivity schrieb: packet counters are will within 32-bit limits. byte counters not so interesting. Ah OK. I did only byte overflow. Packet overflow will take much longer. It's one of these very rare cases where setting very small MTU is useful... OK, another bug found. Set your MTU to 100. On two hosts, do: HOST1_MTU1500# dd if=/dev/zero | ssh mana...@host2 dd of=/dev/null HOST2_MTU100# dd if=/dev/zero | ssh mana...@host1 dd of=/dev/null HOST2 with MTU 100 will crash after 10-15 minutes (with packet count still not overflown). Intersting. What are the packet counter at crash time (roughly)? My - currently running - test is: Guest 1 (Linux): MTU 150 # cat /dev/zero | nc Guest 2 (Windows 2003 Server): MTU: 1500 # nc -l -p > NUL My packet are currently at 63 million without a problem - yet. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: strange guest slowness after some time
Avi Kivity wrote: Felix Leimbach wrote: eth0 Link encap:Ethernet HWaddr 52:54:00:74:01:01 inet addr:10.75.13.1 Bcast:10.75.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3542104 errors:0 dropped:0 overruns:0 frame:0 TX packets:412546 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:682285568 (650.6 MiB) TX bytes:2907586796 (2.7 GiB) packet counters are will within 32-bit limits. byte counters not so interesting. ah right, I checked the byte counters only. Testing packet counter overflow now (takes a while). Do you experience the slowdown on Windows guests? both Linux and Windows Server 2003. All 32bit. But with me it is not a slowdown but a complete loss of network in the guest. Can't be pinged anymore. Although there might be a slowdown period before the that, I've heard hints in that direction from users. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: strange guest slowness after some time
Avi Kivity wrote: Does idle=poll help things? It can cause tsc breakage similar to cpufreq. On the host, right? Can't test that as I cannot reboot the server. Is tsc breakage still s.th. to watch out after I've upgraded to the Shanghai quadcores? -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: strange guest slowness after some time
Tomasz Chmielewski wrote: Felix Leimbach schrieb: I see similar behavior: After a week one of my guests' network totally stops to respond. Only guests using virtio networking get hit. Both windows and linux guests are affected. Also, does guest reboot help for you (for me, it doesn't)? Or, you have to halt the guest and start it again (i.e. stop kvm/qemu process and start a new one) to make the network working properly again? I have not tried rebooting; always stopped and restarted the qemu instance. Will try on the next occasion. Before I wrote that I tested on kvm-83 and 84 but it turns out the kvm-84 part was wrong: Since the upgrade 4 days ago I have not yet had a hang. I noticed that you Tomasz are also running kvm-83. Maybe kvm-84 fixed the issue already? -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: strange guest slowness after some time
Tomasz Chmielewski wrote: Avi Kivity schrieb: Might it be that some counter overflowed? What are the packet counts on long running guests? I don't think so. I just made both counters (TX, RX) of ifconfig for virtio interfaces overflow several times and everything is still as fast as it should be. I had overflows on the counters as well (32 bit guests) without an problem. Here is the current ifconfig output of a machine which suffered the problem before: eth0 Link encap:Ethernet HWaddr 52:54:00:74:01:01 inet addr:10.75.13.1 Bcast:10.75.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3542104 errors:0 dropped:0 overruns:0 frame:0 TX packets:412546 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:682285568 (650.6 MiB) TX bytes:2907586796 (2.7 GiB) (output of ifconfig, even on an unaffected e1000 guest, might help) currently I have e1000 only on windows guests. Is there a way to gather relevant statistics there too? -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: strange guest slowness after some time
Tomasz Chmielewski wrote: Felix Leimbach schrieb: Out of 3 e1000 guests none has ever been hit. Observed with kvm-83 and kvm-84 with the host running in-kernel KVM code (linux 2.6.25.7) Could you add a (unused) e1000 interface to your virtio guests? As this issue happens rarely for me, maybe you could help to reproduce it as well (i.e. if network gets slow on virtio interface, give e1000 a IP address, and try if network is also slow on e1000 on the very same guest). Will do and report BTW, what CPU do you have? One dual core Opteron 2212 Note: I will upgrade to two Shanghai Quad-Cores in 2 weeks and test with those as well. processor : 1 vendor_id : AuthenticAMD cpu family : 15 model : 65 model name : Dual-Core AMD Opteron(tm) Processor 2212 stepping: 2 cpu MHz : 1994.996 cache size : 1024 KB physical id : 0 siblings: 2 core id : 1 cpu cores : 2 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dno wext 3dnow rep_good nopl pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy bogomips: 3990.06 TLB size: 1024 4K pages clflush size: 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp tm stc -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: strange guest slowness after some time
Tomasz Chmielewski wrote: Avi Kivity schrieb: Tomasz Chmielewski wrote: After a week or so, network in one guest got slow with kvm-84 and no cpufreq. This is virtio, right? What about e1000? (I realize it takes a week to reproduce, but maybe you have some more experience) Yes, all affected had virtio. Probably because I didn't have many guests with e1000 interface. After a guest gets slow, I stop it and add another interface, e1000. If it gets slow again, I'll check if e1000 interface is slow as well. Will keep you updated. I see similar behavior: After a week one of my guests' network totally stops to respond. Only guests using virtio networking get hit. Both windows and linux guests are affected. My guests in production use e1000 and have never been hit. While that can be a coincidence it seems very unlikely: Out of 3 virtio guests 2 have been hit, one repeatedly. Out of 3 e1000 guests none has ever been hit. Observed with kvm-83 and kvm-84 with the host running in-kernel KVM code (linux 2.6.25.7) -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM-74 and network timeout?
Sean Mahrt wrote: I’ve noticed the guest with a lot of Disk I/O (commercial detection) after a while has a lot of NFS timeouts…. Virtio or E1000 give me the same result. I noticed exactly the same problem after moving from kvm-64 on a 2.6.25.3 host to kvm-74 on a 2.6.26.3 host. Adding to your observations: - CIFS shares are affected as well: Under heavy traffic I get timeouts from the server, see [1] - ne2k_pci and rtl8139 guests seem to be affected as well Now the real bad part, I’m getting pings in the order of ms, like 20-100ms on a bridged connection… and NFS is going crazy... My pings also increased from < 0.1ms to 16ms when the physical interface of the bridge was maxed out. Don't know, whether transferring from VM to VM would also trigger that. I’m using smp on the guests (and the host), and 2.6.25 on the guests… My guests where UP and mostly Windows Server 2003 and one Gentoo 2.6.26, so I think the culprit is elsewhere. Where should I start looking? Is this a KVM-74 issue? Bump to KVM-75? You might try kvm-64 which is rock-solid for me when paired with a 2.6.25 KVM kernel module. [1] Sep 6 17:42:58 [EMAIL PROTECTED] CIFS VFS: server not responding Sep 6 17:42:58 [EMAIL PROTECTED] CIFS VFS: No response to cmd 46 mid 30836 Sep 6 17:42:58 [EMAIL PROTECTED] CIFS VFS: Send error in read = -11 Sep 6 17:51:28 [EMAIL PROTECTED] CIFS VFS: server not responding Sep 6 17:51:28 [EMAIL PROTECTED] CIFS VFS: No response for cmd 50 mid 30850 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
How to use current KVM with non-modular kernel
Hello list, due to security concerns I have kvm hosts which have non-modular kernels, i.e. CONFIG_MODULES is not set. I'd like to use the latest KVM kernel code with those, but cannot not always upgrade to the latest kernel.org kernel. Is it possible to update kernel sources to include the latest and greatest kvm? For example to have 2.6.25.16 kernel with the kvm-74 module *compiled in*. -Felix -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Sharing disks between two kvm guests
This is my first post to this list. I have already installed kvm-70 under rhel5.2. My intention is to share on disk image betwwen two rhel5.2 kvm guests. Is it possible to accomplish this in kvm like xen or vmware does?? How can I do?? I didn't find any reference abou this on kvm documentation ... Have a look at KVM/QEMU's -smb option in http://bellard.org/qemu/qemu-doc.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
