[Qemu-devel] livepatch kernel to use md-clear without reboot
Hello, i'm interested if it is possible to livepatch the guest kernel without reboot. (host kernel and microcode is patched) Greets, Stefan
Re: [Qemu-devel] cpu.fail / MDS fixes
Am 15.05.19 um 19:54 schrieb Daniel P. Berrangé: > On Wed, May 15, 2019 at 07:13:56PM +0200, Stefan Priebe - Profihost AG wrote: >> Hello list, >> >> i've updated my host to kernel 4.19.43 and applied the following patch >> to my qemu 2.12.1: >> https://bugzilla.suse.com/attachment.cgi?id=798722 >> >> But my guest running 4.19.43 still says: >> Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state >> unknown >> >> while the host says: >> Vulnerable: Clear CPU buffers attempted, SMT Host state unknown > > That suggests your host OS hasn't got the new microcode installed > or has not loaded it. No it does not. A not loaded Microcode looks like this: Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable but in my case it is: Mitigation: Clear CPU buffers; SMT vulnerable on the host as hyper threading is still enabled. > You want the host to report that it is Mitigated, and for the > host's /proc/cpuinfo to report "md-clear" exists. > >> I expected the guest can use the new microcode. > > You've not said what CPU model you've given to the guest. > > You need either "-cpu host", or if using a named CPU model > you need to explicitly turn on the "md-clear" feature > (and all previous fixes) > >eg "-cpu Haswell,+spec-ctrl,+ssbd,+md-clear" hah yes you're true i need to specifiy +md-clean Thanks! > Regards, > Daniel >
[Qemu-devel] cpu.fail / MDS fixes
Hello list, i've updated my host to kernel 4.19.43 and applied the following patch to my qemu 2.12.1: https://bugzilla.suse.com/attachment.cgi?id=798722 But my guest running 4.19.43 still says: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown while the host says: Vulnerable: Clear CPU buffers attempted, SMT Host state unknown I expected the guest can use the new microcode. Greets, Stefan
Re: [Qemu-devel] Overcommiting cpu results in all vms offline
Am 17.09.2018 um 11:40 schrieb Jack Wang: > Stefan Priebe - Profihost AG 于2018年9月17日周一 上午9:00写道: >> >> Hi, >> >> Am 17.09.2018 um 08:38 schrieb Jack Wang: >>> Stefan Priebe - Profihost AG 于2018年9月16日周日 下午3:31写道: >>>> >>>> Hello, >>>> >>>> while overcommiting cpu I had several situations where all vms gone >>>> offline while two vms saturated all cores. >>>> >>>> I believed all vms would stay online but would just not be able to use all >>>> their cores? >>>> >>>> My original idea was to automate live migration on high host load to move >>>> vms to another node but that makes only sense if all vms stay online. >>>> >>>> Is this expected? Anything special needed to archive this? >>>> >>>> Greets, >>>> Stefan >>>> >>> Hi, Stefan, >>> >>> Do you have any logs when all VMs go offline? >>> Maybe OOMkiller play a role there? >> >> After reviewing i think this is memory related but OOM did not play a role. >> All kvm processes where spinning trying to get > 100% CPU and i was not >> able to even login to ssh. After 5-10 minutes i was able to login. > So the VMs are not really offline, what the result if you run > query-status via qmp? I can't as i can't connect to the host in that stage. >> There were about 150GB free mem. >> >> Relevant settings (no local storage involved): >> vm.dirty_background_ratio: >> 3 >> vm.dirty_ratio: >> 10 >> vm.min_free_kbytes: >> 10567004 >> >> # cat /sys/kernel/mm/transparent_hugepage/defrag >> always defer [defer+madvise] madvise never >> >> # cat /sys/kernel/mm/transparent_hugepage/enabled >> [always] madvise never >> >> After that i had the following traces on the host node: >> https://pastebin.com/raw/0VhyQmAv > > The call trace looks ceph related deadlock or hung. Yes but i can also show you traces where nothing from ceph is involved the only thing they have in common is the beginning in page_fault. >> Thanks! >> >> Greets, >> Stefan
Re: [Qemu-devel] Overcommiting cpu results in all vms offline
May be amissing piece: vm.overcommit_memory=0 Greets, Stefan Am 17.09.2018 um 09:00 schrieb Stefan Priebe - Profihost AG: > Hi, > > Am 17.09.2018 um 08:38 schrieb Jack Wang: >> Stefan Priebe - Profihost AG 于2018年9月16日周日 下午3:31写道: >>> >>> Hello, >>> >>> while overcommiting cpu I had several situations where all vms gone offline >>> while two vms saturated all cores. >>> >>> I believed all vms would stay online but would just not be able to use all >>> their cores? >>> >>> My original idea was to automate live migration on high host load to move >>> vms to another node but that makes only sense if all vms stay online. >>> >>> Is this expected? Anything special needed to archive this? >>> >>> Greets, >>> Stefan >>> >> Hi, Stefan, >> >> Do you have any logs when all VMs go offline? >> Maybe OOMkiller play a role there? > > After reviewing i think this is memory related but OOM did not play a role. > All kvm processes where spinning trying to get > 100% CPU and i was not > able to even login to ssh. After 5-10 minutes i was able to login. > > There were about 150GB free mem. > > Relevant settings (no local storage involved): > vm.dirty_background_ratio: > 3 > vm.dirty_ratio: > 10 > vm.min_free_kbytes: > 10567004 > > # cat /sys/kernel/mm/transparent_hugepage/defrag > always defer [defer+madvise] madvise never > > # cat /sys/kernel/mm/transparent_hugepage/enabled > [always] madvise never > > After that i had the following traces on the host node: > https://pastebin.com/raw/0VhyQmAv > > Thanks! > > Greets, > Stefan >
Re: [Qemu-devel] Overcommiting cpu results in all vms offline
Hi, Am 17.09.2018 um 08:38 schrieb Jack Wang: > Stefan Priebe - Profihost AG 于2018年9月16日周日 下午3:31写道: >> >> Hello, >> >> while overcommiting cpu I had several situations where all vms gone offline >> while two vms saturated all cores. >> >> I believed all vms would stay online but would just not be able to use all >> their cores? >> >> My original idea was to automate live migration on high host load to move >> vms to another node but that makes only sense if all vms stay online. >> >> Is this expected? Anything special needed to archive this? >> >> Greets, >> Stefan >> > Hi, Stefan, > > Do you have any logs when all VMs go offline? > Maybe OOMkiller play a role there? After reviewing i think this is memory related but OOM did not play a role. All kvm processes where spinning trying to get > 100% CPU and i was not able to even login to ssh. After 5-10 minutes i was able to login. There were about 150GB free mem. Relevant settings (no local storage involved): vm.dirty_background_ratio: 3 vm.dirty_ratio: 10 vm.min_free_kbytes: 10567004 # cat /sys/kernel/mm/transparent_hugepage/defrag always defer [defer+madvise] madvise never # cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise never After that i had the following traces on the host node: https://pastebin.com/raw/0VhyQmAv Thanks! Greets, Stefan
[Qemu-devel] Overcommiting cpu results in all vms offline
Hello, while overcommiting cpu I had several situations where all vms gone offline while two vms saturated all cores. I believed all vms would stay online but would just not be able to use all their cores? My original idea was to automate live migration on high host load to move vms to another node but that makes only sense if all vms stay online. Is this expected? Anything special needed to archive this? Greets, Stefan
Re: [Qemu-devel] Qemu and Spectre_V4 + l1tf + IBRS_FW
Am 17.08.2018 um 11:41 schrieb Daniel P. Berrangé: > On Fri, Aug 17, 2018 at 08:44:38AM +0200, Stefan Priebe - Profihost AG wrote: >> Hello, >> >> i haven't found anything on the web regarding qemu and mentioned variants. >> >> While my host says: >> l1tf:Mitigation: PTE Inversion; VMX: SMT vulnerable, L1D conditional >> cache flushes >> meltdown:Mitigation: PTI >> spec_store_bypass:Mitigation: Speculative Store Bypass disabled via >> prctl and seccomp >> spectre_v1:Mitigation: __user pointer sanitization >> spectre_v2:Mitigation: Full generic retpoline, IBPB, IBRS_FW >> >> My guests bootet with pcid and spec-ctrl only say: >> l1tf:Mitigation: PTE Inversion >> meltdown:Mitigation: PTI >> spec_store_bypass:Vulnerable >> spectre_v1:Mitigation: __user pointer sanitization >> spectre_v2:Mitigation: Full generic retpoline, IBPB >> >> * What is about spec_store_bypass in Qemu? > > The guest needs an 'ssbd' feature for Intel CPU models and either a > 'virt-ssbd' or 'amd-ssbd' feature for AMD CPU models. Ah thanks. That works fine. >> * What is about IBRS_FW feature? > > I'm not sure what IBRS_FW is referring to, but don't worry about it. > The fact the the guest kernel says "Mitigation" instead of "Vulnerable" > means you are protected with your current config. > > For Intel CPU models Spectre v2 needs the guest to have the 'spec-ctrl' > feature. On AMD models Spectre v2 the guest needs 'ibpb' feature. > >> * What is about L1TF? > > No extra CPU flags are required for QEMU guests for L1TF. The new CPU > feature is merely an perf optimization for the host hypervisor fixes. > > Note that with L1TF there are extra steps you need to consider wrt > hyperthreading, that won't be reflected in the 'vulnerabilities' > data published by the kernel. > > You can read more about the procedure for dealing with L1TF in > virt hosts in the "Resolve" tab of this article: > > https://access.redhat.com/security/vulnerabilities/L1TF > >> Or are those just irrelevant to Qemu guests? Would be great to have some >> informations. > > We have some QEMU docs providing guidance on guest CPU model/feature config > but they are not yet published. In the meantime this blog post of mine gives > the same info, covering what's needed for Spectre v2, Meltdown and SSBD and > guidance in general for CPU config: > > > https://www.berrange.com/posts/2018/06/29/cpu-model-configuration-for-qemu-kvm-on-x86-hosts/ Thanks, that one was really helpful. Greets, Stefan
[Qemu-devel] Qemu and Spectre_V4 + l1tf + IBRS_FW
Hello, i haven't found anything on the web regarding qemu and mentioned variants. While my host says: l1tf:Mitigation: PTE Inversion; VMX: SMT vulnerable, L1D conditional cache flushes meltdown:Mitigation: PTI spec_store_bypass:Mitigation: Speculative Store Bypass disabled via prctl and seccomp spectre_v1:Mitigation: __user pointer sanitization spectre_v2:Mitigation: Full generic retpoline, IBPB, IBRS_FW My guests bootet with pcid and spec-ctrl only say: l1tf:Mitigation: PTE Inversion meltdown:Mitigation: PTI spec_store_bypass:Vulnerable spectre_v1:Mitigation: __user pointer sanitization spectre_v2:Mitigation: Full generic retpoline, IBPB * What is about spec_store_bypass in Qemu? * What is about IBRS_FW feature? * What is about L1TF? Or are those just irrelevant to Qemu guests? Would be great to have some informations. Thanks a lot! Greets, Stefan
Re: [Qemu-devel] how to pass pcid to guest?
Am 08.01.2018 um 23:07 schrieb Eric Blake: > On 01/08/2018 02:03 PM, Stefan Priebe - Profihost AG wrote: >> Hello, >> >> for meltdown mitigation and performance it's important to have the pcid >> flag passed down to the guest (f.e. >> https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU). > > Indeed; you are still waiting on the qemu patch mentioned here: > https://www.qemu.org/2018/01/04/spectre/ > > which is still undergoing the review process, but should be up (in the > form of 2.11.1) "in the next few days". OK thanks as the performance difference is significant: no pcid: # time for i in $(seq 1 1 50); do du -sx /; done ... real0m26.614s user0m17.548s sys 0m9.056s kvm started with +pcid: # time for i in $(seq 1 1 50); do du -sx /; done ... real0m14.734s user0m7.755s sys 0m6.973s Greets, Stefan
[Qemu-devel] how to pass pcid to guest?
Hello, for meltdown mitigation and performance it's important to have the pcid flag passed down to the guest (f.e. https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU). My host shows the flag: # grep ' pcid ' /proc/cpuinfo | wc -l 56 But the guest does not: # grep pcid /proc/cpuinfo # Guest was started with: -cpu IvyBridge,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,vendor=GenuineIntel Qemu is 2.9.1 Thanks! Greets, Stefan
[Qemu-devel] What's needed to pass pciid cpu flag to guest?
Hello, for meltdown mitigation and performance it's important to have the pcid flag passed down to the guest (f.e. https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU). My host shows the flag: # grep ' pcid ' /proc/cpuinfo | wc -l 56 But the guest does not: # grep pcid /proc/cpuinfo # Guest was started with: -cpu IvyBridge,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,vendor=GenuineIntel Qemu is 2.9.1 Thanks! Greets, Stefan
Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches
Thanks! But it‘s very difficult to get all opinions all together. SuSE Enterprise told me to update: - kernel - qemu - Intel microcode And the released already updates for all of them. Stefan Excuse my typo sent from my mobile phone. > Am 05.01.2018 um 09:33 schrieb Paolo Bonzini : > >> On 04/01/2018 21:15, Stefan Priebe - Profihost AG wrote: >> attached the relevant patch for everybody who needs it. > > This is the original patch from Intel, which doesn't work unless you > have a patched kernel (which you almost certainly don't have) and > doesn't even warn you about that. > > In other words, it's rubbish. Please read > https://www.qemu.org/2018/01/04/spectre/ several times, until you > understand why there is no urgent need to update QEMU. > > Days are 24 hours for QEMU developers just like for you (and believe me, > we wished several times that they weren't during the last two months). > We are prioritizing the fixes according to their effect in mitigating > the vulnerability, their applicability and the availability of patches > to the lower levels of the stack. Right now, the most urgent part is > the simple mitigations that can go in Linux 4.15 and stable kernels. > > Paolo
Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches
attached the relevant patch for everybody who needs it. Greets, Stefan Am 04.01.2018 um 16:53 schrieb Paolo Bonzini: > On 04/01/2018 09:35, Alexandre DERUMIER wrote: >>>> So you need: >>>> 1.) intel / amd cpu microcode update >>>> 2.) qemu update to pass the new MSR and CPU flags from the microcode >>>> update >>>> 3.) host kernel update >>>> 4.) guest kernel update >> >> are you sure we need to patch guest kernel if we are able to patch qemu ? > > Patching the guest kernel is only required to protect the guest kernel > from guest usermode. > >> If I understand, patching the host kernel, should avoid that a vm is reading >> memory of another vm. >> (the most critical) > > Correct. > >> patching the guest kernel, to avoid that a process from the vm have access >> to memory of another process of same vm. > > Correct. > > The QEMU updates are pretty boring, mostly taking care of new MSR and > CPUID flags (and adding new CPU models). > > They are not needed to protect the guest from "Meltdown", only > "Spectre"---the former only needs a guest kernel update. Also, to have > any effect, the guest kernels must also have "Spectre" patches which > aren't upstream yet for either KVM or the rest of Linux. So the QEMU > patches are much less important than the kernel side. > >>> https://access.redhat.com/solutions/3307851 >>> "Impacts of CVE-2017-5754, CVE-2017-5753, and CVE-2017-5715 to Red Hat >>> Virtualization products" > > It mostly repeats the contents of the RHEL document > https://access.redhat.com/security/vulnerabilities/speculativeexecution, > with some information specific to RHV. > > Thanks, > > Paolo > >> i don't have one but the content might be something like this: >> https://www.suse.com/de-de/support/kb/doc/?id=7022512 >> >> So you need: >> 1.) intel / amd cpu microcode update >> 2.) qemu update to pass the new MSR and CPU flags from the microcode update >> 3.) host kernel update >> 4.) guest kernel update >> >> The microcode update and the kernel update is publicly available but i'm >> missing the qemu one. >> >> Greets, >> Stefan >> >>> - Mail original - >>> De: "aderumier" >>> À: "Stefan Priebe, Profihost AG" >>> Cc: "qemu-devel" >>> Envoyé: Jeudi 4 Janvier 2018 08:24:34 >>> Objet: Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches >>> >>>>> Can anybody point me to the relevant qemu patches? >>> >>> I don't have find them yet. >>> >>> Do you known if a vm using kvm64 cpu model is protected or not ? >>> >>> - Mail original - >>> De: "Stefan Priebe, Profihost AG" >>> À: "qemu-devel" >>> Envoyé: Jeudi 4 Janvier 2018 07:27:01 >>> Objet: [Qemu-devel] CVE-2017-5715: relevant qemu patches >>> >>> Hello, >>> >>> i've seen some vendors have updated qemu regarding meltdown / spectre. >>> >>> f.e.: >>> >>> CVE-2017-5715: QEMU was updated to allow passing through new MSR and >>> CPUID flags from the host VM to the CPU, to allow enabling/disabling >>> branch prediction features in the Intel CPU. (bsc#1068032) >>> >>> Can anybody point me to the relevant qemu patches? >>> >>> Thanks! >>> >>> Greets, >>> Stefan >>> >> >> > >From b4fdfeb4545c09a0fdf01edc938f9cce8fcaa5c6 Mon Sep 17 00:00:00 2001 From: Wei Wang Date: Tue, 7 Nov 2017 16:39:49 +0800 Subject: [PATCH] i386/kvm: MSR_IA32_SPEC_CTRL and MSR_IA32_PRED_CMD CPUID(EAX=0X7,ECX=0).EDX[26]/[27] indicates the support of MSR_IA32_SPEC_CTRL and MSR_IA32_PRED_CMD. Expose the CPUID to the guest. Also add the support of transferring the MSRs during live migration. Signed-off-by: Wei Wang [BR: BSC#1068032 CVE-2017-5715] Signed-off-by: Bruce Rogers --- target/i386/cpu.c | 3 ++- target/i386/cpu.h | 4 target/i386/kvm.c | 15 ++- target/i386/machine.c | 20 4 files changed, 40 insertions(+), 2 deletions(-) diff --git a/target/i386/cpu.c b/target/i386/cpu.c index 55f72b679f..01761db3fc 100644 --- a/target/i386/cpu.c +++ b/target/i386/cpu.c @@ -2823,13 +2823,14 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, case 7: /* Structured Extended Feature Flags Enumeration Leaf */ if (count == 0) { +host_c
Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches
Nobody? Is this something they did on their own? Stefan Am 04.01.2018 um 07:27 schrieb Stefan Priebe - Profihost AG: > Hello, > > i've seen some vendors have updated qemu regarding meltdown / spectre. > > f.e.: > > CVE-2017-5715: QEMU was updated to allow passing through new MSR and > CPUID flags from the host VM to the CPU, to allow enabling/disabling > branch prediction features in the Intel CPU. (bsc#1068032) > > Can anybody point me to the relevant qemu patches? > > Thanks! > > Greets, > Stefan >
Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches
Am 04.01.2018 um 09:35 schrieb Alexandre DERUMIER: >>> So you need: >>> 1.) intel / amd cpu microcode update >>> 2.) qemu update to pass the new MSR and CPU flags from the microcode update >>> 3.) host kernel update >>> 4.) guest kernel update > > are you sure we need to patch guest kernel if we are able to patch qemu ? >> I have some pretty old guest (linux and windows) > > If I understand, patching the host kernel, should avoid that a vm is reading > memory of another vm. > (the most critical) Yes - this was just to complete the mitigation on all layers. > > patching the guest kernel, to avoid that a process from the vm have access to > memory of another process of same vm. Yes. Stefan > > > > - Mail original - > De: "Stefan Priebe, Profihost AG" > À: "aderumier" > Cc: "qemu-devel" > Envoyé: Jeudi 4 Janvier 2018 09:17:41 > Objet: Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches > > Am 04.01.2018 um 08:27 schrieb Alexandre DERUMIER: >> does somebody have a redhat account to see te content of: >> >> https://access.redhat.com/solutions/3307851 >> "Impacts of CVE-2017-5754, CVE-2017-5753, and CVE-2017-5715 to Red Hat >> Virtualization products" > > i don't have one but the content might be something like this: > https://www.suse.com/de-de/support/kb/doc/?id=7022512 > > So you need: > 1.) intel / amd cpu microcode update > 2.) qemu update to pass the new MSR and CPU flags from the microcode update > 3.) host kernel update > 4.) guest kernel update > > The microcode update and the kernel update is publicly available but i'm > missing the qemu one. > > Greets, > Stefan > >> - Mail original - >> De: "aderumier" >> À: "Stefan Priebe, Profihost AG" >> Cc: "qemu-devel" >> Envoyé: Jeudi 4 Janvier 2018 08:24:34 >> Objet: Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches >> >>>> Can anybody point me to the relevant qemu patches? >> >> I don't have find them yet. >> >> Do you known if a vm using kvm64 cpu model is protected or not ? >> >> - Mail original - >> De: "Stefan Priebe, Profihost AG" >> À: "qemu-devel" >> Envoyé: Jeudi 4 Janvier 2018 07:27:01 >> Objet: [Qemu-devel] CVE-2017-5715: relevant qemu patches >> >> Hello, >> >> i've seen some vendors have updated qemu regarding meltdown / spectre. >> >> f.e.: >> >> CVE-2017-5715: QEMU was updated to allow passing through new MSR and >> CPUID flags from the host VM to the CPU, to allow enabling/disabling >> branch prediction features in the Intel CPU. (bsc#1068032) >> >> Can anybody point me to the relevant qemu patches? >> >> Thanks! >> >> Greets, >> Stefan >> >
Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches
Am 04.01.2018 um 08:27 schrieb Alexandre DERUMIER: > does somebody have a redhat account to see te content of: > > https://access.redhat.com/solutions/3307851 > "Impacts of CVE-2017-5754, CVE-2017-5753, and CVE-2017-5715 to Red Hat > Virtualization products" i don't have one but the content might be something like this: https://www.suse.com/de-de/support/kb/doc/?id=7022512 So you need: 1.) intel / amd cpu microcode update 2.) qemu update to pass the new MSR and CPU flags from the microcode update 3.) host kernel update 4.) guest kernel update The microcode update and the kernel update is publicly available but i'm missing the qemu one. Greets, Stefan > - Mail original - > De: "aderumier" > À: "Stefan Priebe, Profihost AG" > Cc: "qemu-devel" > Envoyé: Jeudi 4 Janvier 2018 08:24:34 > Objet: Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches > >>> Can anybody point me to the relevant qemu patches? > > I don't have find them yet. > > Do you known if a vm using kvm64 cpu model is protected or not ? > > - Mail original - > De: "Stefan Priebe, Profihost AG" > À: "qemu-devel" > Envoyé: Jeudi 4 Janvier 2018 07:27:01 > Objet: [Qemu-devel] CVE-2017-5715: relevant qemu patches > > Hello, > > i've seen some vendors have updated qemu regarding meltdown / spectre. > > f.e.: > > CVE-2017-5715: QEMU was updated to allow passing through new MSR and > CPUID flags from the host VM to the CPU, to allow enabling/disabling > branch prediction features in the Intel CPU. (bsc#1068032) > > Can anybody point me to the relevant qemu patches? > > Thanks! > > Greets, > Stefan >
[Qemu-devel] CVE-2017-5715: relevant qemu patches
Hello, i've seen some vendors have updated qemu regarding meltdown / spectre. f.e.: CVE-2017-5715: QEMU was updated to allow passing through new MSR and CPUID flags from the host VM to the CPU, to allow enabling/disabling branch prediction features in the Intel CPU. (bsc#1068032) Can anybody point me to the relevant qemu patches? Thanks! Greets, Stefan
Re: [Qemu-devel] dropped pkts with Qemu on tap interace (RX)
Am 03.01.2018 um 09:14 schrieb Alexandre DERUMIER: > Hi Stefan, > >>> The tap devices on the target vm shows dropped RX packages on BOTH tap >>> interfaces - strangely with the same amount of pkts? > > that's strange indeed. > if you tcpdump tap interfaces, do you see incoming traffic only on 1 > interface, or both random ? complete independend random traffic as it should. > (can you provide the network configuration in the guest for both interfaces ?) inside the guest? where the drop counter stays 0? auto eth0 iface eth0 inet dhcp auto eth1 iface eth1 inet static address 192.168.0.2 netmask 255.255.255.0 that's it. > I'm seeing that you have enable multiqueue on 1 of the interfaces, do you > have setup correctly the multiqueue part inside the guest. uh oh? What is needed inside the guest? > do you have enough vcpu to handle all the queues ? Yes. Stefan > - Mail original - > De: "Stefan Priebe, Profihost AG" > À: "qemu-devel" > Envoyé: Mardi 2 Janvier 2018 12:17:29 > Objet: [Qemu-devel] dropped pkts with Qemu on tap interace (RX) > > Hello, > > currently i'm trying to fix a problem where we have "random" missing > packets. > > We're doing an ssh connect from machine a to machine b every 5 minutes > via rsync and ssh. > > Sometimes it happens that we get this cron message: > "Connection to 192.168.0.2 closed by remote host. > rsync: connection unexpectedly closed (0 bytes received so far) [sender] > rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] > ssh: connect to host 192.168.0.2 port 22: Connection refused" > > The tap devices on the target vm shows dropped RX packages on BOTH tap > interfaces - strangely with the same amount of pkts? > > # ifconfig tap317i0; ifconfig tap317i1 > tap317i0 Link encap:Ethernet HWaddr 6e:cb:65:94:bb:bf > UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > RX packets:2238445 errors:0 dropped:13159 overruns:0 frame:0 > TX packets:9655853 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:177991267 (169.7 MiB) TX bytes:910412749 (868.2 MiB) > > tap317i1 Link encap:Ethernet HWaddr 96:f8:b5:d0:9a:07 > UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > RX packets:1516085 errors:0 dropped:13159 overruns:0 frame:0 > TX packets:1446964 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:1597564313 (1.4 GiB) TX bytes:3517734365 (3.2 GiB) > > Any ideas how to inspect this issue? > > Greets, > Stefan >
Re: [Qemu-devel] dropped pkts with Qemu on tap interace (RX)
Am 03.01.2018 um 04:57 schrieb Wei Xu: > On Tue, Jan 02, 2018 at 10:17:25PM +0100, Stefan Priebe - Profihost AG wrote: >> >> Am 02.01.2018 um 18:04 schrieb Wei Xu: >>> On Tue, Jan 02, 2018 at 04:24:33PM +0100, Stefan Priebe - Profihost AG >>> wrote: >>>> Hi, >>>> Am 02.01.2018 um 15:20 schrieb Wei Xu: >>>>> On Tue, Jan 02, 2018 at 12:17:29PM +0100, Stefan Priebe - Profihost AG >>>>> wrote: >>>>>> Hello, >>>>>> >>>>>> currently i'm trying to fix a problem where we have "random" missing >>>>>> packets. >>>>>> >>>>>> We're doing an ssh connect from machine a to machine b every 5 minutes >>>>>> via rsync and ssh. >>>>>> >>>>>> Sometimes it happens that we get this cron message: >>>>>> "Connection to 192.168.0.2 closed by remote host. >>>>>> rsync: connection unexpectedly closed (0 bytes received so far) [sender] >>>>>> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] >>>>>> ssh: connect to host 192.168.0.2 port 22: Connection refused" >>>>> >>>>> Hi Stefan, >>>>> What kind of virtio-net backend are you using? Can you paste your qemu >>>>> command line here? >>>> >>>> Sure netdev part: >>>> -netdev >>>> type=tap,id=net0,ifname=tap317i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on >>>> -device >>>> virtio-net-pci,mac=EA:37:42:5C:F3:33,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 >>>> -netdev >>>> type=tap,id=net1,ifname=tap317i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on,queues=4 >>>> -device >>>> virtio-net-pci,mac=6A:8E:74:45:1A:0B,nedev=net1,bus=pci.0,addr=0x13,id=net1,vectors=10,mq=on,bootindex=301 >>> >>> According to what you have mentioned, the traffic is not heavy for the >>> guests, >>> the dropping shouldn't happen for regular case. >> >> The avg traffic is around 300kb/s. >> >>> What is your hardware platform? >> >> Dual Intel Xeon E5-2680 v4 >> >>> and Which versions are you using for both >>> guest/host kernel >> Kernel v4.4.103 >> >>> and qemu? >> 2.9.1 >> >>> Are there other VMs on the same host? >> Yes. > > What about the CPU load? Host: 80-90% Idle LoadAvg: 6-7 VM: 97%-99% Idle >>>>> 'Connection refused' usually means that the client gets a TCP Reset rather >>>>> than losing packets, so this might not be a relevant issue. >>>> >>>> Mhm so you mean these might be two seperate ones? >>> >>> Yes. >>> >>>> >>>>> Also you can do a tcpdump on both guests and see what happened to SSH >>>>> packets >>>>> (tcpdump -i tapXXX port 22). >>>> >>>> Sadly not as there's too much traffic on that part as rsync is syncing >>>> every 5 minutes through ssh. >>> >>> You can do a tcpdump for the entire traffic from the guest and host and >>> compare >>> what kind of packets are dropped if the traffic is not overloaded. >> >> Are you sure? I don't get why the same amount and same kind of packets >> should be received by both tap which are connected to different bridges >> to different HW and physical interfaces. > > Exactly, possibly this would be a host or guest kernel bug cos than qemu issue > you are using vhost kernel as the backend and the two stats are independent, > you might have to check out what is happening inside the traffic. What do you mean by inside the traffic? Stefan
Re: [Qemu-devel] dropped pkts with Qemu on tap interace (RX)
Am 02.01.2018 um 18:04 schrieb Wei Xu: > On Tue, Jan 02, 2018 at 04:24:33PM +0100, Stefan Priebe - Profihost AG wrote: >> Hi, >> Am 02.01.2018 um 15:20 schrieb Wei Xu: >>> On Tue, Jan 02, 2018 at 12:17:29PM +0100, Stefan Priebe - Profihost AG >>> wrote: >>>> Hello, >>>> >>>> currently i'm trying to fix a problem where we have "random" missing >>>> packets. >>>> >>>> We're doing an ssh connect from machine a to machine b every 5 minutes >>>> via rsync and ssh. >>>> >>>> Sometimes it happens that we get this cron message: >>>> "Connection to 192.168.0.2 closed by remote host. >>>> rsync: connection unexpectedly closed (0 bytes received so far) [sender] >>>> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] >>>> ssh: connect to host 192.168.0.2 port 22: Connection refused" >>> >>> Hi Stefan, >>> What kind of virtio-net backend are you using? Can you paste your qemu >>> command line here? >> >> Sure netdev part: >> -netdev >> type=tap,id=net0,ifname=tap317i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on >> -device >> virtio-net-pci,mac=EA:37:42:5C:F3:33,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 >> -netdev >> type=tap,id=net1,ifname=tap317i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on,queues=4 >> -device >> virtio-net-pci,mac=6A:8E:74:45:1A:0B,nedev=net1,bus=pci.0,addr=0x13,id=net1,vectors=10,mq=on,bootindex=301 > > According to what you have mentioned, the traffic is not heavy for the guests, > the dropping shouldn't happen for regular case. The avg traffic is around 300kb/s. > What is your hardware platform? Dual Intel Xeon E5-2680 v4 > and Which versions are you using for both > guest/host kernel Kernel v4.4.103 > and qemu? 2.9.1 > Are there other VMs on the same host? Yes. >>> 'Connection refused' usually means that the client gets a TCP Reset rather >>> than losing packets, so this might not be a relevant issue. >> >> Mhm so you mean these might be two seperate ones? > > Yes. > >> >>> Also you can do a tcpdump on both guests and see what happened to SSH >>> packets >>> (tcpdump -i tapXXX port 22). >> >> Sadly not as there's too much traffic on that part as rsync is syncing >> every 5 minutes through ssh. > > You can do a tcpdump for the entire traffic from the guest and host and > compare > what kind of packets are dropped if the traffic is not overloaded. Are you sure? I don't get why the same amount and same kind of packets should be received by both tap which are connected to different bridges to different HW and physical interfaces. Stefan > Wei > >> >>>> The tap devices on the target vm shows dropped RX packages on BOTH tap >>>> interfaces - strangely with the same amount of pkts? >>>> >>>> # ifconfig tap317i0; ifconfig tap317i1 >>>> tap317i0 Link encap:Ethernet HWaddr 6e:cb:65:94:bb:bf >>>> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >>>> RX packets:2238445 errors:0 dropped:13159 overruns:0 frame:0 >>>> TX packets:9655853 errors:0 dropped:0 overruns:0 carrier:0 >>>> collisions:0 txqueuelen:1000 >>>> RX bytes:177991267 (169.7 MiB) TX bytes:910412749 (868.2 MiB) >>>> >>>> tap317i1 Link encap:Ethernet HWaddr 96:f8:b5:d0:9a:07 >>>> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >>>> RX packets:1516085 errors:0 dropped:13159 overruns:0 frame:0 >>>> TX packets:1446964 errors:0 dropped:0 overruns:0 carrier:0 >>>> collisions:0 txqueuelen:1000 >>>> RX bytes:1597564313 (1.4 GiB) TX bytes:3517734365 (3.2 GiB) >>>> >>>> Any ideas how to inspect this issue? >>> >>> It seems both tap interfaces lose RX pkts, dropping pkts of RX means the >>> host(backend) cann't receive packets from the guest as fast as the guest >>> sends. >> >> Inside the guest i see no dropped packets at all. It's only on the host >> and strangely on both taps at the same value? And both are connected to >> absolutely different networks. >> >>> Are you running some symmetrical test on both guests? >> >> No. >> >> Stefan >> >> >>> Wei >>> >>>> >>>> Greets, >>>> Stefan >>>>
Re: [Qemu-devel] dropped pkts with Qemu on tap interace (RX)
Hi, Am 02.01.2018 um 15:20 schrieb Wei Xu: > On Tue, Jan 02, 2018 at 12:17:29PM +0100, Stefan Priebe - Profihost AG wrote: >> Hello, >> >> currently i'm trying to fix a problem where we have "random" missing >> packets. >> >> We're doing an ssh connect from machine a to machine b every 5 minutes >> via rsync and ssh. >> >> Sometimes it happens that we get this cron message: >> "Connection to 192.168.0.2 closed by remote host. >> rsync: connection unexpectedly closed (0 bytes received so far) [sender] >> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] >> ssh: connect to host 192.168.0.2 port 22: Connection refused" > > Hi Stefan, > What kind of virtio-net backend are you using? Can you paste your qemu > command line here? Sure netdev part: -netdev type=tap,id=net0,ifname=tap317i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=EA:37:42:5C:F3:33,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -netdev type=tap,id=net1,ifname=tap317i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on,queues=4 -device virtio-net-pci,mac=6A:8E:74:45:1A:0B,nedev=net1,bus=pci.0,addr=0x13,id=net1,vectors=10,mq=on,bootindex=301 > 'Connection refused' usually means that the client gets a TCP Reset rather > than losing packets, so this might not be a relevant issue. Mhm so you mean these might be two seperate ones? > Also you can do a tcpdump on both guests and see what happened to SSH packets > (tcpdump -i tapXXX port 22). Sadly not as there's too much traffic on that part as rsync is syncing every 5 minutes through ssh. >> The tap devices on the target vm shows dropped RX packages on BOTH tap >> interfaces - strangely with the same amount of pkts? >> >> # ifconfig tap317i0; ifconfig tap317i1 >> tap317i0 Link encap:Ethernet HWaddr 6e:cb:65:94:bb:bf >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >> RX packets:2238445 errors:0 dropped:13159 overruns:0 frame:0 >> TX packets:9655853 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:177991267 (169.7 MiB) TX bytes:910412749 (868.2 MiB) >> >> tap317i1 Link encap:Ethernet HWaddr 96:f8:b5:d0:9a:07 >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >> RX packets:1516085 errors:0 dropped:13159 overruns:0 frame:0 >> TX packets:1446964 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:1597564313 (1.4 GiB) TX bytes:3517734365 (3.2 GiB) >> >> Any ideas how to inspect this issue? > > It seems both tap interfaces lose RX pkts, dropping pkts of RX means the > host(backend) cann't receive packets from the guest as fast as the guest > sends. Inside the guest i see no dropped packets at all. It's only on the host and strangely on both taps at the same value? And both are connected to absolutely different networks. > Are you running some symmetrical test on both guests? No. Stefan > Wei > >> >> Greets, >> Stefan >>
[Qemu-devel] dropped pkts with Qemu on tap interace (RX)
Hello, currently i'm trying to fix a problem where we have "random" missing packets. We're doing an ssh connect from machine a to machine b every 5 minutes via rsync and ssh. Sometimes it happens that we get this cron message: "Connection to 192.168.0.2 closed by remote host. rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] ssh: connect to host 192.168.0.2 port 22: Connection refused" The tap devices on the target vm shows dropped RX packages on BOTH tap interfaces - strangely with the same amount of pkts? # ifconfig tap317i0; ifconfig tap317i1 tap317i0 Link encap:Ethernet HWaddr 6e:cb:65:94:bb:bf UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:2238445 errors:0 dropped:13159 overruns:0 frame:0 TX packets:9655853 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:177991267 (169.7 MiB) TX bytes:910412749 (868.2 MiB) tap317i1 Link encap:Ethernet HWaddr 96:f8:b5:d0:9a:07 UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:1516085 errors:0 dropped:13159 overruns:0 frame:0 TX packets:1446964 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1597564313 (1.4 GiB) TX bytes:3517734365 (3.2 GiB) Any ideas how to inspect this issue? Greets, Stefan
Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support
Hello, Am 22.11.2017 um 20:41 schrieb Dr. David Alan Gilbert: > * Paolo Bonzini (pbonz...@redhat.com) wrote: >> On 06/11/2017 12:09, Stefan Priebe - Profihost AG wrote: >>> HI Paolo, >>> >>> could this patchset be related? >> >> Uh oh, yes it should. Jason, any ways to fix it? I suppose we need to >> disable UFO in the newest machine types, but do we also have to do >> (software) UFO in vhost-net and QEMU for migration compatibility? > > Was there a solution to this? it will be this one: https://patchwork.ozlabs.org/patch/840094/ Stefan > Dave > >> Thanks, >> >> Paolo >> >>> Greets, >>> Stefan >>> >>> Am 06.11.2017 um 10:52 schrieb Stefan Priebe - Profihost AG: >>>> Hi Paolo, >>>> >>>> Am 06.11.2017 um 10:49 schrieb Paolo Bonzini: >>>>> On 06/11/2017 10:48, Stefan Priebe - Profihost AG wrote: >>>>>> Hi Paolo, >>>>>> >>>>>> Am 06.11.2017 um 10:40 schrieb Paolo Bonzini: >>>>>>> On 06/11/2017 10:38, Stefan Priebe - Profihost AG wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> i've upgraded some servers from kernel 4.4 to 4.12 - both running Qemu >>>>>>>> 2.9.1. >>>>>>>> >>>>>>>> If i migrate a VM from a host running kernel 4.4 to a host running 4.12 >>>>>>>> i get: >>>>>>>> >>>>>>>> kvm: virtio-net: saved image requires TUN_F_UFO support >>>>>>>> kvm: Failed to load virtio-net-device:tmp >>>>>>>> kvm: Failed to load virtio-net:virtio >>>>>>>> kvm: error while loading state for instance 0x0 of device >>>>>>>> ':00:12.0/virtio-net' >>>>>>>> kvm: load of migration failed: Invalid argument >>>>>>>> >>>>>>>> >>>>>>>> while migrating from 4.12 to 4.4 works fine. >>>>>>>> >>>>>>>> Can anybody help? Is this expected? >>>>>>> >>>>>>> Can you check why peer_has_ufo failed (in hw/net/virtio-net.c)? >>>>>> >>>>>> May be - how can i archieve this? Patching the code is not a problem if >>>>>> you can give me a hint. >>>>>> >>>>>>> Also, did this ioctl fail when the tap device was set up on the 4.12 >>>>>>> destination? >>>>>>> int tap_probe_has_ufo(int fd) >>>>>>> { >>>>>>> unsigned offload; >>>>>>> >>>>>>> offload = TUN_F_CSUM | TUN_F_UFO; >>>>>>> >>>>>>> if (ioctl(fd, TUNSETOFFLOAD, offload) < 0) >>>>>>> return 0; >>>>>>> >>>>>>> return 1; >>>>>>> } >>>>>> >>>>>> Should there be any kernel output or how can i detect / check it? >>>>> >>>>> For both, the simplest answer is probably just using printf. >>>> >>>> arg i missed an important part. The kernel is an opensuse SLE15 one. >>>> >>>> I've seen it contains the following patchset: >>>> https://www.spinics.net/lists/netdev/msg443821.html >>>> >>>> Greets, >>>> Stefan >>>> >> >> > -- > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK >
[Qemu-devel] kvm: Failed to flush the L2 table cache: Input/output error
Hello, while using qemu 2.9.1 and doing a backup of a disk: I have sometimes the following output: Formatting '/mnt/qemu-249-2017_11_19-04_00_05.qcow2', fmt=qcow2 size=236223201280 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 followed by: kvm: Failed to flush the L2 table cache: Input/output error kvm: Failed to flush the refcount block cache: Input/output error If this error happens (Failed to flush) the whole backup is incomplete. Host kernel is a 4.4 based openSuSE 42.3 kernel. Is this a known bug? Greets, Stefan
Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support
Hello, Am 10.11.2017 um 05:18 schrieb Jason Wang: > > > On 2017年11月08日 19:22, Jason Wang wrote: >> >> >> On 2017年11月08日 18:46, Paolo Bonzini wrote: >>> On 08/11/2017 09:21, Jason Wang wrote: >>>> >>>> On 2017年11月08日 17:05, Stefan Priebe - Profihost AG wrote: >>>>> Am 08.11.2017 um 08:54 schrieb Jason Wang: >>>>>> On 2017年11月08日 15:41, Stefan Priebe - Profihost AG wrote: >>>>>>> Hi Paolo, >>>>>>> >>>>>>> Am 06.11.2017 um 12:27 schrieb Paolo Bonzini: >>>>>>>> On 06/11/2017 12:09, Stefan Priebe - Profihost AG wrote: >>>>>>>>> HI Paolo, >>>>>>>>> >>>>>>>>> could this patchset be related? >>>>>>>> Uh oh, yes it should. Jason, any ways to fix it? I suppose we >>>>>>>> need to >>>>>>>> disable UFO in the newest machine types, but do we also have to do >>>>>>>> (software) UFO in vhost-net and QEMU for migration compatibility? >>>>>>> Any news on this? >>>>>>> >>>>>>> Greets, >>>>>> Since we probe UFO support, it will be disabled automatically on >>>>>> device >>>>>> startup. >>>>>> >>>>>> For the issue of migration, I think the only way is trying to fix >>>>>> it in >>>>>> kernel. >>>>> But 4.14 is around the corner and the patchset is already included. >>>>> Shouldn't this get fixed before 4.14 is released? >>>> We will try to seek a solution as soon as possible. If we can't catch >>>> 4.14, we will do it for stable for sure. >>> Jason, can you write to netdev and Cc Linus and me? >>> >>> Thanks, >>> >>> Paolo >> >> Paolo, see this https://www.spinics.net/lists/netdev/msg465454.html >> >> Just notice this today since I'm not on the cc list. >> >> Thanks >> >> > > An update: > > After some discussions on netdev, we will try to bring UFO back > partially for just TAP. Willem promise to fix this. 4.14 is too late > consider the fix is not trivial, it will go through stable tree. OK is it save to just revert the UFO patchset for my local branch? Greets, Stefan > > Thanks
Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support
Am 08.11.2017 um 08:54 schrieb Jason Wang: > > > On 2017年11月08日 15:41, Stefan Priebe - Profihost AG wrote: >> Hi Paolo, >> >> Am 06.11.2017 um 12:27 schrieb Paolo Bonzini: >>> On 06/11/2017 12:09, Stefan Priebe - Profihost AG wrote: >>>> HI Paolo, >>>> >>>> could this patchset be related? >>> Uh oh, yes it should. Jason, any ways to fix it? I suppose we need to >>> disable UFO in the newest machine types, but do we also have to do >>> (software) UFO in vhost-net and QEMU for migration compatibility? >> Any news on this? >> >> Greets, > > Since we probe UFO support, it will be disabled automatically on device > startup. > > For the issue of migration, I think the only way is trying to fix it in > kernel. But 4.14 is around the corner and the patchset is already included. Shouldn't this get fixed before 4.14 is released? Stefan > > Thanks
Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support
Hi Paolo, Am 06.11.2017 um 12:27 schrieb Paolo Bonzini: > On 06/11/2017 12:09, Stefan Priebe - Profihost AG wrote: >> HI Paolo, >> >> could this patchset be related? > > Uh oh, yes it should. Jason, any ways to fix it? I suppose we need to > disable UFO in the newest machine types, but do we also have to do > (software) UFO in vhost-net and QEMU for migration compatibility? Any news on this? Greets, Stefan > Thanks, > > Paolo > >> Greets, >> Stefan >> >> Am 06.11.2017 um 10:52 schrieb Stefan Priebe - Profihost AG: >>> Hi Paolo, >>> >>> Am 06.11.2017 um 10:49 schrieb Paolo Bonzini: >>>> On 06/11/2017 10:48, Stefan Priebe - Profihost AG wrote: >>>>> Hi Paolo, >>>>> >>>>> Am 06.11.2017 um 10:40 schrieb Paolo Bonzini: >>>>>> On 06/11/2017 10:38, Stefan Priebe - Profihost AG wrote: >>>>>>> Hello, >>>>>>> >>>>>>> i've upgraded some servers from kernel 4.4 to 4.12 - both running Qemu >>>>>>> 2.9.1. >>>>>>> >>>>>>> If i migrate a VM from a host running kernel 4.4 to a host running 4.12 >>>>>>> i get: >>>>>>> >>>>>>> kvm: virtio-net: saved image requires TUN_F_UFO support >>>>>>> kvm: Failed to load virtio-net-device:tmp >>>>>>> kvm: Failed to load virtio-net:virtio >>>>>>> kvm: error while loading state for instance 0x0 of device >>>>>>> ':00:12.0/virtio-net' >>>>>>> kvm: load of migration failed: Invalid argument >>>>>>> >>>>>>> >>>>>>> while migrating from 4.12 to 4.4 works fine. >>>>>>> >>>>>>> Can anybody help? Is this expected? >>>>>> >>>>>> Can you check why peer_has_ufo failed (in hw/net/virtio-net.c)? >>>>> >>>>> May be - how can i archieve this? Patching the code is not a problem if >>>>> you can give me a hint. >>>>> >>>>>> Also, did this ioctl fail when the tap device was set up on the 4.12 >>>>>> destination? >>>>>> int tap_probe_has_ufo(int fd) >>>>>> { >>>>>> unsigned offload; >>>>>> >>>>>> offload = TUN_F_CSUM | TUN_F_UFO; >>>>>> >>>>>> if (ioctl(fd, TUNSETOFFLOAD, offload) < 0) >>>>>> return 0; >>>>>> >>>>>> return 1; >>>>>> } >>>>> >>>>> Should there be any kernel output or how can i detect / check it? >>>> >>>> For both, the simplest answer is probably just using printf. >>> >>> arg i missed an important part. The kernel is an opensuse SLE15 one. >>> >>> I've seen it contains the following patchset: >>> https://www.spinics.net/lists/netdev/msg443821.html >>> >>> Greets, >>> Stefan >>> >
Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support
HI Paolo, could this patchset be related? Greets, Stefan Am 06.11.2017 um 10:52 schrieb Stefan Priebe - Profihost AG: > Hi Paolo, > > Am 06.11.2017 um 10:49 schrieb Paolo Bonzini: >> On 06/11/2017 10:48, Stefan Priebe - Profihost AG wrote: >>> Hi Paolo, >>> >>> Am 06.11.2017 um 10:40 schrieb Paolo Bonzini: >>>> On 06/11/2017 10:38, Stefan Priebe - Profihost AG wrote: >>>>> Hello, >>>>> >>>>> i've upgraded some servers from kernel 4.4 to 4.12 - both running Qemu >>>>> 2.9.1. >>>>> >>>>> If i migrate a VM from a host running kernel 4.4 to a host running 4.12 >>>>> i get: >>>>> >>>>> kvm: virtio-net: saved image requires TUN_F_UFO support >>>>> kvm: Failed to load virtio-net-device:tmp >>>>> kvm: Failed to load virtio-net:virtio >>>>> kvm: error while loading state for instance 0x0 of device >>>>> ':00:12.0/virtio-net' >>>>> kvm: load of migration failed: Invalid argument >>>>> >>>>> >>>>> while migrating from 4.12 to 4.4 works fine. >>>>> >>>>> Can anybody help? Is this expected? >>>> >>>> Can you check why peer_has_ufo failed (in hw/net/virtio-net.c)? >>> >>> May be - how can i archieve this? Patching the code is not a problem if >>> you can give me a hint. >>> >>>> Also, did this ioctl fail when the tap device was set up on the 4.12 >>>> destination? >>>> int tap_probe_has_ufo(int fd) >>>> { >>>> unsigned offload; >>>> >>>> offload = TUN_F_CSUM | TUN_F_UFO; >>>> >>>> if (ioctl(fd, TUNSETOFFLOAD, offload) < 0) >>>> return 0; >>>> >>>> return 1; >>>> } >>> >>> Should there be any kernel output or how can i detect / check it? >> >> For both, the simplest answer is probably just using printf. > > arg i missed an important part. The kernel is an opensuse SLE15 one. > > I've seen it contains the following patchset: > https://www.spinics.net/lists/netdev/msg443821.html > > Greets, > Stefan >
Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support
Hi Paolo, Am 06.11.2017 um 10:49 schrieb Paolo Bonzini: > On 06/11/2017 10:48, Stefan Priebe - Profihost AG wrote: >> Hi Paolo, >> >> Am 06.11.2017 um 10:40 schrieb Paolo Bonzini: >>> On 06/11/2017 10:38, Stefan Priebe - Profihost AG wrote: >>>> Hello, >>>> >>>> i've upgraded some servers from kernel 4.4 to 4.12 - both running Qemu >>>> 2.9.1. >>>> >>>> If i migrate a VM from a host running kernel 4.4 to a host running 4.12 >>>> i get: >>>> >>>> kvm: virtio-net: saved image requires TUN_F_UFO support >>>> kvm: Failed to load virtio-net-device:tmp >>>> kvm: Failed to load virtio-net:virtio >>>> kvm: error while loading state for instance 0x0 of device >>>> ':00:12.0/virtio-net' >>>> kvm: load of migration failed: Invalid argument >>>> >>>> >>>> while migrating from 4.12 to 4.4 works fine. >>>> >>>> Can anybody help? Is this expected? >>> >>> Can you check why peer_has_ufo failed (in hw/net/virtio-net.c)? >> >> May be - how can i archieve this? Patching the code is not a problem if >> you can give me a hint. >> >>> Also, did this ioctl fail when the tap device was set up on the 4.12 >>> destination? >>> int tap_probe_has_ufo(int fd) >>> { >>> unsigned offload; >>> >>> offload = TUN_F_CSUM | TUN_F_UFO; >>> >>> if (ioctl(fd, TUNSETOFFLOAD, offload) < 0) >>> return 0; >>> >>> return 1; >>> } >> >> Should there be any kernel output or how can i detect / check it? > > For both, the simplest answer is probably just using printf. arg i missed an important part. The kernel is an opensuse SLE15 one. I've seen it contains the following patchset: https://www.spinics.net/lists/netdev/msg443821.html Greets, Stefan
Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support
Hi Paolo, Am 06.11.2017 um 10:40 schrieb Paolo Bonzini: > On 06/11/2017 10:38, Stefan Priebe - Profihost AG wrote: >> Hello, >> >> i've upgraded some servers from kernel 4.4 to 4.12 - both running Qemu >> 2.9.1. >> >> If i migrate a VM from a host running kernel 4.4 to a host running 4.12 >> i get: >> >> kvm: virtio-net: saved image requires TUN_F_UFO support >> kvm: Failed to load virtio-net-device:tmp >> kvm: Failed to load virtio-net:virtio >> kvm: error while loading state for instance 0x0 of device >> ':00:12.0/virtio-net' >> kvm: load of migration failed: Invalid argument >> >> >> while migrating from 4.12 to 4.4 works fine. >> >> Can anybody help? Is this expected? > > Can you check why peer_has_ufo failed (in hw/net/virtio-net.c)? May be - how can i archieve this? Patching the code is not a problem if you can give me a hint. > Also, did this ioctl fail when the tap device was set up on the 4.12 > destination? > int tap_probe_has_ufo(int fd) > { > unsigned offload; > > offload = TUN_F_CSUM | TUN_F_UFO; > > if (ioctl(fd, TUNSETOFFLOAD, offload) < 0) > return 0; > > return 1; > } Should there be any kernel output or how can i detect / check it? Greets, Stefan > Thanks, > > Paolo >
[Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support
Hello, i've upgraded some servers from kernel 4.4 to 4.12 - both running Qemu 2.9.1. If i migrate a VM from a host running kernel 4.4 to a host running 4.12 i get: kvm: virtio-net: saved image requires TUN_F_UFO support kvm: Failed to load virtio-net-device:tmp kvm: Failed to load virtio-net:virtio kvm: error while loading state for instance 0x0 of device ':00:12.0/virtio-net' kvm: load of migration failed: Invalid argument while migrating from 4.12 to 4.4 works fine. Can anybody help? Is this expected? Greets, Stefan
Re: [Qemu-devel] virtio: network: lost tcp/ip packets
Hello Stefan, Am 30.08.2017 um 19:17 schrieb Stefan Hajnoczi: > On Fri, Aug 18, 2017 at 04:40:36PM +0200, Stefan Priebe - Profihost AG wrote: >> i've a problem with two VMs running on the SAME host machine using qemu >> 2.7.1 or 2.9.0 and vhost_net + virtio. >> >> Sometimes TCP packets going from machine a to machine b are simply lost. >> I see them in VM A using tcpdump going out but they never come in on >> machine B. Both machines have iptables and stuff turned off. >> >> On the host both VMs network interfaces are connected to the same bridge. >> >> Any ideas? May be a known bug? >> >> Host and Guest Kernel is an OpenSuSE 42.3 kernel based on vanilla 4.4.82. > > Have you tcpdumped the tap interfaces on the host? > currently i can't reproduce the issue - even i was for two days. Currently i've no idea what has happend and will report back if it happens again. Thanks! Greets, Stefan
Re: [Qemu-devel] virtio: network: lost tcp/ip packets
Hello, does nobody have an idea? Greets, Stefam Am 18.08.2017 um 16:40 schrieb Stefan Priebe - Profihost AG: > Hello, > > i've a problem with two VMs running on the SAME host machine using qemu > 2.7.1 or 2.9.0 and vhost_net + virtio. > > Sometimes TCP packets going from machine a to machine b are simply lost. > I see them in VM A using tcpdump going out but they never come in on > machine B. Both machines have iptables and stuff turned off. > > On the host both VMs network interfaces are connected to the same bridge. > > Any ideas? May be a known bug? > > Host and Guest Kernel is an OpenSuSE 42.3 kernel based on vanilla 4.4.82. > > Thanks! > > Greets, > Stefan >
[Qemu-devel] virtio: network: lost tcp/ip packets
Hello, i've a problem with two VMs running on the SAME host machine using qemu 2.7.1 or 2.9.0 and vhost_net + virtio. Sometimes TCP packets going from machine a to machine b are simply lost. I see them in VM A using tcpdump going out but they never come in on machine B. Both machines have iptables and stuff turned off. On the host both VMs network interfaces are connected to the same bridge. Any ideas? May be a known bug? Host and Guest Kernel is an OpenSuSE 42.3 kernel based on vanilla 4.4.82. Thanks! Greets, Stefan
Re: [Qemu-devel] any known virtio-net regressions in Qemu 2.7?
Am 19.12.2016 um 12:03 schrieb Stefan Hajnoczi: > On Fri, Dec 16, 2016 at 10:00:36PM +0100, Stefan Priebe - Profihost AG wrote: >> >> Am 15.12.2016 um 07:46 schrieb Alexandre DERUMIER: >>> does rollbacking the kernel to previous version fix the problem ? >> >> The culprit is the used tuned agent from Redhat >> (https://github.com/redhat-performance/tuned). The used profile >> virtual-host results in these problems. Stopping tuned or using another >> profile like throughput-performance everything is fine again. > > Interesting discovery. Have you filed a bug report about it? As we're not using RHEL, nor CentOS i didn't expect this to make sense ;-) I just believed that something in the kernel changed. Greets, Stefan >> after upgrading a cluster OS, Qemu, ... i'm experiencing slow and >> volatile network speeds inside my VMs. >> >> Currently I've no idea what causes this but it's related to the host >> upgrades. Before i was running Qemu 2.6.2. >> >> I'm using virtio for the network cards. > > Stefan >
Re: [Qemu-devel] any known virtio-net regressions in Qemu 2.7?
Am 15.12.2016 um 07:46 schrieb Alexandre DERUMIER: > does rollbacking the kernel to previous version fix the problem ? The culprit is the used tuned agent from Redhat (https://github.com/redhat-performance/tuned). The used profile virtual-host results in these problems. Stopping tuned or using another profile like throughput-performance everything is fine again. Geets, Stefan > > i'm not sure if "perf" could give you some hints > - Mail original - > De: "Stefan Priebe, Profihost AG" > À: "aderumier" > Cc: "qemu-devel" > Envoyé: Mercredi 14 Décembre 2016 21:36:23 > Objet: Re: [Qemu-devel] any known virtio-net regressions in Qemu 2.7? > > Am 14.12.2016 um 16:33 schrieb Alexandre DERUMIER: >> Hi Stefan, >> >> do you have upgraded kernel ? > > Yes sure. But i'm out of ideas how to debug. Sometimes it gives me > constant 80MB/s, sometimes 125 and sometimes only 6. While on the host > the cards are not busy. > > Greets, > Stefan > >> >> maybe it could be related to vhost-net module too. >> >> >> - Mail original - >> De: "Stefan Priebe, Profihost AG" >> À: "qemu-devel" >> Envoyé: Mercredi 14 Décembre 2016 16:04:08 >> Objet: [Qemu-devel] any known virtio-net regressions in Qemu 2.7? >> >> Hello, >> >> after upgrading a cluster OS, Qemu, ... i'm experiencing slow and >> volatile network speeds inside my VMs. >> >> Currently I've no idea what causes this but it's related to the host >> upgrades. Before i was running Qemu 2.6.2. >> >> I'm using virtio for the network cards. >> >> Greets, >> Stefan >> >
Re: [Qemu-devel] any known virtio-net regressions in Qemu 2.7?
Am 14.12.2016 um 16:33 schrieb Alexandre DERUMIER: > Hi Stefan, > > do you have upgraded kernel ? Yes sure. But i'm out of ideas how to debug. Sometimes it gives me constant 80MB/s, sometimes 125 and sometimes only 6. While on the host the cards are not busy. Greets, Stefan > > maybe it could be related to vhost-net module too. > > > - Mail original - > De: "Stefan Priebe, Profihost AG" > À: "qemu-devel" > Envoyé: Mercredi 14 Décembre 2016 16:04:08 > Objet: [Qemu-devel] any known virtio-net regressions in Qemu 2.7? > > Hello, > > after upgrading a cluster OS, Qemu, ... i'm experiencing slow and > volatile network speeds inside my VMs. > > Currently I've no idea what causes this but it's related to the host > upgrades. Before i was running Qemu 2.6.2. > > I'm using virtio for the network cards. > > Greets, > Stefan >
[Qemu-devel] any known virtio-net regressions in Qemu 2.7?
Hello, after upgrading a cluster OS, Qemu, ... i'm experiencing slow and volatile network speeds inside my VMs. Currently I've no idea what causes this but it's related to the host upgrades. Before i was running Qemu 2.6.2. I'm using virtio for the network cards. Greets, Stefan
Re: [Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'
Am 15.11.2016 um 12:07 schrieb Ladi Prosek: > Hi, > > On Tue, Nov 15, 2016 at 11:37 AM, Stefan Priebe - Profihost AG > wrote: >> Hello, >> >> Am 15.11.2016 um 11:30 schrieb Dr. David Alan Gilbert: >>> * Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote: >>>> Hello, >>>> >>>> today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM >>>> is running windows and virtio-balloon and with machine type pc-i440fx-2.5. >>>> >>>> The output of the target qemu process was: >>>> kvm_apic_post_load: Yeh >>>> kvm_apic_post_load: Yeh >>>> kvm_apic_post_load: Yeh >>>> kvm_apic_post_load: Yeh >>>> kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4 >>>> kvm: error while loading state for instance 0x0 of device >>>> ':00:03.0/virtio-balloon' >>>> kvm: load of migration failed: Operation not permitted >>> >>> Yes that's a known bug; only seems to affect windows guests, and I believe >>> doesn't even need to cross versions. >>> >>> There's a bunch of fixes that Stefan applied to virtio code >>> that I think fix this; I see that he cc'd qemu-stable. >>> I think it's 4b7f91ed, but I'm not sure if there are others needed. >> >> thanks for pointing to that commit. >> >> Stefan can you tell me whether it's enough to cherry-pick 4b7f91ed into >> 2.7.0 ? > > I don't believe that 4b7f91ed will help here (no device reset on > migration). We've seen this error with QEMU running without: > > commit 4eae2a657d1ff5ada56eb9b4966eae0eff333b0b > Author: Ladi Prosek > Date: Tue Mar 1 12:14:03 2016 +0100 > > balloon: fix segfault and harden the stats queue > > > Is it possible that the VM has run on such a QEMU, then was > live-migrated to 2.6.2, and then to 2.7.0? Hi, yes, it was started under Qemu 2.5.0. Was then migrated to 2.6.2 and then to 2.7.0. Greets, Stefan > > Thanks, > Ladi > >> Greets, >> Stefan >> >>> >>> Dave >>>> >>>> Greets, >>>> Stefan >>>> >>> -- >>> Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK >>> >>
Re: [Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'
Hello, Am 15.11.2016 um 11:30 schrieb Dr. David Alan Gilbert: > * Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote: >> Hello, >> >> today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM >> is running windows and virtio-balloon and with machine type pc-i440fx-2.5. >> >> The output of the target qemu process was: >> kvm_apic_post_load: Yeh >> kvm_apic_post_load: Yeh >> kvm_apic_post_load: Yeh >> kvm_apic_post_load: Yeh >> kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4 >> kvm: error while loading state for instance 0x0 of device >> ':00:03.0/virtio-balloon' >> kvm: load of migration failed: Operation not permitted > > Yes that's a known bug; only seems to affect windows guests, and I believe > doesn't even need to cross versions. > > There's a bunch of fixes that Stefan applied to virtio code > that I think fix this; I see that he cc'd qemu-stable. > I think it's 4b7f91ed, but I'm not sure if there are others needed. thanks for pointing to that commit. Stefan can you tell me whether it's enough to cherry-pick 4b7f91ed into 2.7.0 ? Greets, Stefan > > Dave >> >> Greets, >> Stefan >> > -- > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK >
[Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'
Hello, today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM is running windows and virtio-balloon and with machine type pc-i440fx-2.5. The output of the target qemu process was: kvm_apic_post_load: Yeh kvm_apic_post_load: Yeh kvm_apic_post_load: Yeh kvm_apic_post_load: Yeh kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4 kvm: error while loading state for instance 0x0 of device ':00:03.0/virtio-balloon' kvm: load of migration failed: Operation not permitted Greets, Stefan
Re: [Qemu-devel] [PATCH] net: fix qemu_announce_self not emitting packets
Am 07.06.2016 um 09:37 schrieb Peter Lieven: > commit fefe2a78 accidently dropped the code path for injecting > raw packets. This feature is needed for sending gratuitous ARPs > after an incoming migration has completed. The result is increased > network downtime for vservers where the network card is not virtio-net > with the VIRTIO_NET_F_GUEST_ANNOUNCE feature. > > Fixes: fefe2a78abde932e0f340b21bded2c86def1d242 > Cc: qemu-sta...@nongnu.org > Cc: yan...@cn.fujitsu.com > Signed-off-by: Peter Lieven > --- > net/net.c | 6 +- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/net/net.c b/net/net.c > index 5f3e5a9..d5834ea 100644 > --- a/net/net.c > +++ b/net/net.c > @@ -722,7 +722,11 @@ ssize_t qemu_deliver_packet_iov(NetClientState *sender, > return 0; > } > > -if (nc->info->receive_iov) { > +if (flags & QEMU_NET_PACKET_FLAG_RAW && iovcnt == 1 && > +nc->info->receive_raw) { > +/* this is required for qemu_announce_self() */ > +ret = nc->info->receive_raw(nc, iov[0].iov_base, iov[0].iov_len); > +} else if (nc->info->receive_iov) { > ret = nc->info->receive_iov(nc, iov, iovcnt); > } else { > ret = nc_sendv_compat(nc, iov, iovcnt, flags); > Thanks for the patch. Sadly it does not fix our problem. So we still have another Problem. Thanks! Greets, Stefan
Re: [Qemu-devel] [Qemu-stable] Broken live Migration in Qemu 2.5.1.1?
Am 07.06.2016 um 09:38 schrieb Peter Lieven: > Am 06.06.2016 um 18:13 schrieb Stefan Priebe - Profihost AG: >> We're most probably seeing the same while migrating a machine running >> balanceng but haven't thought this might be a qemu bug. Instead we're >> investigating with balanceng people. >> >> Waiting for your further results. > > Can you try the patch > > net: fix qemu_announce_self not emitting packets Thx - will wait for a new patch based on Paolo's comments - if that's ok. Stefan > I just send to the mailing list. > > Thanks, > Peter >
Re: [Qemu-devel] [Qemu-stable] Broken live Migration in Qemu 2.5.1.1?
We're most probably seeing the same while migrating a machine running balanceng but haven't thought this might be a qemu bug. Instead we're investigating with balanceng people. Waiting for your further results. Greets, Stefan Excuse my typo sent from my mobile phone. > Am 06.06.2016 um 17:51 schrieb Peter Lieven : > >> Am 06.06.2016 um 15:32 schrieb Peter Lieven: >> Hi, >> >> during internal testing of Qemu 2.5.1.1 I found a vServer running Ubuntu >> 12.04 (kernel 3.13) and a slave SQL server to >> stop replicating from the master. This seems to be reproducible. It is >> possible to continue replication when issuing a slave stop / slave start. >> There is no error visible on the vServer. >> >> Has anyone a fix in mind that could be related to such an issue? >> >> Host kernel in Linux 4.4, Guest kernel 3.13. Guest driver is virtio-blk via >> iSCSI. Emulated vCPU is Westmere. > > After a lot of testing I found out that obviously thats no block driver > problem, but a regression in the virtio-net or the network stack. > > qemu_announce_self() is generating packets for all NICs, but it seems they > are no longer emitted. This worked at least in qemu-2.2.0 with > the same guest kernel and host kernel. > > I will continue debugging tomorrow why this happens. > > Peter >
Re: [Qemu-devel] drive-backup
Am 25.02.2016 um 20:53 schrieb John Snow: On 02/25/2016 02:49 AM, Stefan Priebe - Profihost AG wrote: Am 22.02.2016 um 23:08 schrieb John Snow: On 02/22/2016 03:21 PM, Stefan Priebe wrote: Hello, is there any chance or hack to work with a bigger cluster size for the drive backup job? See: http://git.qemu.org/?p=qemu.git;a=blob;f=block/backup.c;h=16105d40b193be9bb40346027bdf58e62b956a96;hb=98d2c6f2cd80afaa2dc10091f5e35a97c181e4f5 This is very slow with ceph - may be due to the 64k block size. I would like to check whether this is faster with cephs native block size of 4mb. Greets, Stefan It's hardcoded to 64K at the moment, but I am checking in a patch to round up the cluster size to be the bigger of (64k, $target_cluster_size) in order to make sure that incremental backups in particular never copy a fraction of a cluster. As a side-effect, the same round-up will happen for all modes (sync=top,none,full). If QEMU is aware of the target cluster size of 4MB, this would immediately jump the copy-size up to 4MB clusters for you. See: https://lists.nongnu.org/archive/html/qemu-devel/2016-02/msg02839.html Thanks for your patches and thanks for your great answer. But our problem is not the target but the source ;-) The target has a local cache and don't care about the cluster size but the source does not. But it works fine if we change the default cluster size to 4MB. So it has point us to the right direction. Stefan Ah, sorry, I misunderstood. It's easy to change anyway! I am in favor of adding a configurable parameter, as long as it keeps the other constraints I mentioned in mind. ah great and thanks! Stefan --js Otherwise, after my trivial fix, you should find cluster_size to be a mutable concept and perhaps one that you could introduce a runtime parameter for if you could convince the necessary parties that it's needed in the API. You'd have to be careful in the case of incremental that all the various cluster sizes work well together: - Granularity of bitmap (Defaults to cluster size of source, or 64K if unknown. can be configured to be arbitrarily larger.) - Cluster size of source file (For qcow2, defaults to 64k) - Cluster size of target file - Cluster size of backup routine (Currently always 64K) I think that LCM(source_cluster_size, target_cluster_size, backup_cluster_size) = MAX(A, B, C) will always be a safe minimum. Bitmap granularity is more flexible, and it is more efficient when the bitmap granularity matches the backup cluster size, but it can cope with mismatches. If the bitmap is smaller or larger than the backup cluster size, it generally means that more data that is clean is going to be transferred across the pipe. (Hmm, I wonder if it's worth checking in code to adjust a bitmap granularity after it has been created so people can easily experiment with performance tweaking here.) Let me know if any of my ramble sounds interesting :) --John
Re: [Qemu-devel] drive-backup
Am 22.02.2016 um 23:08 schrieb John Snow: > > > On 02/22/2016 03:21 PM, Stefan Priebe wrote: >> Hello, >> >> is there any chance or hack to work with a bigger cluster size for the >> drive backup job? >> >> See: >> http://git.qemu.org/?p=qemu.git;a=blob;f=block/backup.c;h=16105d40b193be9bb40346027bdf58e62b956a96;hb=98d2c6f2cd80afaa2dc10091f5e35a97c181e4f5 >> >> >> This is very slow with ceph - may be due to the 64k block size. I would >> like to check whether this is faster with cephs native block size of 4mb. >> >> Greets, >> Stefan >> > > It's hardcoded to 64K at the moment, but I am checking in a patch to > round up the cluster size to be the bigger of (64k, > $target_cluster_size) in order to make sure that incremental backups in > particular never copy a fraction of a cluster. As a side-effect, the > same round-up will happen for all modes (sync=top,none,full). > > If QEMU is aware of the target cluster size of 4MB, this would > immediately jump the copy-size up to 4MB clusters for you. > > See: https://lists.nongnu.org/archive/html/qemu-devel/2016-02/msg02839.html Thanks for your patches and thanks for your great answer. But our problem is not the target but the source ;-) The target has a local cache and don't care about the cluster size but the source does not. But it works fine if we change the default cluster size to 4MB. So it has point us to the right direction. Stefan > > Otherwise, after my trivial fix, you should find cluster_size to be a > mutable concept and perhaps one that you could introduce a runtime > parameter for if you could convince the necessary parties that it's > needed in the API. > > You'd have to be careful in the case of incremental that all the various > cluster sizes work well together: > > - Granularity of bitmap (Defaults to cluster size of source, or 64K if > unknown. can be configured to be arbitrarily larger.) > - Cluster size of source file (For qcow2, defaults to 64k) > - Cluster size of target file > - Cluster size of backup routine (Currently always 64K) > > I think that LCM(source_cluster_size, target_cluster_size, > backup_cluster_size) = MAX(A, B, C) will always be a safe minimum. > > Bitmap granularity is more flexible, and it is more efficient when the > bitmap granularity matches the backup cluster size, but it can cope with > mismatches. If the bitmap is smaller or larger than the backup cluster > size, it generally means that more data that is clean is going to be > transferred across the pipe. > > (Hmm, I wonder if it's worth checking in code to adjust a bitmap > granularity after it has been created so people can easily experiment > with performance tweaking here.) > > > Let me know if any of my ramble sounds interesting :) > --John >
[Qemu-devel] drive-backup
Hello, is there any chance or hack to work with a bigger cluster size for the drive backup job? See: http://git.qemu.org/?p=qemu.git;a=blob;f=block/backup.c;h=16105d40b193be9bb40346027bdf58e62b956a96;hb=98d2c6f2cd80afaa2dc10091f5e35a97c181e4f5 This is very slow with ceph - may be due to the 64k block size. I would like to check whether this is faster with cephs native block size of 4mb. Greets, Stefan
Re: [Qemu-devel] kernel 4.4.2: kvm_irq_delivery_to_api / rwsem_down_read_failed
Am 22.02.2016 um 18:36 schrieb Paolo Bonzini: On 20/02/2016 11:44, Stefan Priebe wrote: Hi, while testing Kernel 4.4.2 and starting 20 Qemu 2.4.1 virtual machines. I got those traces and a load of 500 on those system. I was only abler to recover by sysrq-trigger. It seems like something happening at the VM level. A task took the mm semaphore and hung everyone else. Difficult to debug without a core (and without knowing who held the semaphore). Sorry. OK thank you anyway. Is there anything i can do if this happens again? Stefan Paolo All traces: INFO: task pvedaemon worke:7470 blocked for more than 120 seconds. Not tainted 4.4.2+1-ph #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. pvedaemon worke D 88239c367ca0 0 7470 7468 0x0008 88239c367ca0 8840a6232500 8823ed83a500 88239c367c90 88239c368000 8845f5f070e8 8845f5f07100 7ffc73b48e58 88239c367cc0 b66a4d89 88239c367cf0 Call Trace: [] schedule+0x39/0x80 [] rwsem_down_read_failed+0xc7/0x120 [] call_rwsem_down_read_failed+0x14/0x30 [] ? down_read+0x17/0x20 [] __access_remote_vm+0x3e/0x1c0 [] ? call_rwsem_down_read_failed+0x14/0x30 [] access_remote_vm+0x1f/0x30 [] proc_pid_cmdline_read+0x16e/0x4f0 [] ? acct_account_cputime+0x1c/0x20 [] __vfs_read+0x18/0x40 [] vfs_read+0x8e/0x140 [] SyS_read+0x4f/0xa0 [] entry_SYSCALL_64_fastpath+0x12/0x71 INFO: task pvestatd:7633 blocked for more than 120 seconds. Not tainted 4.4.2+1-ph #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. pvestatd D 88239f16fd40 0 7633 1 0x0008 88239f16fd40 8824e76a8000 8823e5fc2500 8823e5fc2500 88239f17 8845f5f070e8 8845f5f07100 8845f5f07080 0341bf10 88239f16fd60 b66a4d89 024000d00058 Call Trace: [] schedule+0x39/0x80 [] rwsem_down_read_failed+0xc7/0x120 [] call_rwsem_down_read_failed+0x14/0x30 [] ? down_read+0x17/0x20 [] proc_pid_cmdline_read+0xac/0x4f0 [] ? acct_account_cputime+0x1c/0x20 [] ? account_user_time+0x73/0x80 [] ? vtime_account_user+0x4e/0x70 [] __vfs_read+0x18/0x40 [] vfs_read+0x8e/0x140 [] SyS_read+0x4f/0xa0 [] entry_SYSCALL_64_fastpath+0x12/0x71
[Qemu-devel] kernel 4.4.2: kvm_irq_delivery_to_api / rwsem_down_read_failed
Hi, while testing Kernel 4.4.2 and starting 20 Qemu 2.4.1 virtual machines. I got those traces and a load of 500 on those system. I was only abler to recover by sysrq-trigger. All traces: INFO: task pvedaemon worke:7470 blocked for more than 120 seconds. Not tainted 4.4.2+1-ph #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. pvedaemon worke D 88239c367ca0 0 7470 7468 0x0008 88239c367ca0 8840a6232500 8823ed83a500 88239c367c90 88239c368000 8845f5f070e8 8845f5f07100 7ffc73b48e58 88239c367cc0 b66a4d89 88239c367cf0 Call Trace: [] schedule+0x39/0x80 [] rwsem_down_read_failed+0xc7/0x120 [] call_rwsem_down_read_failed+0x14/0x30 [] ? down_read+0x17/0x20 [] __access_remote_vm+0x3e/0x1c0 [] ? call_rwsem_down_read_failed+0x14/0x30 [] access_remote_vm+0x1f/0x30 [] proc_pid_cmdline_read+0x16e/0x4f0 [] ? acct_account_cputime+0x1c/0x20 [] __vfs_read+0x18/0x40 [] vfs_read+0x8e/0x140 [] SyS_read+0x4f/0xa0 [] entry_SYSCALL_64_fastpath+0x12/0x71 INFO: task pvestatd:7633 blocked for more than 120 seconds. Not tainted 4.4.2+1-ph #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. pvestatd D 88239f16fd40 0 7633 1 0x0008 88239f16fd40 8824e76a8000 8823e5fc2500 8823e5fc2500 88239f17 8845f5f070e8 8845f5f07100 8845f5f07080 0341bf10 88239f16fd60 b66a4d89 024000d00058 Call Trace: [] schedule+0x39/0x80 [] rwsem_down_read_failed+0xc7/0x120 [] call_rwsem_down_read_failed+0x14/0x30 [] ? down_read+0x17/0x20 [] proc_pid_cmdline_read+0xac/0x4f0 [] ? acct_account_cputime+0x1c/0x20 [] ? account_user_time+0x73/0x80 [] ? vtime_account_user+0x4e/0x70 [] __vfs_read+0x18/0x40 [] vfs_read+0x8e/0x140 [] SyS_read+0x4f/0xa0 [] entry_SYSCALL_64_fastpath+0x12/0x71 INFO: task kvm:11766 blocked for more than 120 seconds. Not tainted 4.4.2+1-ph #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kvm D 88452a2d3dd0 0 11766 1 0x0008 88452a2d3dd0 880166c74a00 8845b7354a00 b617fc8e 88452a2d4000 8845f5f070e8 8845f5f07100 88452a2d3f58 8845b7354a00 88452a2d3df0 b66a4d89 7fa807abbf80 Call Trace: [] ? __handle_mm_fault+0xd1e/0x1260 [] schedule+0x39/0x80 [] rwsem_down_read_failed+0xc7/0x120 [] call_rwsem_down_read_failed+0x14/0x30 [] ? down_read+0x17/0x20 [] __do_page_fault+0x2b7/0x380 [] ? account_user_time+0x73/0x80 [] ? vtime_account_user+0x4e/0x70 [] do_page_fault+0x37/0x90 [] page_fault+0x28/0x30 INFO: task kvm:11824 blocked for more than 120 seconds. Not tainted 4.4.2+1-ph #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kvm D 8840a867faa0 0 11824 1 0x0008 8840a867faa0 8845866a4a00 8840a6232500 0001 8840a868 8845f5f070e8 8845f5f07100 8840a867fc0e 8840a867fac0 b66a4d89 c0606a06 Call Trace: [] schedule+0x39/0x80 [] ? kvm_irq_delivery_to_apic+0x56/0x220 [kvm] [] rwsem_down_read_failed+0xc7/0x120 [] call_rwsem_down_read_failed+0x14/0x30 [] ? down_read+0x17/0x20 [] kvm_host_page_size+0x60/0xa0 [kvm] [] mapping_level+0x5c/0x130 [kvm] [] tdp_page_fault+0x9b/0x260 [kvm] [] ? kernel_pio+0x2d/0x40 [kvm] [] kvm_mmu_page_fault+0x31/0x120 [kvm] [] handle_ept_violation+0xa4/0x170 [kvm_intel] [] vmx_handle_exit+0x257/0x490 [kvm_intel] [] ? __vtime_account_system+0x31/0x40 [] vcpu_enter_guest+0x6af/0xff0 [kvm] [] ? kvm_apic_local_deliver+0x5d/0x60 [kvm] [] kvm_arch_vcpu_ioctl_run+0xc4/0x3c0 [kvm] [] kvm_vcpu_ioctl+0x324/0x5d0 [kvm] [] ? acct_account_cputime+0x1c/0x20 [] ? account_user_time+0x73/0x80 [] do_vfs_ioctl+0x83/0x4e0 [] ? enter_from_user_mode+0x1f/0x50 [] ? syscall_trace_enter_phase1+0xc1/0x110 [] SyS_ioctl+0x4c/0x80 [] entry_SYSCALL_64_fastpath+0x12/0x71 INFO: task kvm:11825 blocked for more than 120 seconds. Not tainted 4.4.2+1-ph #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kvm D 88458d6a3aa0 0 11825 1 0x00080002 88458d6a3aa0 880167302500 8840a6234a00 0001 88458d6a4000 8845f5f070e8 8845f5f07100 88458d6a3c0e 88458d6a3ac0 b66a4d89 c0606a06 Call Trace: [] schedule+0x39/0x80 [] ? kvm_irq_delivery_to_apic+0x56/0x220 [kvm] [] rwsem_down_read_failed+0xc7/0x120 [] call_rwsem_down_read_failed+0x14/0x30 [] ? down_read+0x17/0x20 [] kvm_host_page_size+0x60/0xa0 [kvm] [] mapping_level+0x5c/0x130 [kvm] [] tdp_page_fault+0x9b/0x260 [kvm] [] kvm_mmu_page_fault+0x31/0x120 [kvm] [] handle_ept_violation+0xa4/0x170 [kvm_intel] [] vmx_handle_exit+0x257/0x490 [kvm_intel] [] ? __vtime_account_system+0x31/0x40 [] vcpu_enter_guest+0x6af/0xff0 [kvm] [] ? kvm_apic_local_deliver+0x5d/0x60 [kvm] [] kvm_arch_vcpu_ioctl_run+0xc4/0x3c0 [kvm] [] kvm_vcpu_ioctl+0x324/0x5d0 [kvm] [] ? acct_account_cputime+0x1c/0x20 [] ? account_user_time+0x73/0x80 [] do_vfs_ioctl+0x83/0x4e0 [] ? enter_from_
Re: [Qemu-devel] trace in arch/x86/kernel/apic/apic.c:1309 setup_local_APIC
Am 26.01.2016 um 12:39 schrieb Yang Zhang: > On 2016/1/26 18:43, Stefan Priebe - Profihost AG wrote: >> >> Am 26.01.2016 um 11:13 schrieb Yang Zhang: >>> On 2016/1/26 15:22, Stefan Priebe - Profihost AG wrote: >>>> Hi, >>>> >>>> Am 26.01.2016 um 02:46 schrieb Han, Huaitong: >>>>> What is the host kernel version and host dmesg information? And does >>>>> the problem exist when you use latest kernel and QEMU to replace old >>>>> binary file? >>>> >>>> Guest and Host is both 4.1.15. You mean the complete dmesg output from >>>> host? >>>> >>>> What do you mean with replace old binary file? I haven't tested Kernel >>>> 4.4 as we use 4.1 as it is a long term stable kernel release. >>> >>> Have you seen this before? I mean use the old KVM like 3.10? >> >> Guest or host? To test with a guest would be quite easy. Downgrading the >> host is very difficult not sure if the hw is supported. > > Host. Does the issue only exist on the Westmere CPU? Yes. All E5 Xeons v1, v2, v3 are working fine and i've never seen this message. Stefan > >> >>> >>>> >>>> Stefan >>>> >>>>> On Mon, 2016-01-25 at 14:51 +0100, Stefan Priebe - Profihost AG wrote: >>>>>> Hi, >>>>>> >>>>>> while running qemu 2.4 on whestmere CPUs i'm pretty often getting >>>>>> this >>>>>> one while booting: >>>>>> [0.811645] Switched APIC routing to physical x2apic. >>>>>> [1.835678] [ cut here ] >>>>>> [1.835704] WARNING: CPU: 0 PID: 1 at >>>>>> arch/x86/kernel/apic/apic.c:1309 setup_local_APIC+0x284/0x340() >>>>>> [1.835714] Modules linked in: >>>>>> [1.835724] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.15+72-ph >>>>>> #1 >>>>>> [1.835731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >>>>>> BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 >>>>>> [1.835743] b69ffcea 88042d5e3d68 b669c37b >>>>>> 0918 >>>>>> [1.835754] 88042d5e3da8 b6080d67 >>>>>> 88042d5e3da8 >>>>>> [1.835765] 0001 8000 >>>>>> >>>>>> [1.835777] Call Trace: >>>>>> [1.835789] [] dump_stack+0x45/0x57 >>>>>> [1.835799] [] warn_slowpath_common+0x97/0xe0 >>>>>> [1.835806] [] warn_slowpath_null+0x1a/0x20 >>>>>> [1.835813] [] setup_local_APIC+0x284/0x340 >>>>>> [1.835824] [] apic_bsp_setup+0x5b/0xb0 >>>>>> [1.835832] [] >>>>>> native_smp_prepare_cpus+0x23b/0x295 >>>>>> [1.835842] [] kernel_init_freeable+0xc7/0x20f >>>>>> [1.835853] [] ? rest_init+0x80/0x80 >>>>>> [1.835860] [] kernel_init+0xe/0xf0 >>>>>> [1.835870] [] ret_from_fork+0x42/0x70 >>>>>> [1.835877] [] ? rest_init+0x80/0x80 >>>>>> [1.835891] ---[ end trace bdbe630a8de2832c ]--- >>>>>> [1.837613] Spurious LAPIC timer interrupt on cpu 0 >>>>>> [1.837957] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 >>>>>> [1.939574] smpboot: CPU0: Intel Westmere E56xx/L56xx/X56xx >>>>>> (Nehalem-C) (fam: 06, model: 2c, stepping: 01) >>>>>> [1.939630] Performance Events: unsupported p6 CPU model 44 no PMU >>>>>> driver, software events only. >>>>>> [1.950868] KVM setup paravirtual spinlock >>>>>> >>>>>> Greets, >>>>>> Stefan >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>> the body of a message to majord...@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>> the body of a message to majord...@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>> >>> > >
Re: [Qemu-devel] trace in arch/x86/kernel/apic/apic.c:1309 setup_local_APIC
Am 26.01.2016 um 11:13 schrieb Yang Zhang: > On 2016/1/26 15:22, Stefan Priebe - Profihost AG wrote: >> Hi, >> >> Am 26.01.2016 um 02:46 schrieb Han, Huaitong: >>> What is the host kernel version and host dmesg information? And does >>> the problem exist when you use latest kernel and QEMU to replace old >>> binary file? >> >> Guest and Host is both 4.1.15. You mean the complete dmesg output from >> host? >> >> What do you mean with replace old binary file? I haven't tested Kernel >> 4.4 as we use 4.1 as it is a long term stable kernel release. > > Have you seen this before? I mean use the old KVM like 3.10? Guest or host? To test with a guest would be quite easy. Downgrading the host is very difficult not sure if the hw is supported. > >> >> Stefan >> >>> On Mon, 2016-01-25 at 14:51 +0100, Stefan Priebe - Profihost AG wrote: >>>> Hi, >>>> >>>> while running qemu 2.4 on whestmere CPUs i'm pretty often getting >>>> this >>>> one while booting: >>>> [0.811645] Switched APIC routing to physical x2apic. >>>> [1.835678] [ cut here ] >>>> [1.835704] WARNING: CPU: 0 PID: 1 at >>>> arch/x86/kernel/apic/apic.c:1309 setup_local_APIC+0x284/0x340() >>>> [1.835714] Modules linked in: >>>> [1.835724] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.15+72-ph >>>> #1 >>>> [1.835731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >>>> BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 >>>> [1.835743] b69ffcea 88042d5e3d68 b669c37b >>>> 0918 >>>> [1.835754] 88042d5e3da8 b6080d67 >>>> 88042d5e3da8 >>>> [1.835765] 0001 8000 >>>> >>>> [1.835777] Call Trace: >>>> [1.835789] [] dump_stack+0x45/0x57 >>>> [1.835799] [] warn_slowpath_common+0x97/0xe0 >>>> [1.835806] [] warn_slowpath_null+0x1a/0x20 >>>> [1.835813] [] setup_local_APIC+0x284/0x340 >>>> [1.835824] [] apic_bsp_setup+0x5b/0xb0 >>>> [1.835832] [] >>>> native_smp_prepare_cpus+0x23b/0x295 >>>> [1.835842] [] kernel_init_freeable+0xc7/0x20f >>>> [1.835853] [] ? rest_init+0x80/0x80 >>>> [1.835860] [] kernel_init+0xe/0xf0 >>>> [1.835870] [] ret_from_fork+0x42/0x70 >>>> [1.835877] [] ? rest_init+0x80/0x80 >>>> [1.835891] ---[ end trace bdbe630a8de2832c ]--- >>>> [1.837613] Spurious LAPIC timer interrupt on cpu 0 >>>> [1.837957] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 >>>> [1.939574] smpboot: CPU0: Intel Westmere E56xx/L56xx/X56xx >>>> (Nehalem-C) (fam: 06, model: 2c, stepping: 01) >>>> [1.939630] Performance Events: unsupported p6 CPU model 44 no PMU >>>> driver, software events only. >>>> [1.950868] KVM setup paravirtual spinlock >>>> >>>> Greets, >>>> Stefan >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>> the body of a message to majord...@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- >> To unsubscribe from this list: send the line "unsubscribe kvm" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > >
Re: [Qemu-devel] trace in arch/x86/kernel/apic/apic.c:1309 setup_local_APIC
Hi, Am 26.01.2016 um 02:46 schrieb Han, Huaitong: > What is the host kernel version and host dmesg information? And does > the problem exist when you use latest kernel and QEMU to replace old > binary file? Guest and Host is both 4.1.15. You mean the complete dmesg output from host? What do you mean with replace old binary file? I haven't tested Kernel 4.4 as we use 4.1 as it is a long term stable kernel release. Stefan > On Mon, 2016-01-25 at 14:51 +0100, Stefan Priebe - Profihost AG wrote: >> Hi, >> >> while running qemu 2.4 on whestmere CPUs i'm pretty often getting >> this >> one while booting: >> [0.811645] Switched APIC routing to physical x2apic. >> [1.835678] [ cut here ] >> [1.835704] WARNING: CPU: 0 PID: 1 at >> arch/x86/kernel/apic/apic.c:1309 setup_local_APIC+0x284/0x340() >> [1.835714] Modules linked in: >> [1.835724] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.15+72-ph >> #1 >> [1.835731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >> BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 >> [1.835743] b69ffcea 88042d5e3d68 b669c37b >> 0918 >> [1.835754] 88042d5e3da8 b6080d67 >> 88042d5e3da8 >> [1.835765] 0001 8000 >> >> [1.835777] Call Trace: >> [1.835789] [] dump_stack+0x45/0x57 >> [1.835799] [] warn_slowpath_common+0x97/0xe0 >> [1.835806] [] warn_slowpath_null+0x1a/0x20 >> [1.835813] [] setup_local_APIC+0x284/0x340 >> [1.835824] [] apic_bsp_setup+0x5b/0xb0 >> [1.835832] [] >> native_smp_prepare_cpus+0x23b/0x295 >> [1.835842] [] kernel_init_freeable+0xc7/0x20f >> [1.835853] [] ? rest_init+0x80/0x80 >> [1.835860] [] kernel_init+0xe/0xf0 >> [1.835870] [] ret_from_fork+0x42/0x70 >> [1.835877] [] ? rest_init+0x80/0x80 >> [1.835891] ---[ end trace bdbe630a8de2832c ]--- >> [1.837613] Spurious LAPIC timer interrupt on cpu 0 >> [1.837957] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 >> [1.939574] smpboot: CPU0: Intel Westmere E56xx/L56xx/X56xx >> (Nehalem-C) (fam: 06, model: 2c, stepping: 01) >> [1.939630] Performance Events: unsupported p6 CPU model 44 no PMU >> driver, software events only. >> [1.950868] KVM setup paravirtual spinlock >> >> Greets, >> Stefan >> -- >> To unsubscribe from this list: send the line "unsubscribe kvm" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html
[Qemu-devel] trace in arch/x86/kernel/apic/apic.c:1309 setup_local_APIC
Hi, while running qemu 2.4 on whestmere CPUs i'm pretty often getting this one while booting: [0.811645] Switched APIC routing to physical x2apic. [1.835678] [ cut here ] [1.835704] WARNING: CPU: 0 PID: 1 at arch/x86/kernel/apic/apic.c:1309 setup_local_APIC+0x284/0x340() [1.835714] Modules linked in: [1.835724] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.15+72-ph #1 [1.835731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 [1.835743] b69ffcea 88042d5e3d68 b669c37b 0918 [1.835754] 88042d5e3da8 b6080d67 88042d5e3da8 [1.835765] 0001 8000 [1.835777] Call Trace: [1.835789] [] dump_stack+0x45/0x57 [1.835799] [] warn_slowpath_common+0x97/0xe0 [1.835806] [] warn_slowpath_null+0x1a/0x20 [1.835813] [] setup_local_APIC+0x284/0x340 [1.835824] [] apic_bsp_setup+0x5b/0xb0 [1.835832] [] native_smp_prepare_cpus+0x23b/0x295 [1.835842] [] kernel_init_freeable+0xc7/0x20f [1.835853] [] ? rest_init+0x80/0x80 [1.835860] [] kernel_init+0xe/0xf0 [1.835870] [] ret_from_fork+0x42/0x70 [1.835877] [] ? rest_init+0x80/0x80 [1.835891] ---[ end trace bdbe630a8de2832c ]--- [1.837613] Spurious LAPIC timer interrupt on cpu 0 [1.837957] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [1.939574] smpboot: CPU0: Intel Westmere E56xx/L56xx/X56xx (Nehalem-C) (fam: 06, model: 2c, stepping: 01) [1.939630] Performance Events: unsupported p6 CPU model 44 no PMU driver, software events only. [1.950868] KVM setup paravirtual spinlock Greets, Stefan
Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever
> - Original Message - >> From: "Alexandre DERUMIER" >> To: "ceph-devel" >> Cc: "qemu-devel" , jdur...@redhat.com >> Sent: Monday, November 9, 2015 5:48:45 AM >> Subject: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and >> vm_stop is hanging forever >> >> adding to ceph.conf >> >> [client] >> rbd_non_blocking_aio = false >> >> >> fix the problem for me (with rbd_cache=false) >> >> >> (@cc jdur...@redhat.com) +1 same to me. Stefan >> >> >> >> - Mail original - >> De: "Denis V. Lunev" >> À: "aderumier" , "ceph-devel" >> , "qemu-devel" >> Envoyé: Lundi 9 Novembre 2015 08:22:34 >> Objet: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop >> is hanging forever >> >> On 11/09/2015 10:19 AM, Denis V. Lunev wrote: >>> On 11/09/2015 06:10 AM, Alexandre DERUMIER wrote: Hi, with qemu (2.4.1), if I do an internal snapshot of an rbd device, then I pause the vm with vm_stop, the qemu process is hanging forever monitor commands to reproduce: # snapshot_blkdev_internal drive-virtio0 yoursnapname # stop I don't see this with qcow2 or sheepdog block driver for example. Regards, Alexandre >>> this could look like the problem I have recenty trying to >>> fix with dataplane enabled. Patch series is named as >>> >>> [PATCH for 2.5 v6 0/10] dataplane snapshot fixes >>> >>> Den >> >> anyway, even if above will not help, can you collect gdb >> traces from all threads in QEMU process. May be I'll be >> able to give a hit. >> >> Den >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >
Re: [Qemu-devel] guest: NO_HZ FULL will not work with unstable sched clock
same with kvm clock: [0.195073] Switched to clocksource kvm-clock [0.195509] [ cut here ] [0.195509] WARNING: CPU: 1 PID: 1 at kernel/time/tick-sched.c:192 can_stop_full_tick+0x1c1/0x200() [0.195515] NO_HZ FULL will not work with unstable sched clock [0.195518] Modules linked in: [0.195531] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.1.5+11-ph #1 [0.195535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.1-0-g4adadbd-20150316_085822-nilsson.home.kraxel.org 04/01/2014 [0.195544] 819fc200 88023fc43e88 8162dd36 121a [0.195551] 88023fc43ed8 88023fc43ec8 8107dcb7 88023fc43ea8 [0.195558] 0001 81e667c0 81cf1540 [0.195564] Call Trace: [0.195568][] dump_stack+0x45/0x57 [0.195591] [] warn_slowpath_common+0x97/0xe0 [0.195596] [] warn_slowpath_fmt+0x46/0x50 [0.195600] [] can_stop_full_tick+0x1c1/0x200 [0.195604] [] tick_nohz_irq_exit+0x97/0xd0 [0.195609] [] irq_exit+0xcd/0x130 [0.195620] [] smp_apic_timer_interrupt+0x4a/0x60 [0.195627] [] apic_timer_interrupt+0x6e/0x80 [0.195631][] ? trace_event_enum_update+0x1c7/0x3b0 [0.195644] [] ? trace_event_enum_update+0x20c/0x3b0 [0.195649] [] ? stop_trace_on_warning+0x3f/0x3f [0.195654] [] tracer_init_tracefs+0x11a/0x23b [0.195661] [] do_one_initcall+0xd8/0x210 [0.195669] [] ? parse_args+0x1b0/0x450 [0.195676] [] ? __wake_up+0x53/0x70 [0.195684] [] kernel_init_freeable+0x178/0x20f [0.195689] [] ? do_early_param+0x8c/0x8c [0.195700] [] ? create_pipe_files+0x1f2/0x200 [0.195706] [] ? rest_init+0x80/0x80 [0.195711] [] kernel_init+0xe/0xf0 [0.195715] [] ret_from_fork+0x42/0x70 [0.195719] [] ? rest_init+0x80/0x80 [0.195729] ---[ end trace cf665146248feec1 ]--- Stefan Am 15.08.2015 um 20:44 schrieb Stefan Priebe: Hi, while switching to a FULL tickless kernel i detected that all our VMs produce the following stack trace while running under qemu 2.3.0. [0.195160] HPET: 3 timers in total, 0 timers will be used for per-cpu timer [0.195181] hpet0: at MMIO 0xfed0, IRQs 2, 8, 0 [0.195188] hpet0: 3 comparators, 64-bit 100.00 MHz counter [0.200119] Switched to clocksource hpet [0.201164] [ cut here ] [0.201193] WARNING: CPU: 1 PID: 1 at kernel/time/tick-sched.c:192 can_stop_full_tick+0x1c1/0x200() [0.201199] NO_HZ FULL will not work with unstable sched clock [0.201203] Modules linked in: [0.201214] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 3.18.20+55-ph #1 [0.201219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.1-0-g4adadbd-20150316_085822-nilsson.home.kraxel.org 04/01/2014 [0.201226] 0009 88023fc43e88 815f5889 08df [0.201230] 88023fc43ed8 88023fc43ec8 8107433c 88023fc43ea8 [0.201233] 0001 880236c41300 cbbde78b [0.201237] Call Trace: [0.201240][] dump_stack+0x46/0x58 [0.201260] [] warn_slowpath_common+0x8c/0xc0 [0.201264] [] warn_slowpath_fmt+0x46/0x50 [0.201270] [] can_stop_full_tick+0x1c1/0x200 [0.201274] [] tick_nohz_irq_exit+0x97/0xd0 [0.201277] [] irq_exit+0xcd/0x130 [0.201286] [] smp_apic_timer_interrupt+0x4a/0x60 [0.201293] [] apic_timer_interrupt+0x6a/0x70 [0.201296][] ? __d_lookup+0x55/0x160 [0.201309] [] d_lookup+0x35/0x60 [0.201312] [] lookup_dcache+0x3b/0xd0 [0.201315] [] __lookup_hash+0x23/0x50 [0.201319] [] lookup_one_len+0xce/0x120 [0.201328] [] __create_file+0x8f/0x260 [0.201332] [] debugfs_create_dir+0x1b/0x20 [0.201338] [] event_create_dir+0x68/0x520 [0.201342] [] event_trace_init+0x188/0x210 [0.201348] [] ? trace_create_file+0x1b/0x50 [0.201352] [] ? event_trace_enable+0x1a2/0x1a2 [0.201359] [] do_one_initcall+0xd8/0x200 [0.201368] [] ? parse_args+0x1f0/0x470 [0.201373] [] ? __wake_up+0x53/0x70 [0.201378] [] kernel_init_freeable+0x168/0x1ff [0.201381] [] ? do_early_param+0x8c/0x8c [0.201389] [] ? cryptomgr_probe+0x60/0xe0 [0.201398] [] ? rest_init+0x80/0x80 [0.201401] [] kernel_init+0xe/0xf0 [0.201405] [] ret_from_fork+0x58/0x90 [0.201408] [] ? rest_init+0x80/0x80 [0.201415] ---[ end trace a15ba3bc1d247b4c ]--- Stefan
[Qemu-devel] guest: NO_HZ FULL will not work with unstable sched clock
Hi, while switching to a FULL tickless kernel i detected that all our VMs produce the following stack trace while running under qemu 2.3.0. [0.195160] HPET: 3 timers in total, 0 timers will be used for per-cpu timer [0.195181] hpet0: at MMIO 0xfed0, IRQs 2, 8, 0 [0.195188] hpet0: 3 comparators, 64-bit 100.00 MHz counter [0.200119] Switched to clocksource hpet [0.201164] [ cut here ] [0.201193] WARNING: CPU: 1 PID: 1 at kernel/time/tick-sched.c:192 can_stop_full_tick+0x1c1/0x200() [0.201199] NO_HZ FULL will not work with unstable sched clock [0.201203] Modules linked in: [0.201214] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 3.18.20+55-ph #1 [0.201219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.1-0-g4adadbd-20150316_085822-nilsson.home.kraxel.org 04/01/2014 [0.201226] 0009 88023fc43e88 815f5889 08df [0.201230] 88023fc43ed8 88023fc43ec8 8107433c 88023fc43ea8 [0.201233] 0001 880236c41300 cbbde78b [0.201237] Call Trace: [0.201240][] dump_stack+0x46/0x58 [0.201260] [] warn_slowpath_common+0x8c/0xc0 [0.201264] [] warn_slowpath_fmt+0x46/0x50 [0.201270] [] can_stop_full_tick+0x1c1/0x200 [0.201274] [] tick_nohz_irq_exit+0x97/0xd0 [0.201277] [] irq_exit+0xcd/0x130 [0.201286] [] smp_apic_timer_interrupt+0x4a/0x60 [0.201293] [] apic_timer_interrupt+0x6a/0x70 [0.201296][] ? __d_lookup+0x55/0x160 [0.201309] [] d_lookup+0x35/0x60 [0.201312] [] lookup_dcache+0x3b/0xd0 [0.201315] [] __lookup_hash+0x23/0x50 [0.201319] [] lookup_one_len+0xce/0x120 [0.201328] [] __create_file+0x8f/0x260 [0.201332] [] debugfs_create_dir+0x1b/0x20 [0.201338] [] event_create_dir+0x68/0x520 [0.201342] [] event_trace_init+0x188/0x210 [0.201348] [] ? trace_create_file+0x1b/0x50 [0.201352] [] ? event_trace_enable+0x1a2/0x1a2 [0.201359] [] do_one_initcall+0xd8/0x200 [0.201368] [] ? parse_args+0x1f0/0x470 [0.201373] [] ? __wake_up+0x53/0x70 [0.201378] [] kernel_init_freeable+0x168/0x1ff [0.201381] [] ? do_early_param+0x8c/0x8c [0.201389] [] ? cryptomgr_probe+0x60/0xe0 [0.201398] [] ? rest_init+0x80/0x80 [0.201401] [] kernel_init+0xe/0xf0 [0.201405] [] ret_from_fork+0x58/0x90 [0.201408] [] ? rest_init+0x80/0x80 [0.201415] ---[ end trace a15ba3bc1d247b4c ]--- Stefan
Re: [Qemu-devel] [Qemu-stable] [PULL 0/3] Cve 2015 5154 patches
Am 27.07.2015 um 14:28 schrieb John Snow: > > > On 07/27/2015 08:10 AM, Stefan Priebe - Profihost AG wrote: >> >> Am 27.07.2015 um 14:01 schrieb John Snow: >>> The following changes since commit f793d97e454a56d17e404004867985622ca1a63b: >>> >>> Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into >>> staging (2015-07-24 13:07:10 +0100) >>> >>> are available in the git repository at: >>> >>> https://github.com/jnsnow/qemu.git tags/cve-2015-5154-pull-request >> >> Any details on this CVE? Is RCE possible? Only if IDE is used? >> >> Stefan >> > > It's a heap overflow. The most likely outcome is a segfault, but the > guest is allowed to continue writing past the end of the PIO buffer at > its leisure. This makes it similar to CVE-2015-3456. > > This CVE can be mitigated unlike CVE-2015-3456 by just removing the > CD-ROM drive until the patch can be applied. Thanks. The seclist article explicitly references xen. So it does not apply to qemu/kvm? Sorry for asking may be stupid questions. Stefan >>> for you to fetch changes up to cb72cba83021fa42719e73a5249c12096a4d1cfc: >>> >>> ide: Clear DRQ after handling all expected accesses (2015-07-26 23:42:53 >>> -0400) >>> >>> >>> >>> >>> >>> Kevin Wolf (3): >>> ide: Check array bounds before writing to io_buffer (CVE-2015-5154) >>> ide/atapi: Fix START STOP UNIT command completion >>> ide: Clear DRQ after handling all expected accesses >>> >>> hw/ide/atapi.c | 1 + >>> hw/ide/core.c | 32 >>> 2 files changed, 29 insertions(+), 4 deletions(-) >>>
Re: [Qemu-devel] [Qemu-stable] [PULL 0/3] Cve 2015 5154 patches
Am 27.07.2015 um 14:01 schrieb John Snow: > The following changes since commit f793d97e454a56d17e404004867985622ca1a63b: > > Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into > staging (2015-07-24 13:07:10 +0100) > > are available in the git repository at: > > https://github.com/jnsnow/qemu.git tags/cve-2015-5154-pull-request Any details on this CVE? Is RCE possible? Only if IDE is used? Stefan > for you to fetch changes up to cb72cba83021fa42719e73a5249c12096a4d1cfc: > > ide: Clear DRQ after handling all expected accesses (2015-07-26 23:42:53 > -0400) > > > > > > Kevin Wolf (3): > ide: Check array bounds before writing to io_buffer (CVE-2015-5154) > ide/atapi: Fix START STOP UNIT command completion > ide: Clear DRQ after handling all expected accesses > > hw/ide/atapi.c | 1 + > hw/ide/core.c | 32 > 2 files changed, 29 insertions(+), 4 deletions(-) >
Re: [Qemu-devel] Query CPU model / type
Am 15.07.2015 um 22:15 schrieb Andrey Korolyov: On Wed, Jul 15, 2015 at 11:07 PM, Stefan Priebe wrote: Am 15.07.2015 um 13:32 schrieb Andrey Korolyov: On Wed, Jul 15, 2015 at 2:20 PM, Stefan Priebe - Profihost AG wrote: Hi, is there a way to query the current cpu model / type of a running qemu machine? I mean host, kvm64, qemu64, ... Stefan I believe that the most proper one would be 'query-command-line-options'. The CPU definitions are not exposed anywhere else in QMP except their list. But this one only lists the possible arguments not the currently passed ones? Stefan Sorry, my bad, wrong memories. So there is no way to retrieve arguments via monitor. and even no current cpu model ;-( pretty bad for migrations. So i'll have to hack the qemu code. Stefan
Re: [Qemu-devel] Query CPU model / type
Am 15.07.2015 um 13:32 schrieb Andrey Korolyov: On Wed, Jul 15, 2015 at 2:20 PM, Stefan Priebe - Profihost AG wrote: Hi, is there a way to query the current cpu model / type of a running qemu machine? I mean host, kvm64, qemu64, ... Stefan I believe that the most proper one would be 'query-command-line-options'. The CPU definitions are not exposed anywhere else in QMP except their list. But this one only lists the possible arguments not the currently passed ones? Stefan
[Qemu-devel] Query CPU model / type
Hi, is there a way to query the current cpu model / type of a running qemu machine? I mean host, kvm64, qemu64, ... Stefan
Re: [Qemu-devel] [Qemu-stable] [PATCH] fdc: force the fifo access to be in bounds of the allocated buffer
Am 13.05.2015 um 21:05 schrieb Stefan Weil: Am 13.05.2015 um 20:59 schrieb Stefan Priebe: Am 13.05.2015 um 20:51 schrieb Stefan Weil: Hi, I just noticed this patch because my provider told me that my KVM based server needs a reboot because of a CVE (see this German news: http://www.heise.de/newsticker/meldung/Venom-Schwachstelle-Aus-Hypervisor-ausbrechen-und-VMs-ausspionieren-2649614.html) Isn't a live migration to a fixed version enough instead of a reboot? Stefan Good point. A live migration would be sufficient - if there are no bugs in QEMU's live migration. just migrating all our customer machines and wanted to be sure that live migration is enough. Greets, Stefan Stefan
Re: [Qemu-devel] [Qemu-stable] [PATCH] fdc: force the fifo access to be in bounds of the allocated buffer
Am 13.05.2015 um 21:04 schrieb John Snow: On 05/13/2015 02:59 PM, Stefan Priebe wrote: Am 13.05.2015 um 20:51 schrieb Stefan Weil: Hi, I just noticed this patch because my provider told me that my KVM based server needs a reboot because of a CVE (see this German news: http://www.heise.de/newsticker/meldung/Venom-Schwachstelle-Aus-Hypervisor-ausbrechen-und-VMs-ausspionieren-2649614.html) Isn't a live migration to a fixed version enough instead of a reboot? Stefan If your management API or host or whatever lets you migrate back to the same host, or has another host they can migrate you to, yes. i'm my host ;-) I was just interested if for whatever reason live migration is not enough and was wondering why someone wants to reboot a vm for something like this. Thanks! Stefan Am 13.05.2015 um 16:33 schrieb John Snow: From: Petr Matousek During processing of certain commands such as FD_CMD_READ_ID and FD_CMD_DRIVE_SPECIFICATION_COMMAND the fifo memory access could get out of bounds leading to memory corruption with values coming from the guest. Fix this by making sure that the index is always bounded by the allocated memory. This is CVE-2015-3456. Signed-off-by: Petr Matousek Reviewed-by: John Snow Signed-off-by: John Snow --- hw/block/fdc.c | 17 +++-- 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/hw/block/fdc.c b/hw/block/fdc.c index f72a392..d8a8edd 100644 --- a/hw/block/fdc.c +++ b/hw/block/fdc.c @@ -1497,7 +1497,7 @@ static uint32_t fdctrl_read_data(FDCtrl *fdctrl) { FDrive *cur_drv; uint32_t retval = 0; -int pos; +uint32_t pos; cur_drv = get_cur_drv(fdctrl); fdctrl->dsr &= ~FD_DSR_PWRDOWN; @@ -1506,8 +1506,8 @@ static uint32_t fdctrl_read_data(FDCtrl *fdctrl) return 0; } pos = fdctrl->data_pos; +pos %= FD_SECTOR_LEN; I'd combine both statements and perhaps use fdctrl->fifo_size (even if the resulting code will be slightly larger): pos = fdctrl->data_pos % fdctrl->fifo_size; if (fdctrl->msr & FD_MSR_NONDMA) { -pos %= FD_SECTOR_LEN; if (pos == 0) { if (fdctrl->data_pos != 0) if (!fdctrl_seek_to_next_sect(fdctrl, cur_drv)) { @@ -1852,10 +1852,13 @@ static void fdctrl_handle_option(FDCtrl *fdctrl, int direction) static void fdctrl_handle_drive_specification_command(FDCtrl *fdctrl, int direction) { FDrive *cur_drv = get_cur_drv(fdctrl); +uint32_t pos; -if (fdctrl->fifo[fdctrl->data_pos - 1] & 0x80) { +pos = fdctrl->data_pos - 1; +pos %= FD_SECTOR_LEN; Shorter (and more clear): uint32_t pos = (fdctrl->data_pos - 1) % fdctrl->fifo_size; +if (fdctrl->fifo[pos] & 0x80) { /* Command parameters done */ -if (fdctrl->fifo[fdctrl->data_pos - 1] & 0x40) { +if (fdctrl->fifo[pos] & 0x40) { fdctrl->fifo[0] = fdctrl->fifo[1]; fdctrl->fifo[2] = 0; fdctrl->fifo[3] = 0; @@ -1955,7 +1958,7 @@ static uint8_t command_to_handler[256]; static void fdctrl_write_data(FDCtrl *fdctrl, uint32_t value) { FDrive *cur_drv; -int pos; +uint32_t pos; /* Reset mode */ if (!(fdctrl->dor & FD_DOR_nRESET)) { @@ -2004,7 +2007,9 @@ static void fdctrl_write_data(FDCtrl *fdctrl, uint32_t value) } FLOPPY_DPRINTF("%s: %02x\n", __func__, value); -fdctrl->fifo[fdctrl->data_pos++] = value; +pos = fdctrl->data_pos++; +pos %= FD_SECTOR_LEN; +fdctrl->fifo[pos] = value; if (fdctrl->data_pos == fdctrl->data_len) { /* We now have all parameters * and will be able to treat the command Not strictly related to this patch: The code which sets fifo_size could also be improved. fdctrl->fifo = qemu_memalign(512, FD_SECTOR_LEN); fdctrl->fifo_size = 512; The 2nd line should be fdctrl->fifo_size = FD_SECTOR_LEN; As far as I see the original code can read or write illegal memory locations in the address space of the QEMU process. It cannot (as it was claimed) modify the code of the VM host because those memory is usually write protected - at least if QEMU is running without KVM. If the code which is generated for KVM is writable from anywhere in QEMU, we should perhaps consider changing that. Regards Stefan
Re: [Qemu-devel] [Qemu-stable] [PATCH] fdc: force the fifo access to be in bounds of the allocated buffer
Am 13.05.2015 um 20:51 schrieb Stefan Weil: Hi, I just noticed this patch because my provider told me that my KVM based server needs a reboot because of a CVE (see this German news: http://www.heise.de/newsticker/meldung/Venom-Schwachstelle-Aus-Hypervisor-ausbrechen-und-VMs-ausspionieren-2649614.html) Isn't a live migration to a fixed version enough instead of a reboot? Stefan Am 13.05.2015 um 16:33 schrieb John Snow: From: Petr Matousek During processing of certain commands such as FD_CMD_READ_ID and FD_CMD_DRIVE_SPECIFICATION_COMMAND the fifo memory access could get out of bounds leading to memory corruption with values coming from the guest. Fix this by making sure that the index is always bounded by the allocated memory. This is CVE-2015-3456. Signed-off-by: Petr Matousek Reviewed-by: John Snow Signed-off-by: John Snow --- hw/block/fdc.c | 17 +++-- 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/hw/block/fdc.c b/hw/block/fdc.c index f72a392..d8a8edd 100644 --- a/hw/block/fdc.c +++ b/hw/block/fdc.c @@ -1497,7 +1497,7 @@ static uint32_t fdctrl_read_data(FDCtrl *fdctrl) { FDrive *cur_drv; uint32_t retval = 0; -int pos; +uint32_t pos; cur_drv = get_cur_drv(fdctrl); fdctrl->dsr &= ~FD_DSR_PWRDOWN; @@ -1506,8 +1506,8 @@ static uint32_t fdctrl_read_data(FDCtrl *fdctrl) return 0; } pos = fdctrl->data_pos; +pos %= FD_SECTOR_LEN; I'd combine both statements and perhaps use fdctrl->fifo_size (even if the resulting code will be slightly larger): pos = fdctrl->data_pos % fdctrl->fifo_size; if (fdctrl->msr & FD_MSR_NONDMA) { -pos %= FD_SECTOR_LEN; if (pos == 0) { if (fdctrl->data_pos != 0) if (!fdctrl_seek_to_next_sect(fdctrl, cur_drv)) { @@ -1852,10 +1852,13 @@ static void fdctrl_handle_option(FDCtrl *fdctrl, int direction) static void fdctrl_handle_drive_specification_command(FDCtrl *fdctrl, int direction) { FDrive *cur_drv = get_cur_drv(fdctrl); +uint32_t pos; -if (fdctrl->fifo[fdctrl->data_pos - 1] & 0x80) { +pos = fdctrl->data_pos - 1; +pos %= FD_SECTOR_LEN; Shorter (and more clear): uint32_t pos = (fdctrl->data_pos - 1) % fdctrl->fifo_size; +if (fdctrl->fifo[pos] & 0x80) { /* Command parameters done */ -if (fdctrl->fifo[fdctrl->data_pos - 1] & 0x40) { +if (fdctrl->fifo[pos] & 0x40) { fdctrl->fifo[0] = fdctrl->fifo[1]; fdctrl->fifo[2] = 0; fdctrl->fifo[3] = 0; @@ -1955,7 +1958,7 @@ static uint8_t command_to_handler[256]; static void fdctrl_write_data(FDCtrl *fdctrl, uint32_t value) { FDrive *cur_drv; -int pos; +uint32_t pos; /* Reset mode */ if (!(fdctrl->dor & FD_DOR_nRESET)) { @@ -2004,7 +2007,9 @@ static void fdctrl_write_data(FDCtrl *fdctrl, uint32_t value) } FLOPPY_DPRINTF("%s: %02x\n", __func__, value); -fdctrl->fifo[fdctrl->data_pos++] = value; +pos = fdctrl->data_pos++; +pos %= FD_SECTOR_LEN; +fdctrl->fifo[pos] = value; if (fdctrl->data_pos == fdctrl->data_len) { /* We now have all parameters * and will be able to treat the command Not strictly related to this patch: The code which sets fifo_size could also be improved. fdctrl->fifo = qemu_memalign(512, FD_SECTOR_LEN); fdctrl->fifo_size = 512; The 2nd line should be fdctrl->fifo_size = FD_SECTOR_LEN; As far as I see the original code can read or write illegal memory locations in the address space of the QEMU process. It cannot (as it was claimed) modify the code of the VM host because those memory is usually write protected - at least if QEMU is running without KVM. If the code which is generated for KVM is writable from anywhere in QEMU, we should perhaps consider changing that. Regards Stefan
Re: [Qemu-devel] Qemu 2.2.1 black screen of death in windows 2012 R2
Hi, it started to work again with virtio 100 instead of 94. No idea why it works with qemu 2.2.0. Stefan Am 24.03.2015 um 12:15 schrieb Stefan Priebe - Profihost AG: Am 24.03.2015 um 11:45 schrieb Paolo Bonzini: On 24/03/2015 11:39, Stefan Priebe - Profihost AG wrote: after upgrading Qemu from 2.2.0 to 2.2.1 Windows 2012 R2 works after installing. But after applying 72 updates it breaks with a black screen of death. Can you bisect it? Have to try might be possible. Linking to this KB: https://support.microsoft.com/en-us/kb/2939259 That KB, I think, refers to running a hypervisor (e.g. VMware Workstation) _inside_ Windows. Screenshot of the windows screen attached. This is just a blank windows 2012 R2 with virtio 94 drivers installed. Stefan
Re: [Qemu-devel] Qemu 2.2.1 black screen of death in windows 2012 R2
Am 24.03.2015 um 11:45 schrieb Paolo Bonzini: > > > On 24/03/2015 11:39, Stefan Priebe - Profihost AG wrote: >> after upgrading Qemu from 2.2.0 to 2.2.1 >> >> Windows 2012 R2 works after installing. But after applying 72 updates it >> breaks with a black screen of death. > > Can you bisect it? Have to try might be possible. >> Linking to this KB: >> https://support.microsoft.com/en-us/kb/2939259 > > That KB, I think, refers to running a hypervisor (e.g. VMware > Workstation) _inside_ Windows. Screenshot of the windows screen attached. This is just a blank windows 2012 R2 with virtio 94 drivers installed. Stefan
[Qemu-devel] Qemu 2.2.1 black screen of death in windows 2012 R2
Hi, after upgrading Qemu from 2.2.0 to 2.2.1 Windows 2012 R2 works after installing. But after applying 72 updates it breaks with a black screen of death. Linking to this KB: https://support.microsoft.com/en-us/kb/2939259 It works fine with Qemu 2.2.0 Greets, Stefan
Re: [Qemu-devel] live migration fails after host kernel upgrade (3.12 => 3.18)
Hi, Thanks. I fixed it - there is already a patchseries in 4.0 to fix this. It will be backported in 3.18.10 or 3.18.11. Stefan Am 23.03.2015 um 12:54 schrieb Stefan Hajnoczi: > On Sun, Mar 15, 2015 at 09:59:25AM +0100, Stefan Priebe wrote: >> after upgrading the host kernel from 3.12 to 3.18 live migration fails with >> the following qemu output (guest running on a host with 3.12 => host with >> 3.18): >> >> kvm: Features 0x30afffe3 unsupported. Allowed features: 0x79bfbbe7 >> qemu: warning: error while loading state for instance 0x0 of device >> ':00:12.0/virtio-net' >> kvm: load of migration failed: Operation not permitted >> >> But i can't > > Hi Stefan, > I haven't checked the exact feature bits but it might be the UFO feature > problem. The tun driver was changed to drop a feature, this broke live > migration. > > Take a look at > https://lists.linuxfoundation.org/pipermail/virtualization/2015-February/029217.html > > Stefan >
[Qemu-devel] live migration fails after host kernel upgrade (3.12 => 3.18)
Hi, after upgrading the host kernel from 3.12 to 3.18 live migration fails with the following qemu output (guest running on a host with 3.12 => host with 3.18): kvm: Features 0x30afffe3 unsupported. Allowed features: 0x79bfbbe7 qemu: warning: error while loading state for instance 0x0 of device ':00:12.0/virtio-net' kvm: load of migration failed: Operation not permitted But i can't Greets, Stefan
Re: [Qemu-devel] slow speed for virtio-scsi since qemu 2.2
Am 16.02.2015 um 16:50 schrieb Andreas Färber: Am 16.02.2015 um 16:41 schrieb Stefan Priebe - Profihost AG: Am 16.02.2015 um 15:49 schrieb Paolo Bonzini : On 16/02/2015 15:47, Stefan Priebe - Profihost AG wrote: Could it be that this is a results of compiling qemu with --enable-debug to get debugging symbols? Yes. *urg* my fault - sorry. Is there a way to get debugging symbols without using enable-debug / getting slower performance? Yes, just do nothing ("--enable-debug-info" is the default; "--enable-debug" enables debug info _and_ turns off optimization). If I do not enable-debug dh_strip does not extract any debugging symbols. Maybe you forgot --disable-strip? Thanks! Stefan Regards, Andreas
Re: [Qemu-devel] slow speed for virtio-scsi since qemu 2.2
Am 16.02.2015 um 16:49 schrieb Paolo Bonzini: On 16/02/2015 16:41, Stefan Priebe - Profihost AG wrote: Yes, just do nothing ("--enable-debug-info" is the default; "--enable-debug" enables debug info _and_ turns off optimization). If I do not enable-debug dh_strip does not extract any debugging symbols. Then you need --disable-strip too. Thanks! Stefan Paolo
Re: [Qemu-devel] slow speed for virtio-scsi since qemu 2.2
> Am 16.02.2015 um 15:49 schrieb Paolo Bonzini : > > > > On 16/02/2015 15:47, Stefan Priebe - Profihost AG wrote: >>>>>> >>>>>> Could it be that this is a results of compiling qemu with --enable-debug >>>>>> to get debugging symbols? >>>> >>>> Yes. >> *urg* my fault - sorry. Is there a way to get debugging symbols without >> using enable-debug / getting slower performance? > > Yes, just do nothing ("--enable-debug-info" is the default; > "--enable-debug" enables debug info _and_ turns off optimization). If I do not enable-debug dh_strip does not extract any debugging symbols. Stefan > > Paolo
Re: [Qemu-devel] slow speed for virtio-scsi since qemu 2.2
Hi, > Am 16.02.2015 um 15:58 schrieb Andrey Korolyov : > > On Mon, Feb 16, 2015 at 5:47 PM, Stefan Priebe - Profihost AG > wrote: >> Hi, >>> Am 16.02.2015 um 15:44 schrieb Paolo Bonzini: >>> >>> >>>> On 16/02/2015 15:43, Stefan Priebe - Profihost AG wrote: >>>> Hi, >>>> >>>>> Am 16.02.2015 um 13:24 schrieb Paolo Bonzini: >>>>> >>>>> >>>>>> On 15/02/2015 19:46, Stefan Priebe wrote: >>>>>> >>>>>> while i get a constant random 4k i/o write speed of 20.000 iops with >>>>>> qemu 2.1.0 or 2.1.3. I get jumping speeds with qemu 2.2 (jumping between >>>>>> 500 iops and 15.000 iop/s). >>>>>> >>>>>> If i use virtio instead of virtio-scsi speed is the same between 2.2 and >>>>>> 2.1. >>>>> >>>>> http://wiki.qemu.org/Contribute/ReportABug >>>>> >>>>> What is your command line? >>>> >>>> Could it be that this is a results of compiling qemu with --enable-debug >>>> to get debugging symbols? >>> >>> Yes. >> >> *urg* my fault - sorry. Is there a way to get debugging symbols without >> using enable-debug / getting slower performance? > > Hi Stefan, splitdebug will help you there. Also this is a standard way > to ship debugging symbols in most distribuitions, so I wonder if you > may use just a generic build skeleton for this task... I'm using enable-debug with dh_strip from Debian Stefan
Re: [Qemu-devel] slow speed for virtio-scsi since qemu 2.2
Hi, Am 16.02.2015 um 15:44 schrieb Paolo Bonzini: > > > On 16/02/2015 15:43, Stefan Priebe - Profihost AG wrote: >> Hi, >> >> Am 16.02.2015 um 13:24 schrieb Paolo Bonzini: >>> >>> >>> On 15/02/2015 19:46, Stefan Priebe wrote: >>>> >>>> while i get a constant random 4k i/o write speed of 20.000 iops with >>>> qemu 2.1.0 or 2.1.3. I get jumping speeds with qemu 2.2 (jumping between >>>> 500 iops and 15.000 iop/s). >>>> >>>> If i use virtio instead of virtio-scsi speed is the same between 2.2 and >>>> 2.1. >>> >>> http://wiki.qemu.org/Contribute/ReportABug >>> >>> What is your command line? >> >> Could it be that this is a results of compiling qemu with --enable-debug >> to get debugging symbols? > > Yes. *urg* my fault - sorry. Is there a way to get debugging symbols without using enable-debug / getting slower performance? Greets, Stefan
Re: [Qemu-devel] slow speed for virtio-scsi since qemu 2.2
Hi, Am 16.02.2015 um 13:24 schrieb Paolo Bonzini: > > > On 15/02/2015 19:46, Stefan Priebe wrote: >> >> while i get a constant random 4k i/o write speed of 20.000 iops with >> qemu 2.1.0 or 2.1.3. I get jumping speeds with qemu 2.2 (jumping between >> 500 iops and 15.000 iop/s). >> >> If i use virtio instead of virtio-scsi speed is the same between 2.2 and >> 2.1. > > http://wiki.qemu.org/Contribute/ReportABug > > What is your command line? > > Paolo > Could it be that this is a results of compiling qemu with --enable-debug to get debugging symbols? Stefan
[Qemu-devel] slow speed for virtio-scsi since qemu 2.2
Hi, while i get a constant random 4k i/o write speed of 20.000 iops with qemu 2.1.0 or 2.1.3. I get jumping speeds with qemu 2.2 (jumping between 500 iops and 15.000 iop/s). If i use virtio instead of virtio-scsi speed is the same between 2.2 and 2.1. Is there a known regression? Greets, Stefan
[Qemu-devel] kvm segfault in iov_to_buf / virtio_scsi_parse_req
Hi, while migrating a bunch of VMs i saw multiple times segaults with qemu 2.1.2. Is this a known bug? Full backtrace: Program terminated with signal 11, Segmentation fault. #0 0x7ff9c73bca90 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x7ff9c73bca90 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x7ff9cde46b8e in iov_to_buf (iov=0x7ff9b805c0b0, iov_cnt=1, offset=0, buf=0x7ff9b8060118, bytes=51) at util/iov.c:49 #2 0x7ff9cdacdc56 in virtio_scsi_parse_req (req=0x7ff9b8054090, req_size=51, resp_size=108) at qemu-kvm/hw/scsi/virtio-scsi.c:140 #3 0x7ff9cdace060 in virtio_scsi_load_request (f=0x7ff9b803b620, sreq=0x7ff9b8089550) at qemu-kvm/hw/scsi/virtio-scsi.c:215 #4 0x7ff9cdcf5737 in get_scsi_requests (f=0x7ff9b803b620, pv=0x7ff9cfe4a440, size=0) at hw/scsi/scsi-bus.c:1875 #5 0x7ff9cdb76e0c in vmstate_load_state (f=0x7ff9b803b620, vmsd=0x7ff9ce24cac0, opaque=0x7ff9cfe4a440, version_id=1) at vmstate.c:105 #6 0x7ff9cdb76de8 in vmstate_load_state (f=0x7ff9b803b620, vmsd=0x7ff9ce24c500, opaque=0x7ff9cfe4a440, version_id=1) at vmstate.c:102 #7 0x7ff9cda935b6 in vmstate_load (f=0x7ff9b803b620, se=0x7ff9cfe95570, version_id=1) at qemu-kvm/savevm.c:572 #8 0x7ff9cda94652 in qemu_loadvm_state (f=0x7ff9b803b620) at qemu-kvm/savevm.c:951 #9 0x7ff9cdb748bd in process_incoming_migration_co (opaque=0x7ff9b803b620) at migration.c:97 #10 0x7ff9cdddb433 in coroutine_trampoline (i0=-1207852976, i1=32761) at coroutine-ucontext.c:118 #11 0x7ff9c72dbd10 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x7fff5e55a830 in ?? () #13 0x in ?? () Stefan
Re: [Qemu-devel] [PATCH] aio: fix qemu_bh_schedule() bh->ctx race condition
Tested-by: Stefan Priebe Am 03.06.2014 11:21, schrieb Stefan Hajnoczi: > qemu_bh_schedule() is supposed to be thread-safe at least the first time > it is called. Unfortunately this is not quite true: > > bh->scheduled = 1; > aio_notify(bh->ctx); > > Since another thread may run the BH callback once it has been scheduled, > there is a race condition if the callback frees the BH before > aio_notify(bh->ctx) has a chance to run. > > Reported-by: Stefan Priebe > Signed-off-by: Stefan Hajnoczi > --- > async.c | 14 ++ > 1 file changed, 10 insertions(+), 4 deletions(-) > > diff --git a/async.c b/async.c > index 6930185..5b6fe6b 100644 > --- a/async.c > +++ b/async.c > @@ -117,15 +117,21 @@ void qemu_bh_schedule_idle(QEMUBH *bh) > > void qemu_bh_schedule(QEMUBH *bh) > { > +AioContext *ctx; > + > if (bh->scheduled) > return; > +ctx = bh->ctx; > bh->idle = 0; > -/* Make sure that idle & any writes needed by the callback are done > - * before the locations are read in the aio_bh_poll. > +/* Make sure that: > + * 1. idle & any writes needed by the callback are done before the > + *locations are read in the aio_bh_poll. > + * 2. ctx is loaded before scheduled is set and the callback has a chance > + *to execute. > */ > -smp_wmb(); > +smp_mb(); > bh->scheduled = 1; > -aio_notify(bh->ctx); > +aio_notify(ctx); > } > > >
Re: [Qemu-devel] qemu 2.0 segfaults in event notifier
Am 02.06.2014 22:45, schrieb Paolo Bonzini: Il 02/06/2014 21:32, Stefan Priebe ha scritto: #0 0x7f69e421c43f in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 #1 0x7f69e3e37afc in aio_notify (ctx=0x0) at async.c:246 #2 0x7f69e3e37697 in qemu_bh_schedule (bh=0x7f5dac217f60) at async.c:128 #3 0x7f69e3e8bc44 in rbd_finish_aiocb (c=0x7f5dac0c3f30, rcb=0x7f5dafa50610) at block/rbd.c:585 #4 0x7f69e17bee44 in librbd::AioCompletion::complete() () from /usr/lib/librbd.so.1 #5 0x7f69e17be832 in librbd::AioCompletion::complete_request(CephContext*, long) () from /usr/lib/librbd.so.1 #6 0x7f69e1c946ba in Context::complete(int) () from /usr/lib/librados.so.2 #7 0x7f69e17f1e85 in ObjectCacher::C_WaitForWrite::finish(int) () from /usr/lib/librbd.so.1 #8 0x7f69e1c946ba in Context::complete(int) () from /usr/lib/librados.so.2 #9 0x7f69e1d373c8 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2 #10 0x7f69dbd43b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #11 0x7f69dba8e13d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x in ?? () Can you also print qemu_aio_context? (gdb) print qemu_aio_context $1 = (AioContext *) 0x7f69e68a4e00 (gdb) print *(AioContext*)0x7f69e68a4e00 $2 = {source = {callback_data = 0x0, callback_funcs = 0x0, source_funcs = 0x7f69e462d020, ref_count = 2, context = 0x7f69e68a5190, priority = 0, flags = 1, source_id = 1, poll_fds = 0x7f69e686aea0, prev = 0x0, next = 0x7f69e743ccd0, name = 0x0, priv = 0x0}, lock = {lock = {lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 2, __spins = 0, __list = { __prev = 0x0, __next = 0x0}}, __size = '\000' times>, "\002", '\000' , __align = 0}}, head = 0, tail = 0, cond = {cond = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' , __align = 0}}, owner_thread = { thread = 0}, nesting = 0, cb = 0x7f69e3e37b4f , cb_opaque = 0x7f69e68a4e00}, aio_handlers = { lh_first = 0x7f69e68a4f60}, walking_handlers = 0, bh_lock = {lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 2, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\002", '\000' times>, __align = 0}}, first_bh = 0x7f69e6b04d10, walking_bh = 0, notifier = {rfd = 4, wfd = 4}, pollfds = 0x7f69e68a4630, thread_pool = 0x0, tlg = {tl = {0x7f69e68a4fa0, 0x7f69e68a5010, 0x7f69e68a5080}}} Also print the backtrace of all threads, using "thread apply all bt full". http://pastebin.com/raw.php?i=uzcpN0zk THanks, Stefan
Re: [Qemu-devel] qemu 2.0 segfaults in event notifier
Am 02.06.2014 15:40, schrieb Stefan Hajnoczi: On Fri, May 30, 2014 at 04:10:39PM +0200, Stefan Priebe wrote: even with +From 271c0f68b4eae72691721243a1c37f46a3232d61 Mon Sep 17 00:00:00 2001 +From: Fam Zheng +Date: Wed, 21 May 2014 10:42:13 +0800 +Subject: [PATCH] aio: Fix use-after-free in cancellation path applied i saw today segfault with the following backtrace: Program terminated with signal 11, Segmentation fault. #0 0x7f9dd633343f in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 97 util/event_notifier-posix.c: No such file or directory. (gdb) bt #0 0x7f9dd633343f in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 #1 0x7f9dd5f4eafc in aio_notify (ctx=0x0) at async.c:246 #2 0x7f9dd5f4e697 in qemu_bh_schedule (bh=0x7f9b98eeeb30) at async.c:128 #3 0x7f9dd5fa2c44 in rbd_finish_aiocb (c=0x7f9dd9069ad0, rcb=0x7f9dd85f1770) at block/rbd.c:585 Hi Stefan, Please print the QEMUBH: (gdb) p *(QEMUBH*)0x7f9b98eeeb30 new trace: (gdb) bt #0 0x7f69e421c43f in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 #1 0x7f69e3e37afc in aio_notify (ctx=0x0) at async.c:246 #2 0x7f69e3e37697 in qemu_bh_schedule (bh=0x7f5dac217f60) at async.c:128 #3 0x7f69e3e8bc44 in rbd_finish_aiocb (c=0x7f5dac0c3f30, rcb=0x7f5dafa50610) at block/rbd.c:585 #4 0x7f69e17bee44 in librbd::AioCompletion::complete() () from /usr/lib/librbd.so.1 #5 0x7f69e17be832 in librbd::AioCompletion::complete_request(CephContext*, long) () from /usr/lib/librbd.so.1 #6 0x7f69e1c946ba in Context::complete(int) () from /usr/lib/librados.so.2 #7 0x7f69e17f1e85 in ObjectCacher::C_WaitForWrite::finish(int) () from /usr/lib/librbd.so.1 #8 0x7f69e1c946ba in Context::complete(int) () from /usr/lib/librados.so.2 #9 0x7f69e1d373c8 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2 #10 0x7f69dbd43b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #11 0x7f69dba8e13d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x in ?? () this i another core dump so address differ: (gdb) p *(QEMUBH*)0x7f5dac217f60 $1 = {ctx = 0x0, cb = 0x7f69e3e8bb75 , opaque = 0x7f5dafa50610, next = 0x7f69e6b04d10, scheduled = false, idle = false, deleted = true} It would also be interesting to print out the qemu_aio_context->first_bh linked list of QEMUBH structs to check whether 0x7f9b98eeeb30 is on the list. Do you mean just this: (gdb) p *(QEMUBH*)qemu_aio_context->first_bh $3 = {ctx = 0x7f69e68a4e00, cb = 0x7f69e41546a5 , opaque = 0x7f69e6b4a5e0, next = 0x7f69e6b4a570, scheduled = false, idle = false, deleted = false} The aio_bh_new() and aio_bh_schedule() APIs are supposed to be thread-safe. In theory the rbd.c code is fine. But maybe there is a race condition somewhere. rbd.c was fine with 1.7.0 Stefan
Re: [Qemu-devel] qemu 2.0 segfaults in event notifier
> Am 02.06.2014 um 15:40 schrieb Stefan Hajnoczi : > >> On Fri, May 30, 2014 at 04:10:39PM +0200, Stefan Priebe wrote: >> even with >> +From 271c0f68b4eae72691721243a1c37f46a3232d61 Mon Sep 17 00:00:00 2001 >> +From: Fam Zheng >> +Date: Wed, 21 May 2014 10:42:13 +0800 >> +Subject: [PATCH] aio: Fix use-after-free in cancellation path >> >> applied i saw today segfault with the following backtrace: >> >> Program terminated with signal 11, Segmentation fault. >> #0 0x7f9dd633343f in event_notifier_set (e=0x124) at >> util/event_notifier-posix.c:97 >> 97 util/event_notifier-posix.c: No such file or directory. >> (gdb) bt >> #0 0x7f9dd633343f in event_notifier_set (e=0x124) at >> util/event_notifier-posix.c:97 >> #1 0x7f9dd5f4eafc in aio_notify (ctx=0x0) at async.c:246 >> #2 0x7f9dd5f4e697 in qemu_bh_schedule (bh=0x7f9b98eeeb30) at async.c:128 >> #3 0x7f9dd5fa2c44 in rbd_finish_aiocb (c=0x7f9dd9069ad0, >> rcb=0x7f9dd85f1770) at block/rbd.c:585 > > Hi Stefan, > Please print the QEMUBH: > (gdb) p *(QEMUBH*)0x7f9b98eeeb30 > > It would also be interesting to print out the qemu_aio_context->first_bh > linked list of QEMUBH structs to check whether 0x7f9b98eeeb30 is on the > list. > > The aio_bh_new() and aio_bh_schedule() APIs are supposed to be > thread-safe. In theory the rbd.c code is fine. But maybe there is a > race condition somewhere. > > If you want to debug interactively, ping me on #qemu on irc.oftc.net. Hi, that would be great what's your username? On trip right now. Will be on irc in 4-5 hours or tomorrow in 16 hours. Greets, Stefan
[Qemu-devel] qemu 2.0 segfaults in event notifier
Hi, even with +From 271c0f68b4eae72691721243a1c37f46a3232d61 Mon Sep 17 00:00:00 2001 +From: Fam Zheng +Date: Wed, 21 May 2014 10:42:13 +0800 +Subject: [PATCH] aio: Fix use-after-free in cancellation path applied i saw today segfault with the following backtrace: Program terminated with signal 11, Segmentation fault. #0 0x7f9dd633343f in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 97 util/event_notifier-posix.c: No such file or directory. (gdb) bt #0 0x7f9dd633343f in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 #1 0x7f9dd5f4eafc in aio_notify (ctx=0x0) at async.c:246 #2 0x7f9dd5f4e697 in qemu_bh_schedule (bh=0x7f9b98eeeb30) at async.c:128 #3 0x7f9dd5fa2c44 in rbd_finish_aiocb (c=0x7f9dd9069ad0, rcb=0x7f9dd85f1770) at block/rbd.c:585 #4 0x7f9dd38d5e44 in librbd::AioCompletion::complete() () from /usr/lib/librbd.so.1 #5 0x7f9dd38d5832 in librbd::AioCompletion::complete_request(CephContext*, long) () from /usr/lib/librbd.so.1 #6 0x7f9dd3dab6ba in Context::complete(int) () from /usr/lib/librados.so.2 #7 0x7f9dd3908e85 in ObjectCacher::C_WaitForWrite::finish(int) () from /usr/lib/librbd.so.1 #8 0x7f9dd3dab6ba in Context::complete(int) () from /usr/lib/librados.so.2 #9 0x7f9dd3e4e3c8 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2 #10 0x7f9dcde5ab50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #11 0x7f9dcdba513d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x in ?? () Am 28.05.2014 21:44, schrieb Stefan Priebe: > is this: > commit 271c0f68b4eae72691721243a1c37f46a3232d61 > Author: Fam Zheng > Date: Wed May 21 10:42:13 2014 +0800 > > aio: Fix use-after-free in cancellation path > > Stefan > > Am 28.05.2014 21:40, schrieb Stefan Priebe: >> Hello, >> >> i mean since using qemu 2.0 i've now seen several times the following >> segfault: >> (gdb) bt >> #0 0x7f2af1196433 in event_notifier_set (e=0x124) at >> util/event_notifier-posix.c:97 >> #1 0x7f2af0db1afc in aio_notify (ctx=0x0) at async.c:246 >> #2 0x7f2af0db1697 in qemu_bh_schedule (bh=0x7f2ad401bec0) at >> async.c:128 >> #3 0x7f2af0e05c44 in rbd_finish_aiocb (c=0x7f2ad5ec4590, >> rcb=0x7f2ad63c5df0) at block/rbd.c:585 >> #4 0x7f2aee738e44 in librbd::AioCompletion::complete() () from >> /usr/lib/librbd.so.1 >> #5 0x7f2aee738832 in >> librbd::AioCompletion::complete_request(CephContext*, long) () from >> /usr/lib/librbd.so.1 >> #6 0x7f2aeec0e6ba in Context::complete(int) () from >> /usr/lib/librados.so.2 >> #7 0x7f2aee76be85 in ObjectCacher::C_WaitForWrite::finish(int) () >> from /usr/lib/librbd.so.1 >> #8 0x7f2aeec0e6ba in Context::complete(int) () from >> /usr/lib/librados.so.2 >> #9 0x7f2aeecb13c8 in Finisher::finisher_thread_entry() () from >> /usr/lib/librados.so.2 >> #10 0x7f2ae8cbdb50 in start_thread () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> #11 0x7f2ae8a080ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #12 0x in ?? () >> (gdb) >> >> >> from another VM: >> #0 0x7f89565ec433 in event_notifier_set (e=0x124) at >> util/event_notifier-posix.c:97 >> #1 0x7f8956207afc in aio_notify (ctx=0x0) at async.c:246 >> #2 0x7f8956207697 in qemu_bh_schedule (bh=0x7f882dd6d340) at >> async.c:128 >> #3 0x7f895625bc44 in rbd_finish_aiocb (c=0x7f882d4c34a0, >> rcb=0x7f882c0ae350) at block/rbd.c:585 >> #4 0x7f8953b8ee44 in librbd::AioCompletion::complete() () from >> /usr/lib/librbd.so.1 >> #5 0x7f8953b8e832 in >> librbd::AioCompletion::complete_request(CephContext*, long) () from >> /usr/lib/librbd.so.1 >> #6 0x7f89540646ba in Context::complete(int) () from >> /usr/lib/librados.so.2 >> #7 0x7f8953bc1e85 in ObjectCacher::C_WaitForWrite::finish(int) () >> from /usr/lib/librbd.so.1 >> #8 0x7f89540646ba in Context::complete(int) () from >> /usr/lib/librados.so.2 >> #9 0x7f89541073c8 in Finisher::finisher_thread_entry() () from >> /usr/lib/librados.so.2 >> #10 0x7f894e113b50 in start_thread () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> #11 0x7f894de5e0ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #12 0x in ?? () >> >> Stefan >>
Re: [Qemu-devel] [Qemu-stable] qemu segfault in librbd / aio_notify
Hi, even with +From 271c0f68b4eae72691721243a1c37f46a3232d61 Mon Sep 17 00:00:00 2001 +From: Fam Zheng +Date: Wed, 21 May 2014 10:42:13 +0800 +Subject: [PATCH] aio: Fix use-after-free in cancellation path applied i saw today segfault with the following backtrace: Program terminated with signal 11, Segmentation fault. #0 0x7f9dd633343f in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 97 util/event_notifier-posix.c: No such file or directory. (gdb) bt #0 0x7f9dd633343f in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 #1 0x7f9dd5f4eafc in aio_notify (ctx=0x0) at async.c:246 #2 0x7f9dd5f4e697 in qemu_bh_schedule (bh=0x7f9b98eeeb30) at async.c:128 #3 0x7f9dd5fa2c44 in rbd_finish_aiocb (c=0x7f9dd9069ad0, rcb=0x7f9dd85f1770) at block/rbd.c:585 #4 0x7f9dd38d5e44 in librbd::AioCompletion::complete() () from /usr/lib/librbd.so.1 #5 0x7f9dd38d5832 in librbd::AioCompletion::complete_request(CephContext*, long) () from /usr/lib/librbd.so.1 #6 0x7f9dd3dab6ba in Context::complete(int) () from /usr/lib/librados.so.2 #7 0x7f9dd3908e85 in ObjectCacher::C_WaitForWrite::finish(int) () from /usr/lib/librbd.so.1 #8 0x7f9dd3dab6ba in Context::complete(int) () from /usr/lib/librados.so.2 #9 0x7f9dd3e4e3c8 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2 #10 0x7f9dcde5ab50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #11 0x7f9dcdba513d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x in ?? () Am 28.05.2014 21:44, schrieb Stefan Priebe: is this: commit 271c0f68b4eae72691721243a1c37f46a3232d61 Author: Fam Zheng Date: Wed May 21 10:42:13 2014 +0800 aio: Fix use-after-free in cancellation path Stefan Am 28.05.2014 21:40, schrieb Stefan Priebe: Hello, i mean since using qemu 2.0 i've now seen several times the following segfault: (gdb) bt #0 0x7f2af1196433 in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 #1 0x7f2af0db1afc in aio_notify (ctx=0x0) at async.c:246 #2 0x7f2af0db1697 in qemu_bh_schedule (bh=0x7f2ad401bec0) at async.c:128 #3 0x7f2af0e05c44 in rbd_finish_aiocb (c=0x7f2ad5ec4590, rcb=0x7f2ad63c5df0) at block/rbd.c:585 #4 0x7f2aee738e44 in librbd::AioCompletion::complete() () from /usr/lib/librbd.so.1 #5 0x7f2aee738832 in librbd::AioCompletion::complete_request(CephContext*, long) () from /usr/lib/librbd.so.1 #6 0x7f2aeec0e6ba in Context::complete(int) () from /usr/lib/librados.so.2 #7 0x7f2aee76be85 in ObjectCacher::C_WaitForWrite::finish(int) () from /usr/lib/librbd.so.1 #8 0x7f2aeec0e6ba in Context::complete(int) () from /usr/lib/librados.so.2 #9 0x7f2aeecb13c8 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2 #10 0x7f2ae8cbdb50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #11 0x7f2ae8a080ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x in ?? () (gdb) from another VM: #0 0x7f89565ec433 in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 #1 0x7f8956207afc in aio_notify (ctx=0x0) at async.c:246 #2 0x7f8956207697 in qemu_bh_schedule (bh=0x7f882dd6d340) at async.c:128 #3 0x7f895625bc44 in rbd_finish_aiocb (c=0x7f882d4c34a0, rcb=0x7f882c0ae350) at block/rbd.c:585 #4 0x7f8953b8ee44 in librbd::AioCompletion::complete() () from /usr/lib/librbd.so.1 #5 0x7f8953b8e832 in librbd::AioCompletion::complete_request(CephContext*, long) () from /usr/lib/librbd.so.1 #6 0x7f89540646ba in Context::complete(int) () from /usr/lib/librados.so.2 #7 0x7f8953bc1e85 in ObjectCacher::C_WaitForWrite::finish(int) () from /usr/lib/librbd.so.1 #8 0x7f89540646ba in Context::complete(int) () from /usr/lib/librados.so.2 #9 0x7f89541073c8 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2 #10 0x7f894e113b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #11 0x7f894de5e0ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x in ?? () Stefan
Re: [Qemu-devel] [Qemu-stable] qemu segfault in librbd / aio_notify
is this: commit 271c0f68b4eae72691721243a1c37f46a3232d61 Author: Fam Zheng Date: Wed May 21 10:42:13 2014 +0800 aio: Fix use-after-free in cancellation path Stefan Am 28.05.2014 21:40, schrieb Stefan Priebe: Hello, i mean since using qemu 2.0 i've now seen several times the following segfault: (gdb) bt #0 0x7f2af1196433 in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 #1 0x7f2af0db1afc in aio_notify (ctx=0x0) at async.c:246 #2 0x7f2af0db1697 in qemu_bh_schedule (bh=0x7f2ad401bec0) at async.c:128 #3 0x7f2af0e05c44 in rbd_finish_aiocb (c=0x7f2ad5ec4590, rcb=0x7f2ad63c5df0) at block/rbd.c:585 #4 0x7f2aee738e44 in librbd::AioCompletion::complete() () from /usr/lib/librbd.so.1 #5 0x7f2aee738832 in librbd::AioCompletion::complete_request(CephContext*, long) () from /usr/lib/librbd.so.1 #6 0x7f2aeec0e6ba in Context::complete(int) () from /usr/lib/librados.so.2 #7 0x7f2aee76be85 in ObjectCacher::C_WaitForWrite::finish(int) () from /usr/lib/librbd.so.1 #8 0x7f2aeec0e6ba in Context::complete(int) () from /usr/lib/librados.so.2 #9 0x7f2aeecb13c8 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2 #10 0x7f2ae8cbdb50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #11 0x7f2ae8a080ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x in ?? () (gdb) from another VM: #0 0x7f89565ec433 in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 #1 0x7f8956207afc in aio_notify (ctx=0x0) at async.c:246 #2 0x7f8956207697 in qemu_bh_schedule (bh=0x7f882dd6d340) at async.c:128 #3 0x7f895625bc44 in rbd_finish_aiocb (c=0x7f882d4c34a0, rcb=0x7f882c0ae350) at block/rbd.c:585 #4 0x7f8953b8ee44 in librbd::AioCompletion::complete() () from /usr/lib/librbd.so.1 #5 0x7f8953b8e832 in librbd::AioCompletion::complete_request(CephContext*, long) () from /usr/lib/librbd.so.1 #6 0x7f89540646ba in Context::complete(int) () from /usr/lib/librados.so.2 #7 0x7f8953bc1e85 in ObjectCacher::C_WaitForWrite::finish(int) () from /usr/lib/librbd.so.1 #8 0x7f89540646ba in Context::complete(int) () from /usr/lib/librados.so.2 #9 0x7f89541073c8 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2 #10 0x7f894e113b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #11 0x7f894de5e0ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x in ?? () Stefan
[Qemu-devel] qemu segfault in librbd / aio_notify
Hello, i mean since using qemu 2.0 i've now seen several times the following segfault: (gdb) bt #0 0x7f2af1196433 in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 #1 0x7f2af0db1afc in aio_notify (ctx=0x0) at async.c:246 #2 0x7f2af0db1697 in qemu_bh_schedule (bh=0x7f2ad401bec0) at async.c:128 #3 0x7f2af0e05c44 in rbd_finish_aiocb (c=0x7f2ad5ec4590, rcb=0x7f2ad63c5df0) at block/rbd.c:585 #4 0x7f2aee738e44 in librbd::AioCompletion::complete() () from /usr/lib/librbd.so.1 #5 0x7f2aee738832 in librbd::AioCompletion::complete_request(CephContext*, long) () from /usr/lib/librbd.so.1 #6 0x7f2aeec0e6ba in Context::complete(int) () from /usr/lib/librados.so.2 #7 0x7f2aee76be85 in ObjectCacher::C_WaitForWrite::finish(int) () from /usr/lib/librbd.so.1 #8 0x7f2aeec0e6ba in Context::complete(int) () from /usr/lib/librados.so.2 #9 0x7f2aeecb13c8 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2 #10 0x7f2ae8cbdb50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #11 0x7f2ae8a080ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x in ?? () (gdb) from another VM: #0 0x7f89565ec433 in event_notifier_set (e=0x124) at util/event_notifier-posix.c:97 #1 0x7f8956207afc in aio_notify (ctx=0x0) at async.c:246 #2 0x7f8956207697 in qemu_bh_schedule (bh=0x7f882dd6d340) at async.c:128 #3 0x7f895625bc44 in rbd_finish_aiocb (c=0x7f882d4c34a0, rcb=0x7f882c0ae350) at block/rbd.c:585 #4 0x7f8953b8ee44 in librbd::AioCompletion::complete() () from /usr/lib/librbd.so.1 #5 0x7f8953b8e832 in librbd::AioCompletion::complete_request(CephContext*, long) () from /usr/lib/librbd.so.1 #6 0x7f89540646ba in Context::complete(int) () from /usr/lib/librados.so.2 #7 0x7f8953bc1e85 in ObjectCacher::C_WaitForWrite::finish(int) () from /usr/lib/librbd.so.1 #8 0x7f89540646ba in Context::complete(int) () from /usr/lib/librados.so.2 #9 0x7f89541073c8 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2 #10 0x7f894e113b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #11 0x7f894de5e0ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x in ?? () Stefan
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Am 14.05.2014 11:00, schrieb Paolo Bonzini: > Il 14/05/2014 10:38, Stefan Priebe - Profihost AG ha scritto: >> Currently it has the same as i already updated the package there too. >> >> So you mean i had done a mistake compiling the old package - so it had >> wrong sizes? > > Yes, probably. > > Can you do an "info mtree" for a machine that's running on the source? > > Paolo here it is: # info mtree memory -7ffe (prio 0, RW): system -dfff (prio 0, RW): alias ram-below-4g @pc.ram -dfff 000a-000b (prio 1, RW): alias smram-region @pci 000a-000b 000c-000c3fff (prio 1, R-): alias pam-rom @pc.ram 000c-000c3fff 000c4000-000c7fff (prio 1, R-): alias pam-rom @pc.ram 000c4000-000c7fff 000c8000-000cbfff (prio 1, R-): alias pam-rom @pc.ram 000c8000-000cbfff 000ca000-000ccfff (prio 1000, RW): alias kvmvapic-rom @pc.ram 000ca000-000ccfff 000cc000-000c (prio 1, R-): alias pam-rom @pc.ram 000cc000-000c 000d-000d3fff (prio 1, RW): alias pam-ram @pc.ram 000d-000d3fff 000d4000-000d7fff (prio 1, RW): alias pam-ram @pc.ram 000d4000-000d7fff 000d8000-000dbfff (prio 1, RW): alias pam-ram @pc.ram 000d8000-000dbfff 000dc000-000d (prio 1, RW): alias pam-ram @pc.ram 000dc000-000d 000e-000e3fff (prio 1, RW): alias pam-ram @pc.ram 000e-000e3fff 000e4000-000e7fff (prio 1, RW): alias pam-ram @pc.ram 000e4000-000e7fff 000e8000-000ebfff (prio 1, RW): alias pam-ram @pc.ram 000e8000-000ebfff 000ec000-000e (prio 1, RW): alias pam-ram @pc.ram 000ec000-000e 000f-000f (prio 1, R-): alias pam-rom @pc.ram 000f-000f e000- (prio 0, RW): alias pci-hole @pci e000- fec0-fec00fff (prio 0, RW): kvm-ioapic fed0-fed003ff (prio 0, RW): hpet fee0-feef (prio 4096, RW): icc-apic-container fee0-feef (prio 0, RW): kvm-apic-msi 0001-00021fff (prio 0, RW): alias ram-above-4g @pc.ram e000-0001 00022000-40021fff (prio 0, RW): alias pci-hole64 @pci 00022000-40021fff I/O - (prio 0, RW): io -0007 (prio 0, RW): dma-chan 0008-000f (prio 0, RW): dma-cont 0020-0021 (prio 0, RW): kvm-pic 0040-0043 (prio 0, RW): kvm-pit 0060-0060 (prio 0, RW): i8042-data 0061-0061 (prio 0, RW): elcr 0064-0064 (prio 0, RW): i8042-cmd 0070-0071 (prio 0, RW): rtc 007e-007f (prio 0, RW): kvmvapic 0080-0080 (prio 0, RW): ioport80 0081-0083 (prio 0, RW): alias dma-page @dma-page 0081-0083 0087-0087 (prio 0, RW): alias dma-page @dma-page 0087-0087 0089-008b (prio 0, RW): alias dma-page @dma-page 0089-008b 008f-008f (prio 0, RW): alias dma-page @dma-page 008f-008f 0092-0092 (prio 0, RW): port92 00a0-00a1 (prio 0, RW): kvm-pic 00b2-00b3 (prio 0, RW): apm-io 00c0-00cf (prio 0, RW): dma-chan 00d0-00df (prio 0, RW): dma-cont 00f0-00f0 (prio 0, RW): ioportF0 0170-0177 (prio 0, RW): alias ide @ide 0170-0177 01f0-01f7 (prio 0, RW): alias ide @ide 01f0-01f7 0376-0376 (prio 0, RW): alias ide @ide 0376-0376 03b0-03df (prio 0, RW): cirrus-io 03f1-03f5 (prio 0, RW): alias fdc @fdc 03f1-03f5 03f6-03f6 (prio 0, RW): alias ide @ide 03f6-03f6 03f7-03f7 (prio 0, RW): alias fdc @fdc 03f7-03f7 04d0-04d0 (prio 0, RW): kvm-elcr 04d1-04d1 (prio 0, R
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Am 14.05.2014 10:36, schrieb Paolo Bonzini: > Il 14/05/2014 10:29, Stefan Priebe - Profihost AG ha scritto: >> Hi, >> >> i compile qemu on my own. >> >> I have the rom files under /usr/share/kvm and they look like this: >> ls -la /usr/share/kvm/*.rom >> -rw-r--r-- 1 root root 173568 May 14 09:39 /usr/share/kvm/efi-e1000.rom >> -rw-r--r-- 1 root root 174592 May 14 09:39 >> /usr/share/kvm/efi-eepro100.rom >> -rw-r--r-- 1 root root 173056 May 14 09:39 >> /usr/share/kvm/efi-ne2k_pci.rom >> -rw-r--r-- 1 root root 173056 May 14 09:39 /usr/share/kvm/efi-pcnet.rom >> -rw-r--r-- 1 root root 176640 May 14 09:39 /usr/share/kvm/efi-rtl8139.rom >> -rw-r--r-- 1 root root 171008 May 14 09:39 /usr/share/kvm/efi-virtio.rom >> -rw-r--r-- 1 root root 67072 May 14 09:39 /usr/share/kvm/pxe-e1000.rom >> -rw-r--r-- 1 root root 61440 May 14 09:39 >> /usr/share/kvm/pxe-eepro100.rom >> -rw-r--r-- 1 root root 61440 May 14 09:39 >> /usr/share/kvm/pxe-ne2k_pci.rom >> -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-pcnet.rom >> -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-rtl8139.rom >> -rw-r--r-- 1 root root 60416 May 14 09:39 /usr/share/kvm/pxe-virtio.rom >> >> currently i don't know what's wrong. > > What about the source machine? Currently it has the same as i already updated the package there too. So you mean i had done a mistake compiling the old package - so it had wrong sizes? Greets. Stefan
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Am 14.05.2014 10:11, schrieb Paolo Bonzini: > Il 14/05/2014 09:17, Stefan Priebe - Profihost AG ha scritto: >> i now was able to catch the error. >> >> It is: >> Length mismatch: :00:12.0/virtio-net-pci.rom: 4 in != 1 >> qemu: warning: error while loading state for instance 0x0 of device 'ram' >> load of migration failed > > This is a bug of your distribution. The file sizes for ROMs should > never change. In particular, if you round the sizes up to the next > power of 2 you should always have: > > 128k for bios.bin > 256k for bios-256k.bin > 64k for pxe-*.rom > 256k for efi-*.rom > 64k for vgabios-* > > Unfortunately, most distribution get pxe-*.rom sizes wrong, because at > some point iPXE grew more features and didn't fit in 64k anymore with > the default configuration. I know at least Fedora does. > > The solution is to copy the binaries from the QEMU git repository > (directory pc-bios/) to /usr/share/qemu. Hi, i compile qemu on my own. I have the rom files under /usr/share/kvm and they look like this: ls -la /usr/share/kvm/*.rom -rw-r--r-- 1 root root 173568 May 14 09:39 /usr/share/kvm/efi-e1000.rom -rw-r--r-- 1 root root 174592 May 14 09:39 /usr/share/kvm/efi-eepro100.rom -rw-r--r-- 1 root root 173056 May 14 09:39 /usr/share/kvm/efi-ne2k_pci.rom -rw-r--r-- 1 root root 173056 May 14 09:39 /usr/share/kvm/efi-pcnet.rom -rw-r--r-- 1 root root 176640 May 14 09:39 /usr/share/kvm/efi-rtl8139.rom -rw-r--r-- 1 root root 171008 May 14 09:39 /usr/share/kvm/efi-virtio.rom -rw-r--r-- 1 root root 67072 May 14 09:39 /usr/share/kvm/pxe-e1000.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-eepro100.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-ne2k_pci.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-pcnet.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-rtl8139.rom -rw-r--r-- 1 root root 60416 May 14 09:39 /usr/share/kvm/pxe-virtio.rom currently i don't know what's wrong. Stefan
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Hi, i now was able to catch the error. It is: Length mismatch: :00:12.0/virtio-net-pci.rom: 4 in != 1 qemu: warning: error while loading state for instance 0x0 of device 'ram' load of migration failed Stefan Am 09.05.2014 19:05, schrieb Paolo Bonzini: > Il 09/05/2014 15:13, Stefan Priebe - Profihost AG ha scritto: >> >> I see no output at the monitor of Qemu 2.0. >> >> # migrate -d tcp:a.b.c.d:6000 >> >> # info migrate >> capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on >> zero-blocks: off >> Migration status: failed >> total time: 0 milliseconds >> >> The target machine is still running at this point with no output. > > Anything on its stdout? > > Another test you could do, in addition to changing the devices, is this: > > 1) try with commit 6141f3bd6904df7cf9519c6444a14a608b9874c4 on the > destination (the next one caused a migration problem that was fixed > later). If it passes, go to step 1a. If it fails, go to step 2. > > 1a) try with commit c01a71c1a56fa27f43449ff59e5d03b2483658a2. If it > passes, go to step 1b. If it fails, report it here. > > 1b) bisect between v2.0.0 (known-bad) and > c01a71c1a56fa27f43449ff59e5d03b2483658a2 (known-good) to find the > culprit. Report results. > > 2) change the source to v1.7.0 and bisect between v1.7.0 and > 6141f3bd6904df7cf9519c6444a14a608b9874c4. Report results. > > Thanks! > > Paolo
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
) Am 09.05.2014 19:05, schrieb Paolo Bonzini: Il 09/05/2014 15:13, Stefan Priebe - Profihost AG ha scritto: I see no output at the monitor of Qemu 2.0. # migrate -d tcp:a.b.c.d:6000 # info migrate capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on zero-blocks: off Migration status: failed total time: 0 milliseconds The target machine is still running at this point with no output. Anything on its stdout? Sadly no. But far more complex than i thought. I'm seeing this problem only on machines originally started under qemu 1.5 ten migrated to qemu 1.7 with machine type=pc440fc-1.5 . Only those do not work. But they also do not work migrating them to qemu 1.7.1. So let's ignore the whole fact and thanks for your bisect ideas. I will just shut them down all and start them with a fresh qemu 2.0. Another test you could do, in addition to changing the devices, is this: 1) try with commit 6141f3bd6904df7cf9519c6444a14a608b9874c4 on the destination (the next one caused a migration problem that was fixed later). If it passes, go to step 1a. If it fails, go to step 2. 1a) try with commit c01a71c1a56fa27f43449ff59e5d03b2483658a2. If it passes, go to step 1b. If it fails, report it here. 1b) bisect between v2.0.0 (known-bad) and c01a71c1a56fa27f43449ff59e5d03b2483658a2 (known-good) to find the culprit. Report results. 2) change the source to v1.7.0 and bisect between v1.7.0 and 6141f3bd6904df7cf9519c6444a14a608b9874c4. Report results. Thanks! Paolo
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Am 09.05.2014 18:29, schrieb Dr. David Alan Gilbert: * Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote: Am 09.05.2014 um 15:41 schrieb "Dr. David Alan Gilbert" : * Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote: Hello list, i was trying to migrate older Qemu (1.5 and 1.7.2) to a machine running Qemu 2.0. I started the target machine with: -machine type=pc-i440fx-1.5 / -machine type=pc-i440fx-1.7 I'd expect you to have to run with the same machine type on both sides. I ran some simple virt-test migrate tests (just the basic one) and got v1.5.3->v1.6.2 v1.5.3->v1.7.1 v1.5.3->v2.0.0-rc1 working for most machine types, pc-i440fx-1.5 passed with all of those. Note that's only the simplest test. My test involved USB Controller virtio Network cards and rbd virtio-scsi drives. That probably makes things more interesting :-) I'd start with the simplest config you can find and add each one of those until it breaks. Can you give me your simple start line so I could test if this works for me while adding some more arguments. I've got a slightly hacked up libvirt, but I'm doing ./run -t qemu --machine-type=$MACTYPE -g Linux.JeOS.19.x86_64.i440fx --qemu-bin=$SRC --qemu-dst-bin=$DST --qemu_sandbox=off --tests migrate.default.tcp --disk-bus=ide and looking at the logs we have: /opt/qemu/v1.5.3/bin/qemu-system-x86_64 \ -S \ -name 'virt-tests-vm1' \ -sandbox off \ -M pc-i440fx-1.5 \ -nodefaults \ -vga std \ -chardev socket,id=hmp_id_hmp1,path=/tmp/monitor-hmp1-20140415-104517-rAfzfDef,server,nowait \ -mon chardev=hmp_id_hmp1,mode=readline \ -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20140415-104517-rAfzfDef,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20140415-104517-rAfzfDef,path=/tmp/seabios-20140415-104517-rAfzfDef,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20140415-104517-rAfzfDef,iobase=0x402 \ -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 \ -drive id=drive_image1,if=none,file=/home/dgilbert/virt-test/shared/data/images/jeos-19-64.qcow2 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=04 \ -device virtio-net-pci,mac=9a:18:19:1a:1b:1c,id=idz0uWFP,vectors=4,netdev=idMwmdMc,bus=pci.0,addr=05 \ -netdev user,id=idMwmdMc,hostfwd=tcp::5000-:22 \ -m 1024 \ -smp 2,maxcpus=2,cores=1,threads=1,sockets=2 \ -cpu 'SandyBridge' \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :1 \ -rtc base=utc,clock=host,driftfix=none \ -boot order=cdn,once=c,menu=off \ -enable-kvm Dave OK biggest bummer it works fine with all freshly started VMs. No idea why. So i'll try with paolos idea migrating old VMs. Thanks! Stefan Dave -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
> Am 09.05.2014 um 15:41 schrieb "Dr. David Alan Gilbert" : > > * Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote: >> Hello list, >> >> i was trying to migrate older Qemu (1.5 and 1.7.2) to a machine running >> Qemu 2.0. >> >> I started the target machine with: >> >> -machine type=pc-i440fx-1.5 / -machine type=pc-i440fx-1.7 > > I'd expect you to have to run with the same machine type on both sides. > I ran some simple virt-test migrate tests (just the basic one) and got > v1.5.3->v1.6.2 > v1.5.3->v1.7.1 > v1.5.3->v2.0.0-rc1 > working for most machine types, pc-i440fx-1.5 passed with all of those. > Note that's only the simplest test. My test involved USB Controller virtio Network cards and rbd virtio-scsi drives. Can you give me your simple start line so I could test if this works for me while adding some more arguments. Thanks! Stefan > > Dave > -- > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
[Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Hello list, i was trying to migrate older Qemu (1.5 and 1.7.2) to a machine running Qemu 2.0. I started the target machine with: -machine type=pc-i440fx-1.5 / -machine type=pc-i440fx-1.7 But the migration simply fails. Migrating Qemu 2.0 to Qemu 2.0 succeeds. I see no output at the monitor of Qemu 2.0. # migrate -d tcp:a.b.c.d:6000 # info migrate capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on zero-blocks: off Migration status: failed total time: 0 milliseconds The target machine is still running at this point with no output. Stefan
Re: [Qemu-devel] [PATCH 00/35] pc: ACPI memory hotplug
Hello Igor, while testing your patchset i came to a very stupid problem. I wanted to test migration and it cames out that the migration works fine after plugging in memory only if i run the target VM without the -daemonize option. If i enable the -daemonize option the target vm tries to read from non readable memory. proc maps shows: 7f9334021000-7f933800 ---p 00:00 0 where it tries to read from. Also the memory layout is different in daemonize mode than in non daemonize mode. Stefan Am 04.04.2014 15:36, schrieb Igor Mammedov: > What's new since v7: > > * Per Andreas' suggestion dropped DIMMBus concept. > > * Added hotplug binding for bus-less devices > > * DIMM device is split to backend and frontend. Therefore following > command/options were added for supporting it: > > For memory-ram backend: > CLI: -object-add memory-ram, > with options: 'id' and 'size' > For dimm frontend: > option "size" became readonly, pulling it's size from attached backend > added option "memdev" for specifying backend by 'id' > > * dropped support for 32 bit guests > > * failed hotplug action doesn't consume 1 slot anymore > > * vaious fixes adressing reviewer's comments most of them in ACPI part > --- > > This series allows to hotplug 'arbitrary' DIMM devices specifying size, > NUMA node mapping (guest side), slot and address where to map it, at runtime. > > Due to ACPI limitation there is need to specify a number of possible > DIMM devices. For this task -m option was extended to support > following format: > > -m [mem=]RamSize[,slots=N,maxmem=M] > > To allow memory hotplug user must specify a pair of additional parameters: > 'slots' - number of possible increments > 'maxmem' - max possible total memory size QEMU is allowed to use, >including RamSize. > > minimal monitor command syntax to hotplug DIMM device: > > object_add memory-ram,id=memX,size=1G > device_add dimm,id=dimmX,memdev=memX > > DIMM device provides following properties that could be used with > device_add / -device to alter default behavior: > > id- unique string identifying device [mandatory] > slot - number in range [0-slots) [optional], if not specified > the first free slot is used > node - NUMA node id [optional] (default: 0) > size - amount of memory to add, readonly derived from backing memdev > start - guest's physical address where to plug DIMM [optional], > if not specified the first gap in hotplug memory region > that fits DIMM is used > > -device option could be used for adding potentially hotunplugable DIMMs > and also for specifying hotplugged DIMMs in migration case. > > Tested guests: > - RHEL 6x64 > - Windows 2012DCx64 > - Windows 2008DCx64 > > Known limitations/bugs/TODOs: > - hot-remove is not supported, yet > - max number of supported DIMM devices 255 (due to ACPI object name >limit), could be increased creating several containers and putting >DIMMs there. (exercise for future) > - e820 table doesn't include DIMM devices added with -device / >(or after reboot devices added with device_add) > - Windows 2008 remembers DIMM configuration, so if DIMM with other >start/size is added into the same slot, it refuses to use it insisting >on old mapping. > > QEMU git tree for testing is available at: > https://github.com/imammedo/qemu/commits/memory-hotplug-v8 > > Example QEMU cmd line: > qemu-system-x86_64 -enable-kvm -monitor unix:/tmp/mon,server,nowait \ > -m 4096,slots=4,maxmem=8G guest.img > > PS: > Windows guest requires SRAT table for hotplug to work so add an extra > option: >-numa node > to QEMU command line. > > > Igor Mammedov (34): > vl: convert -m to QemuOpts > object_add: allow completion handler to get canonical path > add memdev backend infrastructure > vl.c: extend -m option to support options for memory hotplug > add pc-{i440fx,q35}-2.1 machine types > pc: create custom generic PC machine type > qdev: hotplug for buss-less devices > qdev: expose DeviceState.hotplugged field as a property > dimm: implement dimm device abstraction > memory: add memory_region_is_mapped() API > dimm: do not allow to set already busy memdev > pc: initialize memory hotplug address space > pc: exit QEMU if slots > 256 > pc: add 'etc/reserved-memory-end' fw_cfg interface for SeaBIOS > pc: add memory hotplug handler to PC_MACHINE > dimm: add busy address check and address auto-allocation > dimm: add busy slot check and slot auto-allocation > acpi: rename cpu_hotplug_defs.h to acpi_defs.h > acpi: memory hotplug ACPI hardware implementation > trace: add acpi memory hotplug IO region events > trace: add DIMM slot & address allocation for target-i386 > acpi:piix4: make plug/unlug callbacks generic > acpi:piix4: add memory hotplug handling > pc: ich9 lpc: make it work with global/compat properties > acpi:ich9: add m
[Qemu-devel] multiple USB Serial devices qemu load
Hello, i've 4 USB serial devices and one HID device i want to pass to a guest. The passing itself works fine but while the guest has 0 load or cpu usage the qemu process itself has around 40% cpu usage on a single 3,2 ghz E3 xeon. I already tried xhci but it doesn't change anything. Also the latency of the usb devices is very high and fluctuate. Qemu Version 1.7.1 Inside the guest: # lsusb -t /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M |__ Port 1: Dev 2, If 0, Class=HID, Driver=usbhid, 480M |__ Port 2: Dev 3, If 0, Class=HID, Driver=usbfs, 12M |__ Port 3: Dev 4, If 0, Class=vend., Driver=ftdi_sio, 12M |__ Port 4: Dev 5, If 0, Class=hub, Driver=hub/8p, 12M |__ Port 1: Dev 6, If 0, Class=vend., Driver=ftdi_sio, 12M |__ Port 2: Dev 7, If 0, Class=comm., Driver=cdc_acm, 12M |__ Port 2: Dev 7, If 1, Class=data, Driver=cdc_acm, 12M |__ Port 3: Dev 8, If 0, Class=vend., Driver=ftdi_sio, 12M Qremu command line: qemu -pidfile /var/run/qemu-server/102.pid -daemonize -name test -smp sockets=1,cores=1 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu kvm64,+lahf_lm,+x2apic,+sep -k de -m 1024 -device nec-usb-xhci,id=xhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=xhci.0,port=1 -device usb-host,hostbus=2,hostport=1.2 -device usb-host,hostbus=2,hostport=1.3 -device usb-host,hostbus=2,hostport=1.4 -device usb-host,hostbus=2,hostport=1.1.1 -device usb-host,hostbus=2,hostport=1.1.4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/ssdstor/images/102/vm-102-disk-1.qcow2,if=none,id=drive-scsi0,format=qcow2,cache=writeback,discard=on,aio=native -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0 -netdev type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,vhost=on -device virtio-net-pci,mac=3A:53:02:E3:76:59,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 Does anybody have an idea how to lower the CPU usage? Greets, Stefan
Re: [Qemu-devel] memory allocation of migration changed?
Am 24.02.2014 17:13, schrieb Eric Blake: On 02/24/2014 08:00 AM, Stefan Hajnoczi wrote: What is the right way to check for enough free memory and memory usage of a specific vm? I would approach it in terms of guest RAM allocation plus QEMU overhead: host_ram >= num_guests * guest_ram_size + num_guests * qemu_overhead The qemu_overhead is the question mark. It depends on the QEMU features the guests have enabled and number of devices. QEMU also does not have a strict policy on predictable memory consumption, which makes it hard to give a formula for it. In fact, at one point libvirt tried to put an automatic cap on the memory usable by qemu by multiplying RAM size and accounting for a margin of overhead, but no matter what heuristics we tried, we still got complaints from users that their guests were killed when they ran out of memory, and so we ended up reverting the automatic limits from libvirt. (You can still enforce a limit as an end user, although the libvirt documentation no longer recommends attempting that, for as long as the qemu allocation remains unpredictable.) Might something like: vm.overcommit_ratio=100 vm.overcommit_memory=2 help? So that the migration might not happen but that's better than killing a random process? Stefan
Re: [Qemu-devel] memory allocation of migration changed?
Am 14.02.2014 16:03, schrieb Stefan Hajnoczi: On Tue, Feb 11, 2014 at 07:30:54PM +0100, Stefan Priebe wrote: Am 11.02.2014 16:44, schrieb Stefan Hajnoczi: On Tue, Feb 11, 2014 at 3:54 PM, Stefan Priebe - Profihost AG wrote: in the past (Qemu 1.5) a migration failed if there was not enogh memory on the target host available directly at the beginning. Now with Qemu 1.7 i've seen succeeded migrations but the kernel OOM memory killer killing qemu processes. So the migration seems to takes place without having anough memory on the target machine? How much memory is the guest configured with? How much memory does the host have? Guest: 48GB Host: 192GB I wonder if there are zero pages that can be migrated almost "for free" and the destination host doesn't touch. When they are touched for the first time after migration handover, they need to be allocated on the destination host. This can lead to OOM if you overcommitted memory. In the past the migration failed immediatly with exit code 255. Can you reproduce the OOM reliably? It should be possible to debug it and figure out whether it's just bad luck or a true regression. So there is no known patch changing this behaviour? What is about those? fc1c4a5d32e15a4c40c47945da85ef9c1e0c1b54 211ea74022f51164a7729030b28eec90b6c99a08 f1c72795af573b24a7da5eb52375c9aba8a37972 Yes, that's what I was referring to when I mentioned zero pages. The problem might just be that the destination host didn't have enough free memory. Migration succeeded due to memory overcommit on the host, but quickly ran out of memory after handover. The quick answer there is to reconsider your overcommitting memory and also checking memory availability before live migrating. Yes that makes sense in the past it was just working due to a failing qemu process. What is the right way to check for enough free memory and memory usage of a specific vm? Stefan
Re: [Qemu-devel] memory allocation of migration changed?
Am 14.02.2014 15:59, schrieb Stefan Hajnoczi: On Tue, Feb 11, 2014 at 07:32:46PM +0100, Stefan Priebe wrote: Am 11.02.2014 17:22, schrieb Peter Lieven: Am 11.02.2014 um 16:44 schrieb Stefan Hajnoczi : On Tue, Feb 11, 2014 at 3:54 PM, Stefan Priebe - Profihost AG wrote: in the past (Qemu 1.5) a migration failed if there was not enogh memory on the target host available directly at the beginning. Now with Qemu 1.7 i've seen succeeded migrations but the kernel OOM memory killer killing qemu processes. So the migration seems to takes place without having anough memory on the target machine? How much memory is the guest configured with? How much memory does the host have? I wonder if there are zero pages that can be migrated almost "for free" and the destination host doesn't touch. When they are touched for the first time after migration handover, they need to be allocated on the destination host. This can lead to OOM if you overcommitted memory. Can you reproduce the OOM reliably? It should be possible to debug it and figure out whether it's just bad luck or a true regression. Stefan Kernel Version would also be interesting as well as thp and ksm settings. Kernel Host: 3.10.26 What's thp / ksm? how to get those settings? Transparent Huge Pages # cat /sys/kernel/mm/transparent_hugepage/enabled Kernel Samepage Merging # cat /sys/kernel/mm/ksm/run # cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise never # cat /sys/kernel/mm/ksm/run 1 Stefan
Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
Am 13.02.2014 21:06, schrieb Dr. David Alan Gilbert: * Stefan Priebe (s.pri...@profihost.ag) wrote: Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert: * Stefan Priebe (s.pri...@profihost.ag) wrote: i could fix it by explicitly disable xbzrle - it seems its automatically on if i do not set the migration caps to false. So it seems to be a xbzrle bug. Stefan can you give me some more info on your hardware and migration setup; that stressapptest (which is a really nice find!) really batters the memory and it means the migration isn't converging for me, so I'm curious what your setup is. That one is devlopment by google and known to me since a few years. Google has detected that memtest and co are not good enough to stress test memory. Hi Stefan, I've just posted a patch to qemu-devel that fixes two bugs that we found; I've only tried a small stressapptest run and it seems to survive with them (where it didn't before); you might like to try it if you're up for rebuilding qemu. It's the one entitled ' [PATCH] Fix two XBZRLE corruption issues' I'll try and get a larger run done myself, but I'd be interested to hear if it fixes it for you (or anyone else who hit the problem). Yes works fine - now no crash but it's sower than without XBZRLE ;-) Without XBZRLE: i needed migrate_downtime 4 around 60s With XBZRLE: i needed migrate_downtime 16 and 240s Dave -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK