Re: About the performance of hyper-v
== > > Analyze events for all VMs, all VCPUs: > > VM-EXITSamples Samples% Time%Min TimeMax > > Time Avg time > >MSR_WRITE 92404589.96%81.10% 0.42us > > 68.42us 1.26us ( +- 0.07% ) > >DR_ACCESS 44669 4.35% 2.36% 0.32us > > 50.74us 0.76us ( +- 0.32% ) > > EXTERNAL_INTERRUPT 29809 2.90% 6.42% 0.66us > > 70.75us 3.10us ( +- 0.54% ) > > VMCALL 17819 1.73% 5.21% 0.75us > > 15.64us 4.20us ( +- 0.33% > > > > Total Samples:1027227, Total events handled time:1436343.94us. > > === > > > > The result shows the overhead increased. enable the apicv can help to > > reduce the vm-exit > > caused by interrupt injection, but on the other side, there are a lot > > of vm-exit caused by APIC_EOI. > > > > When turning off the hyper-v and using the kvm apicv, there is no such > > overhead. > > I think I know what's happening. We've asked Windows to use synthetic > MSRs to access APIC (HV_APIC_ACCESS_RECOMMENDED) and this can't be > accelerated in hardware. > > Could you please try the following hack (KVM): > > diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c > index c8f2592ccc99..66ee85a83e9a 100644 > --- a/arch/x86/kvm/cpuid.c > +++ b/arch/x86/kvm/cpuid.c > @@ -145,6 +145,13 @@ void kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu) >vcpu->arch.ia32_misc_enable_msr & >MSR_IA32_MISC_ENABLE_MWAIT); > } > + > + /* Dirty hack: force HV_DEPRECATING_AEOI_RECOMMENDED. Not to be > merged! */ > + best = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_ENLIGHTMENT_INFO, 0); > + if (best) { > + best->eax &= ~HV_X64_APIC_ACCESS_RECOMMENDED; > + best->eax |= HV_DEPRECATING_AEOI_RECOMMENDED; > + } > } > EXPORT_SYMBOL_GPL(kvm_update_cpuid_runtime); > > > It seems turning on hyper V related features is not always the best > > choice for a windows guest. > > Generally it is, we'll just need to make QEMU smarter when setting > 'recommendation' bits. > Hi Vitaly, I have tried your patch and found it can help to reduce the overhead. it works as well as the option "" is set in libvirt xml. ===with your patch and stimer enabled= Analyze events for all VMs, all VCPUs: VM-EXITSamples Samples% Time%Min TimeMax Time Avg time APIC_WRITE 17223278.36%68.99% 0.70us 47.71us 1.48us ( +- 0.18% ) DR_ACCESS 19136 8.71% 4.42% 0.55us 4.42us 0.85us ( +- 0.32% ) EXTERNAL_INTERRUPT 15921 7.24%13.84% 0.87us 55.28us 3.21us ( +- 0.55% ) VMCALL 6971 3.17%10.34% 1.16us 12.02us 5.48us ( +- 0.49% Total Samples:219802, Total events handled time:369310.30us. ===with hypervisor disabled= Analyze events for all VMs, all VCPUs: VM-EXITSamples Samples% Time%Min TimeMax Time Avg time APIC_WRITE 20048278.51%68.62% 0.64us 49.51us 1.37us ( +- 0.16% ) DR_ACCESS 24235 9.49% 4.92% 0.55us 3.65us 0.81us ( +- 0.26% ) EXTERNAL_INTERRUPT 17084 6.69%13.20% 0.89us 56.38us 3.09us ( +- 0.53% ) VMCALL 7124 2.79% 9.87% 1.26us 12.39us 5.54us ( +- 0.49% ) EOI_INDUCED 5066 1.98% 1.36% 0.66us 2.64us 1.07us ( +- 0.25% ) IO_INSTRUCTION591 0.23% 1.27% 3.37us 673.23us 8.59us ( +- 13.69% ) Total Samples:255363, Total events handled time:399954.27us. Thanks! Liang
Re: About the performance of hyper-v
Liang Li writes: >> >> > Analyze events for all VMs, all VCPUs: >> >> > VM-EXITSamples Samples% Time%Min TimeMax >> >> > Time Avg time >> >> > EXTERNAL_INTERRUPT 47183159.89%68.58% 0.64us >> >> > 65.42us 2.34us ( +- 0.11% ) >> >> >MSR_WRITE 23893230.33%23.07% 0.48us >> >> > 41.05us 1.56us ( +- 0.14% ) >> >> > >> >> > Total Samples:787803, Total events handled time:1611193.84us. >> >> > >> >> > I tried turning off hyper-v for the same workload and repeat the test, >> >> > the overall virtualization overhead reduced by about of 50%: >> >> > >> >> > --- >> >> > >> >> > Analyze events for all VMs, all VCPUs: >> >> > >> >> > VM-EXITSamples Samples% Time%Min TimeMax >> >> > Time Avg time >> >> > APIC_WRITE 25515274.43%50.72% 0.49us >> >> > 50.01us 1.42us ( +- 0.14% ) >> >> >EPT_MISCONFIG 3996711.66%40.58% 1.55us >> >> > 686.05us 7.27us ( +- 0.43% ) >> >> >DR_ACCESS 3500310.21% 4.64% 0.32us >> >> > 40.03us 0.95us ( +- 0.32% ) >> >> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us >> >> > 57.38us 2.25us ( +- 1.42% ) >> >> > >> >> > Total Samples:342788, Total events handled time:715695.62us. >> >> > >> >> > For this scenario, hyper-v works really bad. stimer works better >> >> > than hpet, but on the other hand, it relies on SynIC which has >> >> > negative effects for IPI intensive workloads. >> >> > Do you have any plans for improvement? >> >> > >> >> >> >> Hey, >> >> >> >> the above can be caused by the fact that when 'hv-synic' is enabled, KVM >> >> automatically disables APICv and this can explain the overhead and the >> >> fact that you're seeing more vmexits. KVM disables APICv because SynIC's >> >> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows >> >> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only >> >> inhibit APICv if the recommendation was ignored. This is implemented in >> >> the following KVM patch series: >> >> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuzn...@redhat.com/ >> >> >> >> It will, however, require a new 'hv-something' flag to QEMU. For now, it >> >> can be tested with 'hv-passthrough'. >> >> >> >> It would be great if you could give it a spin! >> >> >> >> -- >> >> Vitaly >> > >> > It's great to know that you already have a solution for this. :) >> > >> > By the way, is there any requirement for the version of windows or >> > windows updates for the new feature to work? >> >> AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd >> expect WS2008 to ignore it completely (and thus SynIC will always be >> disabling APICv for it). >> > > Hi Vitaly, > I tried your patchset and found it's not helpful to reduce the > virtualization overhead. > here are some perfdata with the same workload > > === > Analyze events for all VMs, all VCPUs: > VM-EXITSamples Samples% Time%Min TimeMax > Time Avg time >MSR_WRITE 92404589.96%81.10% 0.42us > 68.42us 1.26us ( +- 0.07% ) >DR_ACCESS 44669 4.35% 2.36% 0.32us > 50.74us 0.76us ( +- 0.32% ) > EXTERNAL_INTERRUPT 29809 2.90% 6.42% 0.66us > 70.75us 3.10us ( +- 0.54% ) > VMCALL 17819 1.73% 5.21% 0.75us > 15.64us 4.20us ( +- 0.33% > > Total Samples:1027227, Total events handled time:1436343.94us. > === > > The result shows the overhead increased. enable the apicv can help to > reduce the vm-exit > caused by interrupt injection, but on the other side, there are a lot > of vm-exit caused by APIC_EOI. > > When turning off the hyper-v and using the kvm apicv, there is no such > overhead. I think I know what's happening. We've asked Windows to use synthetic MSRs to access APIC (HV_APIC_ACCESS_RECOMMENDED) and this can't be accelerated in hardware. Could you please try the following hack (KVM): diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index c8f2592ccc99..66ee85a83e9a 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -145,6 +145,13 @@ void kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu) vcpu->arch.ia32_misc_enable_msr & MSR_IA32_MISC_ENABLE_MWAIT); } + + /* Dirty hack: force HV_DEPRECATING_AEOI_RECOMMENDED. Not to be merged! */ + best = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_ENLIGHTMENT_INFO, 0); + if (best) { + best->eax &= ~HV_X64_APIC_ACCESS_RECOMMENDED; + best->eax |= HV_DEPRECATING_AEOI_RECOMMENDED; + } } EXPORT_SYMBOL_GPL(kvm_update_cpuid_runtime); > It seems turning on hyper V related features is not always
Re: About the performance of hyper-v
> >> > Analyze events for all VMs, all VCPUs: > >> > VM-EXITSamples Samples% Time%Min TimeMax > >> > Time Avg time > >> > EXTERNAL_INTERRUPT 47183159.89%68.58% 0.64us > >> > 65.42us 2.34us ( +- 0.11% ) > >> >MSR_WRITE 23893230.33%23.07% 0.48us > >> > 41.05us 1.56us ( +- 0.14% ) > >> > > >> > Total Samples:787803, Total events handled time:1611193.84us. > >> > > >> > I tried turning off hyper-v for the same workload and repeat the test, > >> > the overall virtualization overhead reduced by about of 50%: > >> > > >> > --- > >> > > >> > Analyze events for all VMs, all VCPUs: > >> > > >> > VM-EXITSamples Samples% Time%Min TimeMax > >> > Time Avg time > >> > APIC_WRITE 25515274.43%50.72% 0.49us > >> > 50.01us 1.42us ( +- 0.14% ) > >> >EPT_MISCONFIG 3996711.66%40.58% 1.55us > >> > 686.05us 7.27us ( +- 0.43% ) > >> >DR_ACCESS 3500310.21% 4.64% 0.32us > >> > 40.03us 0.95us ( +- 0.32% ) > >> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us > >> > 57.38us 2.25us ( +- 1.42% ) > >> > > >> > Total Samples:342788, Total events handled time:715695.62us. > >> > > >> > For this scenario, hyper-v works really bad. stimer works better > >> > than hpet, but on the other hand, it relies on SynIC which has > >> > negative effects for IPI intensive workloads. > >> > Do you have any plans for improvement? > >> > > >> > >> Hey, > >> > >> the above can be caused by the fact that when 'hv-synic' is enabled, KVM > >> automatically disables APICv and this can explain the overhead and the > >> fact that you're seeing more vmexits. KVM disables APICv because SynIC's > >> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows > >> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only > >> inhibit APICv if the recommendation was ignored. This is implemented in > >> the following KVM patch series: > >> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuzn...@redhat.com/ > >> > >> It will, however, require a new 'hv-something' flag to QEMU. For now, it > >> can be tested with 'hv-passthrough'. > >> > >> It would be great if you could give it a spin! > >> > >> -- > >> Vitaly > > > > It's great to know that you already have a solution for this. :) > > > > By the way, is there any requirement for the version of windows or > > windows updates for the new feature to work? > > AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd > expect WS2008 to ignore it completely (and thus SynIC will always be > disabling APICv for it). > Hi Vitaly, I tried your patchset and found it's not helpful to reduce the virtualization overhead. here are some perfdata with the same workload === Analyze events for all VMs, all VCPUs: VM-EXITSamples Samples% Time%Min TimeMax Time Avg time MSR_WRITE 92404589.96%81.10% 0.42us 68.42us 1.26us ( +- 0.07% ) DR_ACCESS 44669 4.35% 2.36% 0.32us 50.74us 0.76us ( +- 0.32% ) EXTERNAL_INTERRUPT 29809 2.90% 6.42% 0.66us 70.75us 3.10us ( +- 0.54% ) VMCALL 17819 1.73% 5.21% 0.75us 15.64us 4.20us ( +- 0.33% Total Samples:1027227, Total events handled time:1436343.94us. === The result shows the overhead increased. enable the apicv can help to reduce the vm-exit caused by interrupt injection, but on the other side, there are a lot of vm-exit caused by APIC_EOI. When turning off the hyper-v and using the kvm apicv, there is no such overhead. It seems turning on hyper V related features is not always the best choice for a windows guest. Thanks! Liang
Re: About the performance of hyper-v
Liang Li writes: >> > Hi Vitaly, >> > >> > I found a case that the virtualization overhead was almost doubled >> > when turning on Hper-v related features compared to that without any >> > no hyper-v feature. It happens when running a 3D game in windows >> > guest in qemu kvm environment. >> > >> > By investigation, I found there are a lot of IPIs triggered by guest, >> > when turning on the hyer-v related features including stimer, for the >> > apicv is turned off, at least two vm exits are needed for processing a >> > single IPI. >> > >> > >> > perf stat will show something like below [recorded for 5 seconds] >> > >> > - >> > >> > Analyze events for all VMs, all VCPUs: >> > VM-EXITSamples Samples% Time%Min TimeMax >> > Time Avg time >> > EXTERNAL_INTERRUPT 47183159.89%68.58% 0.64us >> > 65.42us 2.34us ( +- 0.11% ) >> >MSR_WRITE 23893230.33%23.07% 0.48us >> > 41.05us 1.56us ( +- 0.14% ) >> > >> > Total Samples:787803, Total events handled time:1611193.84us. >> > >> > I tried turning off hyper-v for the same workload and repeat the test, >> > the overall virtualization overhead reduced by about of 50%: >> > >> > --- >> > >> > Analyze events for all VMs, all VCPUs: >> > >> > VM-EXITSamples Samples% Time%Min TimeMax >> > Time Avg time >> > APIC_WRITE 25515274.43%50.72% 0.49us >> > 50.01us 1.42us ( +- 0.14% ) >> >EPT_MISCONFIG 3996711.66%40.58% 1.55us >> > 686.05us 7.27us ( +- 0.43% ) >> >DR_ACCESS 3500310.21% 4.64% 0.32us >> > 40.03us 0.95us ( +- 0.32% ) >> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us >> > 57.38us 2.25us ( +- 1.42% ) >> > >> > Total Samples:342788, Total events handled time:715695.62us. >> > >> > For this scenario, hyper-v works really bad. stimer works better >> > than hpet, but on the other hand, it relies on SynIC which has >> > negative effects for IPI intensive workloads. >> > Do you have any plans for improvement? >> > >> >> Hey, >> >> the above can be caused by the fact that when 'hv-synic' is enabled, KVM >> automatically disables APICv and this can explain the overhead and the >> fact that you're seeing more vmexits. KVM disables APICv because SynIC's >> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows >> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only >> inhibit APICv if the recommendation was ignored. This is implemented in >> the following KVM patch series: >> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuzn...@redhat.com/ >> >> It will, however, require a new 'hv-something' flag to QEMU. For now, it >> can be tested with 'hv-passthrough'. >> >> It would be great if you could give it a spin! >> >> -- >> Vitaly > > It's great to know that you already have a solution for this. :) > > By the way, is there any requirement for the version of windows or > windows updates for the new feature to work? AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd expect WS2008 to ignore it completely (and thus SynIC will always be disabling APICv for it). -- Vitaly
Re: About the performance of hyper-v
> > Hi Vitaly, > > > > I found a case that the virtualization overhead was almost doubled > > when turning on Hper-v related features compared to that without any > > no hyper-v feature. It happens when running a 3D game in windows > > guest in qemu kvm environment. > > > > By investigation, I found there are a lot of IPIs triggered by guest, > > when turning on the hyer-v related features including stimer, for the > > apicv is turned off, at least two vm exits are needed for processing a > > single IPI. > > > > > > perf stat will show something like below [recorded for 5 seconds] > > > > - > > > > Analyze events for all VMs, all VCPUs: > > VM-EXITSamples Samples% Time%Min TimeMax > > Time Avg time > > EXTERNAL_INTERRUPT 47183159.89%68.58% 0.64us > > 65.42us 2.34us ( +- 0.11% ) > >MSR_WRITE 23893230.33%23.07% 0.48us > > 41.05us 1.56us ( +- 0.14% ) > > > > Total Samples:787803, Total events handled time:1611193.84us. > > > > I tried turning off hyper-v for the same workload and repeat the test, > > the overall virtualization overhead reduced by about of 50%: > > > > --- > > > > Analyze events for all VMs, all VCPUs: > > > > VM-EXITSamples Samples% Time%Min TimeMax > > Time Avg time > > APIC_WRITE 25515274.43%50.72% 0.49us > > 50.01us 1.42us ( +- 0.14% ) > >EPT_MISCONFIG 3996711.66%40.58% 1.55us > > 686.05us 7.27us ( +- 0.43% ) > >DR_ACCESS 3500310.21% 4.64% 0.32us > > 40.03us 0.95us ( +- 0.32% ) > > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us > > 57.38us 2.25us ( +- 1.42% ) > > > > Total Samples:342788, Total events handled time:715695.62us. > > > > For this scenario, hyper-v works really bad. stimer works better > > than hpet, but on the other hand, it relies on SynIC which has > > negative effects for IPI intensive workloads. > > Do you have any plans for improvement? > > > > Hey, > > the above can be caused by the fact that when 'hv-synic' is enabled, KVM > automatically disables APICv and this can explain the overhead and the > fact that you're seeing more vmexits. KVM disables APICv because SynIC's > 'AutoEOI' feature is incompatible with it. We can, however, tell Windows > to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only > inhibit APICv if the recommendation was ignored. This is implemented in > the following KVM patch series: > https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuzn...@redhat.com/ > > It will, however, require a new 'hv-something' flag to QEMU. For now, it > can be tested with 'hv-passthrough'. > > It would be great if you could give it a spin! > > -- > Vitaly It's great to know that you already have a solution for this. :) By the way, is there any requirement for the version of windows or windows updates for the new feature to work? Thanks! Liang
Re: About the performance of hyper-v
Liang Li writes: > [resend for missing cc] > > Hi Vitaly, > > I found a case that the virtualization overhead was almost doubled > when turning on Hper-v related features compared to that without any > no hyper-v feature. It happens when running a 3D game in windows > guest in qemu kvm environment. > > By investigation, I found there are a lot of IPIs triggered by guest, > when turning on the hyer-v related features including stimer, for the > apicv is turned off, at least two vm exits are needed for processing a > single IPI. > > > perf stat will show something like below [recorded for 5 seconds] > > - > > Analyze events for all VMs, all VCPUs: > VM-EXITSamples Samples% Time%Min TimeMax > Time Avg time > EXTERNAL_INTERRUPT 47183159.89%68.58% 0.64us > 65.42us 2.34us ( +- 0.11% ) >MSR_WRITE 23893230.33%23.07% 0.48us > 41.05us 1.56us ( +- 0.14% ) > > Total Samples:787803, Total events handled time:1611193.84us. > > I tried turning off hyper-v for the same workload and repeat the test, > the overall virtualization overhead reduced by about of 50%: > > --- > > Analyze events for all VMs, all VCPUs: > > VM-EXITSamples Samples% Time%Min TimeMax > Time Avg time > APIC_WRITE 25515274.43%50.72% 0.49us > 50.01us 1.42us ( +- 0.14% ) >EPT_MISCONFIG 3996711.66%40.58% 1.55us > 686.05us 7.27us ( +- 0.43% ) >DR_ACCESS 3500310.21% 4.64% 0.32us > 40.03us 0.95us ( +- 0.32% ) > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us > 57.38us 2.25us ( +- 1.42% ) > > Total Samples:342788, Total events handled time:715695.62us. > > For this scenario, hyper-v works really bad. stimer works better > than hpet, but on the other hand, it relies on SynIC which has > negative effects for IPI intensive workloads. > Do you have any plans for improvement? > Hey, the above can be caused by the fact that when 'hv-synic' is enabled, KVM automatically disables APICv and this can explain the overhead and the fact that you're seeing more vmexits. KVM disables APICv because SynIC's 'AutoEOI' feature is incompatible with it. We can, however, tell Windows to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only inhibit APICv if the recommendation was ignored. This is implemented in the following KVM patch series: https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuzn...@redhat.com/ It will, however, require a new 'hv-something' flag to QEMU. For now, it can be tested with 'hv-passthrough'. It would be great if you could give it a spin! -- Vitaly
About the performance of hyper-v
[resend for missing cc] Hi Vitaly, I found a case that the virtualization overhead was almost doubled when turning on Hper-v related features compared to that without any no hyper-v feature. It happens when running a 3D game in windows guest in qemu kvm environment. By investigation, I found there are a lot of IPIs triggered by guest, when turning on the hyer-v related features including stimer, for the apicv is turned off, at least two vm exits are needed for processing a single IPI. perf stat will show something like below [recorded for 5 seconds] - Analyze events for all VMs, all VCPUs: VM-EXITSamples Samples% Time%Min TimeMax Time Avg time EXTERNAL_INTERRUPT 47183159.89%68.58% 0.64us 65.42us 2.34us ( +- 0.11% ) MSR_WRITE 23893230.33%23.07% 0.48us 41.05us 1.56us ( +- 0.14% ) Total Samples:787803, Total events handled time:1611193.84us. I tried turning off hyper-v for the same workload and repeat the test, the overall virtualization overhead reduced by about of 50%: --- Analyze events for all VMs, all VCPUs: VM-EXITSamples Samples% Time%Min TimeMax Time Avg time APIC_WRITE 25515274.43%50.72% 0.49us 50.01us 1.42us ( +- 0.14% ) EPT_MISCONFIG 3996711.66%40.58% 1.55us 686.05us 7.27us ( +- 0.43% ) DR_ACCESS 3500310.21% 4.64% 0.32us 40.03us 0.95us ( +- 0.32% ) EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us 57.38us 2.25us ( +- 1.42% ) Total Samples:342788, Total events handled time:715695.62us. For this scenario, hyper-v works really bad. stimer works better than hpet, but on the other hand, it relies on SynIC which has negative effects for IPI intensive workloads. Do you have any plans for improvement? Thanks! Liang