Re: [PATCH RFC 0/7] KVM: nVMX: enlightened VMCS initial implementation
Paolo Bonzini writes: > On 21/12/2017 13:50, Vitaly Kuznetsov wrote: >> I'm back with (somewhat frustrating) results (E5-2603): > > v4 (that would be Broadwell)? > Sorry, v3, actually. Haswell. (the first one supporting vmcs shadowing afaiu). >> 1) Windows on Hyper-V (no nesting): 1350 cycles >> >> 2) Windows on Hyper-V on Hyper-V: 8600 >> >> 3) Windows on KVM (no nesting): 1150 cycles >> >> 4) Windows on Hyper-V on KVM (no enlightened VMCS): 18200 >> >> 5) Windows on Hyper-V on KVM (enlightened VMCS): 17100 > > What version were you using for KVM? There are quite a few nested virt > optimizations in kvm/queue (which may make enlightened VMCS both more or > less efficient). This is kvm/queue and I rebased enlightened VMCS patches to it. > > In particular, with latest kvm/queue you could try tracing vmread and > vmwrite vmexits, and see if you get any. If you do, that might be an > easy few hundred cycles savings. Will do. -- Vitaly ___ devel mailing list de...@linuxdriverproject.org http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
Re: [PATCH RFC 0/7] KVM: nVMX: enlightened VMCS initial implementation
On 21/12/2017 13:50, Vitaly Kuznetsov wrote: > I'm back with (somewhat frustrating) results (E5-2603): v4 (that would be Broadwell)? > 1) Windows on Hyper-V (no nesting): 1350 cycles > > 2) Windows on Hyper-V on Hyper-V: 8600 > > 3) Windows on KVM (no nesting): 1150 cycles > > 4) Windows on Hyper-V on KVM (no enlightened VMCS): 18200 > > 5) Windows on Hyper-V on KVM (enlightened VMCS): 17100 What version were you using for KVM? There are quite a few nested virt optimizations in kvm/queue (which may make enlightened VMCS both more or less efficient). In particular, with latest kvm/queue you could try tracing vmread and vmwrite vmexits, and see if you get any. If you do, that might be an easy few hundred cycles savings. Paolo ___ devel mailing list de...@linuxdriverproject.org http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
Re: [PATCH RFC 0/7] KVM: nVMX: enlightened VMCS initial implementation
Vitaly Kuznetsov writes: > Paolo Bonzini writes: > >> On 18/12/2017 18:17, Vitaly Kuznetsov wrote: >>> The original author of these patches does no longer work at Red Hat, I >>> agreed to take this over and send upstream. Here is his original >>> description: >>> >>> "Makes KVM implement the enlightened VMCS feature per Hyper-V TLFS 5.0b. >>> I've measured about %5 improvement in cost of a nested VM exit (Hyper-V >>> enabled Windows Server 2016 nested in KVM)." >> >> Can you try reproducing this and see how much a simple CPUID loop costs in: >> >> * Hyper-V on Hyper-V (with enlightened VMCS, as a proxy for a full >> implementation including the clean fields mask) >> >> * Hyper-V on KVM, with and without enlightened VMCS >> >> The latest kvm/queue branch already cut a lot of the cost of a nested VM >> exit (from ~22000 to ~14000 clock cycles for KVM on KVM), so we could >> also see if Hyper-V needs shadowing of more fields. > > I tested this series before sending out and was able to reproduce said > 5% improvement with the feature (but didn't keep record of clock > cycles). I'll try doing tests you mentioned on the same hardware and > come back with the result. Hopefully I'll manage that before holidays. I'm back with (somewhat frustrating) results (E5-2603): 1) Windows on Hyper-V (no nesting): 1350 cycles 2) Windows on Hyper-V on Hyper-V: 8600 3) Windows on KVM (no nesting): 1150 cycles 4) Windows on Hyper-V on KVM (no enlightened VMCS): 18200 5) Windows on Hyper-V on KVM (enlightened VMCS): 17100 -- Vitaly ___ devel mailing list de...@linuxdriverproject.org http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
Re: [PATCH RFC 0/7] KVM: nVMX: enlightened VMCS initial implementation
Paolo Bonzini writes: > On 18/12/2017 18:17, Vitaly Kuznetsov wrote: >> The original author of these patches does no longer work at Red Hat, I >> agreed to take this over and send upstream. Here is his original >> description: >> >> "Makes KVM implement the enlightened VMCS feature per Hyper-V TLFS 5.0b. >> I've measured about %5 improvement in cost of a nested VM exit (Hyper-V >> enabled Windows Server 2016 nested in KVM)." > > Can you try reproducing this and see how much a simple CPUID loop costs in: > > * Hyper-V on Hyper-V (with enlightened VMCS, as a proxy for a full > implementation including the clean fields mask) > > * Hyper-V on KVM, with and without enlightened VMCS > > The latest kvm/queue branch already cut a lot of the cost of a nested VM > exit (from ~22000 to ~14000 clock cycles for KVM on KVM), so we could > also see if Hyper-V needs shadowing of more fields. I tested this series before sending out and was able to reproduce said 5% improvement with the feature (but didn't keep record of clock cycles). I'll try doing tests you mentioned on the same hardware and come back with the result. Hopefully I'll manage that before holidays. Thanks, -- Vitaly ___ devel mailing list de...@linuxdriverproject.org http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
Re: [PATCH RFC 0/7] KVM: nVMX: enlightened VMCS initial implementation
On 18/12/2017 18:17, Vitaly Kuznetsov wrote: > The original author of these patches does no longer work at Red Hat, I > agreed to take this over and send upstream. Here is his original > description: > > "Makes KVM implement the enlightened VMCS feature per Hyper-V TLFS 5.0b. > I've measured about %5 improvement in cost of a nested VM exit (Hyper-V > enabled Windows Server 2016 nested in KVM)." Can you try reproducing this and see how much a simple CPUID loop costs in: * Hyper-V on Hyper-V (with enlightened VMCS, as a proxy for a full implementation including the clean fields mask) * Hyper-V on KVM, with and without enlightened VMCS The latest kvm/queue branch already cut a lot of the cost of a nested VM exit (from ~22000 to ~14000 clock cycles for KVM on KVM), so we could also see if Hyper-V needs shadowing of more fields. > This is just an initial implementation. By leveraging clean fields mask > we can further improve performance. I'm also interested in implementing > the other part of the feature: consuming enlightened VMCS when KVM is > running on top of Hyper-V. I'm also interested in consuming enlightened VMCS on Hyper-V if that can provide better performance for KVM on Azure. Paolo ___ devel mailing list de...@linuxdriverproject.org http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
[PATCH RFC 0/7] KVM: nVMX: enlightened VMCS initial implementation
The original author of these patches does no longer work at Red Hat, I agreed to take this over and send upstream. Here is his original description: "Makes KVM implement the enlightened VMCS feature per Hyper-V TLFS 5.0b. I've measured about %5 improvement in cost of a nested VM exit (Hyper-V enabled Windows Server 2016 nested in KVM)." This is just an initial implementation. By leveraging clean fields mask we can further improve performance. I'm also interested in implementing the other part of the feature: consuming enlightened VMCS when KVM is running on top of Hyper-V. Ladi Prosek (7): KVM: x86: rename HV_X64_MSR_APIC_ASSIST_PAGE to HV_X64_MSR_VP_ASSIST_PAGE KVM: nVMX: modify vmcs12 fields to match Hyper-V enlightened VMCS KVM: nVMX: add I/O exit ECX, ESI, EDI, EIP vmcs12 fields KVM: hyperv: define VP assist page structure and add helpers KVM: nVMX: add KVM_CAP_HYPERV_ENLIGHTENED_VMCS capability KVM: nVMX: add enlightened VMCS state KVM: nVMX: implement enlightened VMPTRLD arch/x86/include/asm/kvm_host.h| 3 + arch/x86/include/asm/vmx.h | 4 + arch/x86/include/uapi/asm/hyperv.h | 20 +- arch/x86/kvm/hyperv.c | 31 ++- arch/x86/kvm/hyperv.h | 4 + arch/x86/kvm/lapic.c | 4 +- arch/x86/kvm/lapic.h | 4 +- arch/x86/kvm/svm.c | 9 + arch/x86/kvm/vmx.c | 467 ++--- arch/x86/kvm/x86.c | 19 +- include/uapi/linux/kvm.h | 1 + 11 files changed, 407 insertions(+), 159 deletions(-) -- 2.14.3 ___ devel mailing list de...@linuxdriverproject.org http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
