[PATCH RFC 2/7] KVM: nVMX: modify vmcs12 fields to match Hyper-V enlightened VMCS
From: Ladi Prosek
Reorders existing fields and adds fields specific to Hyper-V. The layout
now matches Hyper-V TLFS 5.0b 16.11.2 Enlightened VMCS.
Fields used by KVM but missing from Hyper-V are placed in the second half
of the VMCS page to minimize the chances they will clash with future
enlightened VMCS versions.
Signed-off-by: Ladi Prosek
Signed-off-by: Vitaly Kuznetsov
---
[Vitaly]: Update VMCS12_REVISION to some new arbitrary number.
---
arch/x86/kvm/vmx.c | 321 +++--
1 file changed, 187 insertions(+), 134 deletions(-)
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index 8eba631c4dbd..cd5f29a57880 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -239,159 +239,212 @@ struct __packed vmcs12 {
u32 revision_id;
u32 abort;
+ union {
+ u64 hv_vmcs[255];
+ struct {
+ u16 host_es_selector;
+ u16 host_cs_selector;
+ u16 host_ss_selector;
+ u16 host_ds_selector;
+ u16 host_fs_selector;
+ u16 host_gs_selector;
+ u16 host_tr_selector;
+
+ u64 host_ia32_pat;
+ u64 host_ia32_efer;
+
+ /*
+* To allow migration of L1 (complete with its L2
+* guests) between machines of different natural widths
+* (32 or 64 bit), we cannot have unsigned long fields
+* with no explicit size. We use u64 (aliased
+* natural_width) instead. Luckily, x86 is
+* little-endian.
+*/
+ natural_width host_cr0;
+ natural_width host_cr3;
+ natural_width host_cr4;
+
+ natural_width host_ia32_sysenter_esp;
+ natural_width host_ia32_sysenter_eip;
+ natural_width host_rip;
+ u32 host_ia32_sysenter_cs;
+
+ u32 pin_based_vm_exec_control;
+ u32 vm_exit_controls;
+ u32 secondary_vm_exec_control;
+
+ u64 io_bitmap_a;
+ u64 io_bitmap_b;
+ u64 msr_bitmap;
+
+ u16 guest_es_selector;
+ u16 guest_cs_selector;
+ u16 guest_ss_selector;
+ u16 guest_ds_selector;
+ u16 guest_fs_selector;
+ u16 guest_gs_selector;
+ u16 guest_ldtr_selector;
+ u16 guest_tr_selector;
+
+ u32 guest_es_limit;
+ u32 guest_cs_limit;
+ u32 guest_ss_limit;
+ u32 guest_ds_limit;
+ u32 guest_fs_limit;
+ u32 guest_gs_limit;
+ u32 guest_ldtr_limit;
+ u32 guest_tr_limit;
+ u32 guest_gdtr_limit;
+ u32 guest_idtr_limit;
+
+ u32 guest_es_ar_bytes;
+ u32 guest_cs_ar_bytes;
+ u32 guest_ss_ar_bytes;
+ u32 guest_ds_ar_bytes;
+ u32 guest_fs_ar_bytes;
+ u32 guest_gs_ar_bytes;
+ u32 guest_ldtr_ar_bytes;
+ u32 guest_tr_ar_bytes;
+
+ natural_width guest_es_base;
+ natural_width guest_cs_base;
+ natural_width guest_ss_base;
+ natural_width guest_ds_base;
+ natural_width guest_fs_base;
+ natural_width guest_gs_base;
+ natural_width guest_ldtr_base;
+ natural_width guest_tr_base;
+ natural_width guest_gdtr_base;
+ natural_width guest_idtr_base;
+
+ u64 padding64_1[3];
+
+ u64 vm_exit_msr_store_addr;
+ u64 vm_exit_msr_load_addr;
+ u64 vm_entry_msr_load_addr;
+
+ natural_width cr3_target_value0;
+ natural_width cr3_target_value1;
+ natural_width cr3_target_value2;
+ natural_width cr3_target_value3;
+
+ u32 page_fault_error_code_mask;
+ u32 page_fault_error_code_match;
+
+ u32 cr3_target_count;
+ u32 vm_exit_msr_store_count;
+ u32 vm_exit_msr_load_count;
+ u32 vm_entry_msr_load_count;
+
+ u64 tsc_offset;
+ u6
Re: [PATCH RFC 2/7] KVM: nVMX: modify vmcs12 fields to match Hyper-V enlightened VMCS
Jim Mattson writes: > At this point in time, I don't think you can just blithely change the > virtual VMCS layout and revision number. Existing VMs using the old > layout and revision number must continue to work on versions of kvm > past this point. You could tie the layout and revision number changes > to KVM_CAP_HYPERV_ENLIGHTENED_VMCS if you like, but kvm must be able > to continue to service VMs using the previous layout and revision > number in perpetuity. > I see what you mean. In case we need to keep migration of nested workloads working between KVMs of different versions we can't (ever) touch vmcs12. The way to go in this case, I think, is to create a completely separate enlightened_vmcs12 struct and use it when appropriate. We can't possibly support migrating workloads which use enlightened VMCS to an old KVM which doesn't support it. P.S. "If there are changes in this struct, VMCS12_REVISION must be changed." comment needs to be replaced with "Don't even think about changing this" :-) -- Vitaly ___ devel mailing list de...@linuxdriverproject.org http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
Re: [PATCH RFC 2/7] KVM: nVMX: modify vmcs12 fields to match Hyper-V enlightened VMCS
On 19/12/2017 13:25, Vitaly Kuznetsov wrote: > >> At this point in time, I don't think you can just blithely change the >> virtual VMCS layout and revision number. Existing VMs using the old >> layout and revision number must continue to work on versions of kvm >> past this point. You could tie the layout and revision number changes >> to KVM_CAP_HYPERV_ENLIGHTENED_VMCS if you like, but kvm must be able >> to continue to service VMs using the previous layout and revision >> number in perpetuity. >> > I see what you mean. In case we need to keep migration of nested > workloads working between KVMs of different versions we can't (ever) > touch vmcs12. Actually we can, for two reasons. First, the active VMCS is stored in host RAM (not in guest RAM). This means there are clear points where to do the translation, namely vmptrld and the (not yet upstream) ioctl to set VMX state. Therefore you only need to keep an (offset, type) map from old to new layout map; at those two points if you detect an old VMCS12_REVISION you copy the fields one by one instead of doing a memcpy. The next vmclear or vmptrld or get-VMX-state ioctl will automatically update to the new VMCS12_REVISION. Of course, this is a one-way street unless you also add support for writing old VMCS12_REVISIONs. But, second, VMX state migration is not upstream yet, so nested hypervisors are currently not migratable: the active VMCS12 state will not be migrated at all! So in upstream KVM we wouldn't even need to upgrade the VMCS12_REVISION to make changes to vmcs12. That said... > The way to go in this case, I think, is to create a completely separate > enlightened_vmcs12 struct and use it when appropriate. We can't possibly > support migrating workloads which use enlightened VMCS to an old KVM > which doesn't support it. ... this is probably a good idea as well. Paolo ___ devel mailing list de...@linuxdriverproject.org http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
Re: [PATCH RFC 2/7] KVM: nVMX: modify vmcs12 fields to match Hyper-V enlightened VMCS
On 19/12/2017 18:40, Jim Mattson wrote: > I'm not sure that's really the right way to go, since any guest that > has already read the IA32_VMX_BASIC MSR has a right to expect the VMCS > revision to remain unchanged. Hmm, not just that, "the VMCS revision identifier is never written by the processor" according to the SDM. Maybe the code that accesses the vmcs12 can be placed in a .h file and included more than once in vmx.c. Paolo ___ devel mailing list de...@linuxdriverproject.org http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
Re: [PATCH RFC 2/7] KVM: nVMX: modify vmcs12 fields to match Hyper-V enlightened VMCS
Paolo Bonzini writes: > On 19/12/2017 13:25, Vitaly Kuznetsov wrote: >> >>> At this point in time, I don't think you can just blithely change the >>> virtual VMCS layout and revision number. Existing VMs using the old >>> layout and revision number must continue to work on versions of kvm >>> past this point. You could tie the layout and revision number changes >>> to KVM_CAP_HYPERV_ENLIGHTENED_VMCS if you like, but kvm must be able >>> to continue to service VMs using the previous layout and revision >>> number in perpetuity. >>> >> I see what you mean. In case we need to keep migration of nested >> workloads working between KVMs of different versions we can't (ever) >> touch vmcs12. > > Actually we can, for two reasons. > > First, the active VMCS is stored in host RAM (not in guest RAM). This > means there are clear points where to do the translation, namely vmptrld > and the (not yet upstream) ioctl to set VMX state. > > Therefore you only need to keep an (offset, type) map from old to new > layout map; at those two points if you detect an old VMCS12_REVISION you > copy the fields one by one instead of doing a memcpy. The next vmclear > or vmptrld or get-VMX-state ioctl will automatically update to the new > VMCS12_REVISION. Of course, this is a one-way street unless you also > add support for writing old VMCS12_REVISIONs. > > But, second, VMX state migration is not upstream yet, so nested > hypervisors are currently not migratable: the active VMCS12 state will > not be migrated at all! So in upstream KVM we wouldn't even need to > upgrade the VMCS12_REVISION to make changes to vmcs12. > > That said... > >> The way to go in this case, I think, is to create a completely separate >> enlightened_vmcs12 struct and use it when appropriate. We can't possibly >> support migrating workloads which use enlightened VMCS to an old KVM >> which doesn't support it. > > ... this is probably a good idea as well. > One other thing I was thinking about is the shared definition of enlightened vmcs which we'll use for both KVM-on-Hyper-V and Hyper-V on KVM and for that purpose I'd like it to be placed outside of struct vmcs12. We can, of course, embed it at the beginning of vmcs12. Thinking long term (and having in mind that Microsoft will be updating enlightened VMCS on its own schedule) -- what would be the preferred way to go? It seems that personally I'm leaning towards untangling and keeping it separate from vmcs12 but I can't really find a convincing argument... -- Vitaly ___ devel mailing list de...@linuxdriverproject.org http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
