RE: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked
> -Original Message- > From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo > Bonzini > Sent: Friday, October 16, 2015 2:13 AM > To: David Matlack ; Wu, Feng > Cc: alex.william...@redhat.com; Joerg Roedel ; Marcelo > Tosatti ; eric.au...@linaro.org; kvm list > ; io...@lists.linux-foundation.org; linux- > ker...@vger.kernel.org > Subject: Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when > vCPU is blocked > > > > On 15/10/2015 19:39, David Matlack wrote: > > But after spending more time reading the source code this morning I > > found that kvm_vcpu_check_block() eventually calls into > > vmx_sync_pir_to_irr(), which copies PIR to IRR and clears ON. And then > > apic_find_highest_irr() detects the pending posted interrupt. > > Right. And related to this, Feng, can you check if this is still > necessary on kvm/queue: > > @@ -6518,6 +6523,20 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) > kvm_vcpu_reload_apic_access_page(vcpu); > } > > + /* > + * KVM_REQ_EVENT is not set when posted interrupts are set by > + * VT-d hardware, so we have to update RVI unconditionally. > + */ > + if (kvm_lapic_enabled(vcpu)) { > + /* > + * Update architecture specific hints for APIC > + * virtual interrupt delivery. > + */ > + if (kvm_x86_ops->hwapic_irr_update) > + kvm_x86_ops->hwapic_irr_update(vcpu, > + kvm_lapic_find_highest_irr(vcpu)); > + } > + > if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win) { > kvm_apic_accept_events(vcpu); > if (vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) { > @@ -6534,13 +6553,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) > kvm_x86_ops->enable_irq_window(vcpu); > > if (kvm_lapic_enabled(vcpu)) { > - /* > - * Update architecture specific hints for APIC > - * virtual interrupt delivery. > - */ > - if (kvm_x86_ops->hwapic_irr_update) > - kvm_x86_ops->hwapic_irr_update(vcpu, > - kvm_lapic_find_highest_irr(vcpu)); > update_cr8_intercept(vcpu); > kvm_lapic_sync_to_vapic(vcpu); > } > I think the above code is needed, before the place where 'KVM_REQ_EVENT' got checked in vcpu_enter_guest(), VT-d hardware can issue notification event at any time. Consider the following scenario: vcpu_run() { .. for(;;) { point #1 vcpu_enter_guest() } point #2 } For example, if we receive notification events issued by VT-d hardware at point #1 and point#2, then enter vcpu_enter_guest() with 'KVM_REQ_EVENT' not set, the interrupts cannot be delivered to guest during _this_ VM-Entry. The point is that VT-d hardware can issue notification event at any time, but it cannot set 'KVM_REQ_EVENT' like software does. Maybe one thing we can do is only executing the following code when vt-d pi is enabled, + /* + * KVM_REQ_EVENT is not set when posted interrupts are set by + * VT-d hardware, so we have to update RVI unconditionally. + */ + if (kvm_lapic_enabled(vcpu)) { + /* + * Update architecture specific hints for APIC + * virtual interrupt delivery. + */ + if (kvm_x86_ops->hwapic_irr_update) + kvm_x86_ops->hwapic_irr_update(vcpu, + kvm_lapic_find_highest_irr(vcpu)); + } + And do this inside the KVM_REQ_EVENT check when VT-d PI is not enabled. Thanks, Feng > > It may be obsolete now that we have the patch from Radim to set > KVM_REQ_EVENT > in vmx_sync_pir_to_irr > (http://permalink.gmane.org/gmane.linux.kernel/2057138). > > Thanks, > > Paolo
Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked
On 15/10/2015 19:39, David Matlack wrote: > But after spending more time reading the source code this morning I > found that kvm_vcpu_check_block() eventually calls into > vmx_sync_pir_to_irr(), which copies PIR to IRR and clears ON. And then > apic_find_highest_irr() detects the pending posted interrupt. Right. And related to this, Feng, can you check if this is still necessary on kvm/queue: @@ -6518,6 +6523,20 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) kvm_vcpu_reload_apic_access_page(vcpu); } + /* +* KVM_REQ_EVENT is not set when posted interrupts are set by +* VT-d hardware, so we have to update RVI unconditionally. +*/ + if (kvm_lapic_enabled(vcpu)) { + /* +* Update architecture specific hints for APIC +* virtual interrupt delivery. +*/ + if (kvm_x86_ops->hwapic_irr_update) + kvm_x86_ops->hwapic_irr_update(vcpu, + kvm_lapic_find_highest_irr(vcpu)); + } + if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win) { kvm_apic_accept_events(vcpu); if (vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) { @@ -6534,13 +6553,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) kvm_x86_ops->enable_irq_window(vcpu); if (kvm_lapic_enabled(vcpu)) { - /* -* Update architecture specific hints for APIC -* virtual interrupt delivery. -*/ - if (kvm_x86_ops->hwapic_irr_update) - kvm_x86_ops->hwapic_irr_update(vcpu, - kvm_lapic_find_highest_irr(vcpu)); update_cr8_intercept(vcpu); kvm_lapic_sync_to_vapic(vcpu); } It may be obsolete now that we have the patch from Radim to set KVM_REQ_EVENT in vmx_sync_pir_to_irr (http://permalink.gmane.org/gmane.linux.kernel/2057138). Thanks, Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked
On Wed, Oct 14, 2015 at 6:33 PM, Wu, Feng wrote: > >> -Original Message- >> From: David Matlack [mailto:dmatl...@google.com] >> Sent: Thursday, October 15, 2015 7:41 AM >> To: Wu, Feng >> Cc: Paolo Bonzini ; alex.william...@redhat.com; Joerg >> Roedel ; Marcelo Tosatti ; >> eric.au...@linaro.org; kvm list ; iommu@lists.linux- >> foundation.org; linux-ker...@vger.kernel.org >> Subject: Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when >> vCPU is blocked >> >> Hi Feng. >> >> On Fri, Sep 18, 2015 at 7:29 AM, Feng Wu wrote: >> > This patch updates the Posted-Interrupts Descriptor when vCPU >> > is blocked. >> > >> > pre-block: >> > - Add the vCPU to the blocked per-CPU list >> > - Set 'NV' to POSTED_INTR_WAKEUP_VECTOR >> > >> > post-block: >> > - Remove the vCPU from the per-CPU list >> >> I'm wondering what happens if a posted interrupt arrives at the IOMMU >> after pre-block and before post-block. >> >> In pre_block, NV is set to POSTED_INTR_WAKEUP_VECTOR. IIUC, this means >> future posted interrupts will not trigger "Posted-Interrupt Processing" >> (PIR will not get copied to VIRR). Instead, the IOMMU will do ON := 1, >> PIR |= (1 << vector), and send POSTED_INTR_WAKEUP_VECTOR. PIWV calls >> wakeup_handler which does kvm_vcpu_kick. kvm_vcpu_kick does a wait-queue >> wakeup and possibly a scheduler ipi. >> >> But the VCPU is sitting in kvm_vcpu_block. It spins and/or schedules >> (wait queue) until it has a reason to wake up. I couldn't find a code >> path from kvm_vcpu_block that lead to checking ON or PIR. How does the >> blocked VCPU "receive" the posted interrupt? (And when does Posted- >> Interrupt Processing get triggered?) > > In the pre_block, it also change the 'NDST' filed to the pCPU, on which the > vCPU > is put to the per-CPU list 'blocked_vcpu_on_cpu', so when posted-interrupts > come it, it will sent the wakeup notification event to the pCPU above, then in > the wakeup_handler, it can find the vCPU from the per-CPU list, hence > kvm_vcpu_kick can wake up it. Thank you for your response. I was actually confused about something else. After wakeup_handler->kvm_vcpu_kick causes the vcpu to wake up, that vcpu calls kvm_vcpu_check_block() to check if there are pending events, otherwise the vcpu goes back to sleep. I had trouble yesterday finding the code path from kvm_vcpu_check_block() which checks PIR/ON. But after spending more time reading the source code this morning I found that kvm_vcpu_check_block() eventually calls into vmx_sync_pir_to_irr(), which copies PIR to IRR and clears ON. And then apic_find_highest_irr() detects the pending posted interrupt. > > Thanks, > Feng > >> >> Thanks! >> >> > >> > Signed-off-by: Feng Wu >> > --- >> > v9: >> > - Add description for blocked_vcpu_on_cpu_lock in >> Documentation/virtual/kvm/locking.txt >> > - Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then >> > !irq_remapping_cap(IRQ_POSTING_CAP) >> > >> > v8: >> > - Rename 'pi_pre_block' to 'pre_block' >> > - Rename 'pi_post_block' to 'post_block' >> > - Change some comments >> > - Only add the vCPU to the blocking list when the VM has assigned devices. >> > >> > Documentation/virtual/kvm/locking.txt | 12 +++ >> > arch/x86/include/asm/kvm_host.h | 13 +++ >> > arch/x86/kvm/vmx.c| 153 >> ++ >> > arch/x86/kvm/x86.c| 53 +--- >> > include/linux/kvm_host.h | 3 + >> > virt/kvm/kvm_main.c | 3 + >> > 6 files changed, 227 insertions(+), 10 deletions(-) >> > >> > diff --git a/Documentation/virtual/kvm/locking.txt >> b/Documentation/virtual/kvm/locking.txt >> > index d68af4d..19f94a6 100644 >> > --- a/Documentation/virtual/kvm/locking.txt >> > +++ b/Documentation/virtual/kvm/locking.txt >> > @@ -166,3 +166,15 @@ Comment: The srcu read lock must be held while >> accessing memslots (e.g. >> > MMIO/PIO address->device structure mapping (kvm->buses). >> > The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu >> > if it is needed by multiple functions. >> > + >> > +Name: blocked_vcpu_on_cpu_lock >> > +Type: spinlock_t >> > +Arch
RE: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked
> -Original Message- > From: David Matlack [mailto:dmatl...@google.com] > Sent: Thursday, October 15, 2015 7:41 AM > To: Wu, Feng > Cc: Paolo Bonzini ; alex.william...@redhat.com; Joerg > Roedel ; Marcelo Tosatti ; > eric.au...@linaro.org; kvm list ; iommu@lists.linux- > foundation.org; linux-ker...@vger.kernel.org > Subject: Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when > vCPU is blocked > > Hi Feng. > > On Fri, Sep 18, 2015 at 7:29 AM, Feng Wu wrote: > > This patch updates the Posted-Interrupts Descriptor when vCPU > > is blocked. > > > > pre-block: > > - Add the vCPU to the blocked per-CPU list > > - Set 'NV' to POSTED_INTR_WAKEUP_VECTOR > > > > post-block: > > - Remove the vCPU from the per-CPU list > > I'm wondering what happens if a posted interrupt arrives at the IOMMU > after pre-block and before post-block. > > In pre_block, NV is set to POSTED_INTR_WAKEUP_VECTOR. IIUC, this means > future posted interrupts will not trigger "Posted-Interrupt Processing" > (PIR will not get copied to VIRR). Instead, the IOMMU will do ON := 1, > PIR |= (1 << vector), and send POSTED_INTR_WAKEUP_VECTOR. PIWV calls > wakeup_handler which does kvm_vcpu_kick. kvm_vcpu_kick does a wait-queue > wakeup and possibly a scheduler ipi. > > But the VCPU is sitting in kvm_vcpu_block. It spins and/or schedules > (wait queue) until it has a reason to wake up. I couldn't find a code > path from kvm_vcpu_block that lead to checking ON or PIR. How does the > blocked VCPU "receive" the posted interrupt? (And when does Posted- > Interrupt Processing get triggered?) In the pre_block, it also change the 'NDST' filed to the pCPU, on which the vCPU is put to the per-CPU list 'blocked_vcpu_on_cpu', so when posted-interrupts come it, it will sent the wakeup notification event to the pCPU above, then in the wakeup_handler, it can find the vCPU from the per-CPU list, hence kvm_vcpu_kick can wake up it. Thanks, Feng > > Thanks! > > > > > Signed-off-by: Feng Wu > > --- > > v9: > > - Add description for blocked_vcpu_on_cpu_lock in > Documentation/virtual/kvm/locking.txt > > - Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then > > !irq_remapping_cap(IRQ_POSTING_CAP) > > > > v8: > > - Rename 'pi_pre_block' to 'pre_block' > > - Rename 'pi_post_block' to 'post_block' > > - Change some comments > > - Only add the vCPU to the blocking list when the VM has assigned devices. > > > > Documentation/virtual/kvm/locking.txt | 12 +++ > > arch/x86/include/asm/kvm_host.h | 13 +++ > > arch/x86/kvm/vmx.c| 153 > ++ > > arch/x86/kvm/x86.c| 53 +--- > > include/linux/kvm_host.h | 3 + > > virt/kvm/kvm_main.c | 3 + > > 6 files changed, 227 insertions(+), 10 deletions(-) > > > > diff --git a/Documentation/virtual/kvm/locking.txt > b/Documentation/virtual/kvm/locking.txt > > index d68af4d..19f94a6 100644 > > --- a/Documentation/virtual/kvm/locking.txt > > +++ b/Documentation/virtual/kvm/locking.txt > > @@ -166,3 +166,15 @@ Comment: The srcu read lock must be held while > accessing memslots (e.g. > > MMIO/PIO address->device structure mapping (kvm->buses). > > The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu > > if it is needed by multiple functions. > > + > > +Name: blocked_vcpu_on_cpu_lock > > +Type: spinlock_t > > +Arch: x86 > > +Protects: blocked_vcpu_on_cpu > > +Comment: This is a per-CPU lock and it is used for VT-d > > posted-interrupts. > > + When VT-d posted-interrupts is supported and the VM has > > assigned > > + devices, we put the blocked vCPU on the list > > blocked_vcpu_on_cpu > > + protected by blocked_vcpu_on_cpu_lock, when VT-d hardware > issues > > + wakeup notification event since external interrupts from the > > + assigned devices happens, we will find the vCPU on the list > > to > > + wakeup. > > diff --git a/arch/x86/include/asm/kvm_host.h > b/arch/x86/include/asm/kvm_host.h > > index 0ddd353..304fbb5 100644 > > --- a/arch/x86/include/asm/kvm_host.h > > +++ b/arch/x86/include/asm/kvm_host.h > > @@ -552,6 +552,8 @@ struct kvm_vcpu_arch { > > */ > > bool write_fault_to_
Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked
Hi Feng. On Fri, Sep 18, 2015 at 7:29 AM, Feng Wu wrote: > This patch updates the Posted-Interrupts Descriptor when vCPU > is blocked. > > pre-block: > - Add the vCPU to the blocked per-CPU list > - Set 'NV' to POSTED_INTR_WAKEUP_VECTOR > > post-block: > - Remove the vCPU from the per-CPU list I'm wondering what happens if a posted interrupt arrives at the IOMMU after pre-block and before post-block. In pre_block, NV is set to POSTED_INTR_WAKEUP_VECTOR. IIUC, this means future posted interrupts will not trigger "Posted-Interrupt Processing" (PIR will not get copied to VIRR). Instead, the IOMMU will do ON := 1, PIR |= (1 << vector), and send POSTED_INTR_WAKEUP_VECTOR. PIWV calls wakeup_handler which does kvm_vcpu_kick. kvm_vcpu_kick does a wait-queue wakeup and possibly a scheduler ipi. But the VCPU is sitting in kvm_vcpu_block. It spins and/or schedules (wait queue) until it has a reason to wake up. I couldn't find a code path from kvm_vcpu_block that lead to checking ON or PIR. How does the blocked VCPU "receive" the posted interrupt? (And when does Posted- Interrupt Processing get triggered?) Thanks! > > Signed-off-by: Feng Wu > --- > v9: > - Add description for blocked_vcpu_on_cpu_lock in > Documentation/virtual/kvm/locking.txt > - Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then > !irq_remapping_cap(IRQ_POSTING_CAP) > > v8: > - Rename 'pi_pre_block' to 'pre_block' > - Rename 'pi_post_block' to 'post_block' > - Change some comments > - Only add the vCPU to the blocking list when the VM has assigned devices. > > Documentation/virtual/kvm/locking.txt | 12 +++ > arch/x86/include/asm/kvm_host.h | 13 +++ > arch/x86/kvm/vmx.c| 153 > ++ > arch/x86/kvm/x86.c| 53 +--- > include/linux/kvm_host.h | 3 + > virt/kvm/kvm_main.c | 3 + > 6 files changed, 227 insertions(+), 10 deletions(-) > > diff --git a/Documentation/virtual/kvm/locking.txt > b/Documentation/virtual/kvm/locking.txt > index d68af4d..19f94a6 100644 > --- a/Documentation/virtual/kvm/locking.txt > +++ b/Documentation/virtual/kvm/locking.txt > @@ -166,3 +166,15 @@ Comment: The srcu read lock must be held while > accessing memslots (e.g. > MMIO/PIO address->device structure mapping (kvm->buses). > The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu > if it is needed by multiple functions. > + > +Name: blocked_vcpu_on_cpu_lock > +Type: spinlock_t > +Arch: x86 > +Protects: blocked_vcpu_on_cpu > +Comment: This is a per-CPU lock and it is used for VT-d > posted-interrupts. > + When VT-d posted-interrupts is supported and the VM has > assigned > + devices, we put the blocked vCPU on the list > blocked_vcpu_on_cpu > + protected by blocked_vcpu_on_cpu_lock, when VT-d hardware > issues > + wakeup notification event since external interrupts from the > + assigned devices happens, we will find the vCPU on the list to > + wakeup. > diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h > index 0ddd353..304fbb5 100644 > --- a/arch/x86/include/asm/kvm_host.h > +++ b/arch/x86/include/asm/kvm_host.h > @@ -552,6 +552,8 @@ struct kvm_vcpu_arch { > */ > bool write_fault_to_shadow_pgtable; > > + bool halted; > + > /* set at EPT violation at this point */ > unsigned long exit_qualification; > > @@ -864,6 +866,17 @@ struct kvm_x86_ops { > /* pmu operations of sub-arch */ > const struct kvm_pmu_ops *pmu_ops; > > + /* > +* Architecture specific hooks for vCPU blocking due to > +* HLT instruction. > +* Returns for .pre_block(): > +*- 0 means continue to block the vCPU. > +*- 1 means we cannot block the vCPU since some event > +*happens during this period, such as, 'ON' bit in > +*posted-interrupts descriptor is set. > +*/ > + int (*pre_block)(struct kvm_vcpu *vcpu); > + void (*post_block)(struct kvm_vcpu *vcpu); > int (*update_pi_irte)(struct kvm *kvm, unsigned int host_irq, > uint32_t guest_irq, bool set); > }; > diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c > index 902a67d..9968896 100644 > --- a/arch/x86/kvm/vmx.c > +++ b/arch/x86/kvm/vmx.c > @@ -879,6 +879,13 @@ static DEFINE_PER_CPU(struct vmcs *, current_vmcs); > static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu); > static DEFINE_PER_CPU(struct desc_ptr, host_gdt); > > +/* > + * We maintian a per-CPU linked-list of vCPU, so in wakeup_handler() we > + * can find which vCPU should be waken up. > + */ > +static DEFINE_PER_CPU(struct list_head, blocked_vcpu_on_cpu); > +static DEFINE_PER_CPU(spinlock_t, blocked_vcpu_on_cpu_lock); > + > static unsigned long *
RE: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked
> -Original Message- > From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo > Bonzini > Sent: Monday, September 21, 2015 1:33 PM > To: Wu, Feng; Alex Williamson; j...@8bytes.org; Marcelo Tosatti > Cc: io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list; > Eric Auger > Subject: Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when > vCPU is blocked > > > > On 21/09/2015 04:16, Wu, Feng wrote: > > I tested the above patch you suggested, it works fine. Thank you! So > > do I need to resend a new version or you can handle it in your tree? > > I will handle it. Thanks a lot for your review on this series! Thanks, Feng > > Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked
On 21/09/2015 04:16, Wu, Feng wrote: > I tested the above patch you suggested, it works fine. Thank you! So > do I need to resend a new version or you can handle it in your tree? I will handle it. Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked
> -Original Message- > From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo > Bonzini > Sent: Saturday, September 19, 2015 12:07 AM > To: Wu, Feng; Alex Williamson; j...@8bytes.org; Marcelo Tosatti > Cc: io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list; > Eric Auger > Subject: Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when > vCPU is blocked > > > > On 18/09/2015 16:29, Feng Wu wrote: > > This patch updates the Posted-Interrupts Descriptor when vCPU > > is blocked. > > > > pre-block: > > - Add the vCPU to the blocked per-CPU list > > - Set 'NV' to POSTED_INTR_WAKEUP_VECTOR > > > > post-block: > > - Remove the vCPU from the per-CPU list > > > > Signed-off-by: Feng Wu > > > --- > > v9: > > - Add description for blocked_vcpu_on_cpu_lock in > Documentation/virtual/kvm/locking.txt > > - Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then > > !irq_remapping_cap(IRQ_POSTING_CAP) > > > > v8: > > - Rename 'pi_pre_block' to 'pre_block' > > - Rename 'pi_post_block' to 'post_block' > > - Change some comments > > - Only add the vCPU to the blocking list when the VM has assigned devices. > > > > Documentation/virtual/kvm/locking.txt | 12 +++ > > arch/x86/include/asm/kvm_host.h | 13 +++ > > arch/x86/kvm/vmx.c| 153 > ++ > > arch/x86/kvm/x86.c| 53 +--- > > include/linux/kvm_host.h | 3 + > > virt/kvm/kvm_main.c | 3 + > > 6 files changed, 227 insertions(+), 10 deletions(-) > > > > diff --git a/Documentation/virtual/kvm/locking.txt > b/Documentation/virtual/kvm/locking.txt > > index d68af4d..19f94a6 100644 > > --- a/Documentation/virtual/kvm/locking.txt > > +++ b/Documentation/virtual/kvm/locking.txt > > @@ -166,3 +166,15 @@ Comment: The srcu read lock must be held while > accessing memslots (e.g. > > MMIO/PIO address->device structure mapping (kvm->buses). > > The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu > > if it is needed by multiple functions. > > + > > +Name: blocked_vcpu_on_cpu_lock > > +Type: spinlock_t > > +Arch: x86 > > +Protects: blocked_vcpu_on_cpu > > +Comment: This is a per-CPU lock and it is used for VT-d > > posted-interrupts. > > + When VT-d posted-interrupts is supported and the VM has assigned > > + devices, we put the blocked vCPU on the list blocked_vcpu_on_cpu > > + protected by blocked_vcpu_on_cpu_lock, when VT-d hardware issues > > + wakeup notification event since external interrupts from the > > + assigned devices happens, we will find the vCPU on the list to > > + wakeup. > > diff --git a/arch/x86/include/asm/kvm_host.h > b/arch/x86/include/asm/kvm_host.h > > index 0ddd353..304fbb5 100644 > > --- a/arch/x86/include/asm/kvm_host.h > > +++ b/arch/x86/include/asm/kvm_host.h > > @@ -552,6 +552,8 @@ struct kvm_vcpu_arch { > > */ > > bool write_fault_to_shadow_pgtable; > > > > + bool halted; > > + > > /* set at EPT violation at this point */ > > unsigned long exit_qualification; > > > > @@ -864,6 +866,17 @@ struct kvm_x86_ops { > > /* pmu operations of sub-arch */ > > const struct kvm_pmu_ops *pmu_ops; > > > > + /* > > +* Architecture specific hooks for vCPU blocking due to > > +* HLT instruction. > > +* Returns for .pre_block(): > > +*- 0 means continue to block the vCPU. > > +*- 1 means we cannot block the vCPU since some event > > +*happens during this period, such as, 'ON' bit in > > +*posted-interrupts descriptor is set. > > +*/ > > + int (*pre_block)(struct kvm_vcpu *vcpu); > > + void (*post_block)(struct kvm_vcpu *vcpu); > > int (*update_pi_irte)(struct kvm *kvm, unsigned int host_irq, > > uint32_t guest_irq, bool set); > > }; > > diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c > > index 902a67d..9968896 100644 > > --- a/arch/x86/kvm/vmx.c > > +++ b/arch/x86/kvm/vmx.c > > @@ -879,6 +879,13 @@ static DEFINE_PER_CPU(struct vmcs *, > current_vmcs); > > static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu); > > static DEFI
RE: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked
> -Original Message- > From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo > Bonzini > Sent: Saturday, September 19, 2015 12:07 AM > To: Wu, Feng; Alex Williamson; j...@8bytes.org; Marcelo Tosatti > Cc: io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list; > Eric Auger > Subject: Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when > vCPU is blocked > > > > On 18/09/2015 16:29, Feng Wu wrote: > > This patch updates the Posted-Interrupts Descriptor when vCPU > > is blocked. > > > > pre-block: > > - Add the vCPU to the blocked per-CPU list > > - Set 'NV' to POSTED_INTR_WAKEUP_VECTOR > > > > post-block: > > - Remove the vCPU from the per-CPU list > > > > Signed-off-by: Feng Wu > > > --- > > v9: > > - Add description for blocked_vcpu_on_cpu_lock in > Documentation/virtual/kvm/locking.txt > > - Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then > > !irq_remapping_cap(IRQ_POSTING_CAP) > > > > v8: > > - Rename 'pi_pre_block' to 'pre_block' > > - Rename 'pi_post_block' to 'post_block' > > - Change some comments > > - Only add the vCPU to the blocking list when the VM has assigned devices. > > > > Documentation/virtual/kvm/locking.txt | 12 +++ > > arch/x86/include/asm/kvm_host.h | 13 +++ > > arch/x86/kvm/vmx.c| 153 > ++ > > arch/x86/kvm/x86.c| 53 +--- > > include/linux/kvm_host.h | 3 + > > virt/kvm/kvm_main.c | 3 + > > 6 files changed, 227 insertions(+), 10 deletions(-) > > > > diff --git a/Documentation/virtual/kvm/locking.txt > b/Documentation/virtual/kvm/locking.txt > > index d68af4d..19f94a6 100644 > > --- a/Documentation/virtual/kvm/locking.txt > > +++ b/Documentation/virtual/kvm/locking.txt > > @@ -166,3 +166,15 @@ Comment: The srcu read lock must be held while > accessing memslots (e.g. > > MMIO/PIO address->device structure mapping (kvm->buses). > > The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu > > if it is needed by multiple functions. > > + > > +Name: blocked_vcpu_on_cpu_lock > > +Type: spinlock_t > > +Arch: x86 > > +Protects: blocked_vcpu_on_cpu > > +Comment: This is a per-CPU lock and it is used for VT-d > > posted-interrupts. > > + When VT-d posted-interrupts is supported and the VM has assigned > > + devices, we put the blocked vCPU on the list blocked_vcpu_on_cpu > > + protected by blocked_vcpu_on_cpu_lock, when VT-d hardware issues > > + wakeup notification event since external interrupts from the > > + assigned devices happens, we will find the vCPU on the list to > > + wakeup. > > diff --git a/arch/x86/include/asm/kvm_host.h > b/arch/x86/include/asm/kvm_host.h > > index 0ddd353..304fbb5 100644 > > --- a/arch/x86/include/asm/kvm_host.h > > +++ b/arch/x86/include/asm/kvm_host.h > > @@ -552,6 +552,8 @@ struct kvm_vcpu_arch { > > */ > > bool write_fault_to_shadow_pgtable; > > > > + bool halted; > > + > > /* set at EPT violation at this point */ > > unsigned long exit_qualification; > > > > @@ -864,6 +866,17 @@ struct kvm_x86_ops { > > /* pmu operations of sub-arch */ > > const struct kvm_pmu_ops *pmu_ops; > > > > + /* > > +* Architecture specific hooks for vCPU blocking due to > > +* HLT instruction. > > +* Returns for .pre_block(): > > +*- 0 means continue to block the vCPU. > > +*- 1 means we cannot block the vCPU since some event > > +*happens during this period, such as, 'ON' bit in > > +*posted-interrupts descriptor is set. > > +*/ > > + int (*pre_block)(struct kvm_vcpu *vcpu); > > + void (*post_block)(struct kvm_vcpu *vcpu); > > int (*update_pi_irte)(struct kvm *kvm, unsigned int host_irq, > > uint32_t guest_irq, bool set); > > }; > > diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c > > index 902a67d..9968896 100644 > > --- a/arch/x86/kvm/vmx.c > > +++ b/arch/x86/kvm/vmx.c > > @@ -879,6 +879,13 @@ static DEFINE_PER_CPU(struct vmcs *, > current_vmcs); > > static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu); > > static DEFI
Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked
On 18/09/2015 16:29, Feng Wu wrote: > This patch updates the Posted-Interrupts Descriptor when vCPU > is blocked. > > pre-block: > - Add the vCPU to the blocked per-CPU list > - Set 'NV' to POSTED_INTR_WAKEUP_VECTOR > > post-block: > - Remove the vCPU from the per-CPU list > > Signed-off-by: Feng Wu > --- > v9: > - Add description for blocked_vcpu_on_cpu_lock in > Documentation/virtual/kvm/locking.txt > - Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then > !irq_remapping_cap(IRQ_POSTING_CAP) > > v8: > - Rename 'pi_pre_block' to 'pre_block' > - Rename 'pi_post_block' to 'post_block' > - Change some comments > - Only add the vCPU to the blocking list when the VM has assigned devices. > > Documentation/virtual/kvm/locking.txt | 12 +++ > arch/x86/include/asm/kvm_host.h | 13 +++ > arch/x86/kvm/vmx.c| 153 > ++ > arch/x86/kvm/x86.c| 53 +--- > include/linux/kvm_host.h | 3 + > virt/kvm/kvm_main.c | 3 + > 6 files changed, 227 insertions(+), 10 deletions(-) > > diff --git a/Documentation/virtual/kvm/locking.txt > b/Documentation/virtual/kvm/locking.txt > index d68af4d..19f94a6 100644 > --- a/Documentation/virtual/kvm/locking.txt > +++ b/Documentation/virtual/kvm/locking.txt > @@ -166,3 +166,15 @@ Comment: The srcu read lock must be held while accessing > memslots (e.g. > MMIO/PIO address->device structure mapping (kvm->buses). > The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu > if it is needed by multiple functions. > + > +Name:blocked_vcpu_on_cpu_lock > +Type:spinlock_t > +Arch:x86 > +Protects:blocked_vcpu_on_cpu > +Comment: This is a per-CPU lock and it is used for VT-d > posted-interrupts. > + When VT-d posted-interrupts is supported and the VM has assigned > + devices, we put the blocked vCPU on the list blocked_vcpu_on_cpu > + protected by blocked_vcpu_on_cpu_lock, when VT-d hardware issues > + wakeup notification event since external interrupts from the > + assigned devices happens, we will find the vCPU on the list to > + wakeup. > diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h > index 0ddd353..304fbb5 100644 > --- a/arch/x86/include/asm/kvm_host.h > +++ b/arch/x86/include/asm/kvm_host.h > @@ -552,6 +552,8 @@ struct kvm_vcpu_arch { >*/ > bool write_fault_to_shadow_pgtable; > > + bool halted; > + > /* set at EPT violation at this point */ > unsigned long exit_qualification; > > @@ -864,6 +866,17 @@ struct kvm_x86_ops { > /* pmu operations of sub-arch */ > const struct kvm_pmu_ops *pmu_ops; > > + /* > + * Architecture specific hooks for vCPU blocking due to > + * HLT instruction. > + * Returns for .pre_block(): > + *- 0 means continue to block the vCPU. > + *- 1 means we cannot block the vCPU since some event > + *happens during this period, such as, 'ON' bit in > + *posted-interrupts descriptor is set. > + */ > + int (*pre_block)(struct kvm_vcpu *vcpu); > + void (*post_block)(struct kvm_vcpu *vcpu); > int (*update_pi_irte)(struct kvm *kvm, unsigned int host_irq, > uint32_t guest_irq, bool set); > }; > diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c > index 902a67d..9968896 100644 > --- a/arch/x86/kvm/vmx.c > +++ b/arch/x86/kvm/vmx.c > @@ -879,6 +879,13 @@ static DEFINE_PER_CPU(struct vmcs *, current_vmcs); > static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu); > static DEFINE_PER_CPU(struct desc_ptr, host_gdt); > > +/* > + * We maintian a per-CPU linked-list of vCPU, so in wakeup_handler() we > + * can find which vCPU should be waken up. > + */ > +static DEFINE_PER_CPU(struct list_head, blocked_vcpu_on_cpu); > +static DEFINE_PER_CPU(spinlock_t, blocked_vcpu_on_cpu_lock); > + > static unsigned long *vmx_io_bitmap_a; > static unsigned long *vmx_io_bitmap_b; > static unsigned long *vmx_msr_bitmap_legacy; > @@ -2985,6 +2992,8 @@ static int hardware_enable(void) > return -EBUSY; > > INIT_LIST_HEAD(&per_cpu(loaded_vmcss_on_cpu, cpu)); > + INIT_LIST_HEAD(&per_cpu(blocked_vcpu_on_cpu, cpu)); > + spin_lock_init(&per_cpu(blocked_vcpu_on_cpu_lock, cpu)); > > /* >* Now we can enable the vmclear operation in kdump > @@ -6121,6 +6130,25 @@ static void update_ple_window_actual_max(void) > ple_window_grow, INT_MIN); > } > > +/* > + * Handler for POSTED_INTERRUPT_WAKEUP_VECTOR. > + */ > +static void wakeup_handler(void) > +{ > + struct kvm_vcpu *vcpu; > + int cpu = smp_processor_id(); > + > + spin_lock(&per_cpu(blocked_vcpu_on_cpu_lock, cpu)); > + list_for_each_entry(vcpu, &per_cpu(b