Re: [Xen-devel] [PATCH v9 01/17] VT-d Posted-intterrupt (PI) design

[Tian, Kevin]

> From: Dario Faggioli [mailto:dario.faggi...@citrix.com]
> Sent: Tuesday, November 10, 2015 10:52 PM
> >
> > +Here are what we do for the blocked vCPU:
> > +1. Define a per-cpu list 'pi_blocked_vcpu', which stored the blocked
> > +vCPU on the pCPU.
> > +2. When the vCPU is going to block, insert the vCPU
> > +to the per-cpu list belonging to the pCPU it was running.
> > +3. When the vCPU is unblocked, remove the vCPU from the related pCPU
> > list.
> > +
> > +In the handler of 'pi_wakeup_vector', we do:
> > +1. Get the physical CPU.
> > +2. Iterate the list 'pi_blocked_vcpu' of the current pCPU, if 'ON'
> > is set,
> > +we unblock the associated vCPU.
> > +
> This is indeed more general, and the fact that it does no longer
> mentions RUNSTATEs, makes it more adherent to the actual implementation
> (and hence more useful), so thanks for doing this.
> 
> Personally, I'd add a quick comment about how this, despite being
> related to blocking and unblocking, happens actually inside VMX
> handlers, i.e., (quickly) what is the relationship within these sets of
> events (blocking/unblocking VMENTER/EXIT) and why it is ok to do things
> like they are done.
> 
> I'm not too fussed about this, though. So, if others don't think
> something like this is necessary, I'm fine with what you have here.
> 

That type of information is welcomed.

Thanks
Kevin
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v9 01/17] VT-d Posted-intterrupt (PI) design

[Tian, Kevin]

> From: Wu, Feng
> Sent: Tuesday, November 03, 2015 4:43 PM
> 
> 
> diff --git a/docs/misc/vtd-pi.txt b/docs/misc/vtd-pi.txt
> new file mode 100644
> index 000..f9b4637
> --- /dev/null
> +++ b/docs/misc/vtd-pi.txt
> @@ -0,0 +1,329 @@
> +Authors: Feng Wu 
> +
> +VT-d Posted-interrupt (PI) design for XEN
> +
> +Background
> +==
> +With the development of virtualization, there are more and more device
> +assignment requirements. However, today when a VM is running with
> +assigned devices (such as, NIC), external interrupt handling for the assigned
> +devices always needs VMM intervention.
> +
> +VT-d Posted-interrupt is a more enhanced method to handle interrupts
> +in the virtualization environment. Interrupt posting is the process by
> +which an interrupt request is recorded in a memory-resident
> +posted-interrupt-descriptor structure by the root-complex, followed by
> +an optional notification event issued to the CPU complex.

Some clarification required here. Is "Interrupt Posting" only used to
represent the process of VT-d Posted-interrupt, or also for CPU-side
posting through IPI? From above context, and later explanation about
"Processor Support" and "Root-complex Support", looks the former
is true. Then how do we call CPU-side posting?

I think a one-sentence definition about those terms in the start would
be helpful.

> +
> +With VT-d Posted-interrupt we can get the following advantages:
> +- Direct delivery of external interrupts to running vCPUs without VMM
> +intervention
> +- Decrease the interrupt migration complexity. On vCPU migration, software
> +can atomically co-migrate all interrupts targeting the migrating vCPU. For
> +virtual machines with assigned devices, migrating a vCPU across pCPUs
> +either incurs the overhead of forwarding interrupts in software (e.g. via VMM
> +generated IPIs), or complexity to independently migrate each interrupt 
> targeting
> +the vCPU to the new pCPU. However, after enabling VT-d PI, the destination 
> vCPU
> +of an external interrupt from assigned devices is stored in the IRTE (i.e.
> +Posted-interrupt Descriptor Address), when vCPU is migrated to another pCPU,
> +we will set this new pCPU in the 'NDST' filed of Posted-interrupt 
> descriptor, this
> +make the interrupt migration automatic.
> +
> +Here is what Xen currently does for external interrupts from assigned 
> devices:
> +
> +When a VM is running and an external interrupt from an assigned device occurs
> +for it. VM-EXIT happens, then:
> +
> +vmx_do_extint() --> do_IRQ() --> __do_IRQ_guest() --> hvm_do_IRQ_dpci() -->
> +raise_softirq_for(pirq_dpci) --> raise_softirq(HVM_DPCI_SOFTIRQ)
> +
> +softirq HVM_DPCI_SOFTIRQ is bound to dpci_softirq()
> +
> +dpci_softirq() --> hvm_dirq_assist() --> vmsi_deliver_pirq() --> 
> vmsi_deliver() -->
> +vmsi_inj_irq() --> vlapic_set_irq()
> +
> +vlapic_set_irq() does the following things:
> +1. If CPU-side posted-interrupt is supported, call vmx_deliver_posted_intr() 
> to deliver
> +the virtual interrupt via posted-interrupt infrastructure.
> +2. Else if CPU-side posted-interrupt is not supported, set the related vIRR 
> in vLAPIC
> +page and call vcpu_kick() to kick the related vCPU. Before VM-Entry, 
> vmx_intr_assist()
> +will help to inject the interrupt to guests.
> +
> +However, after VT-d PI is supported, when a guest is running in non-root and 
> an
> +external interrupt from an assigned device occurs for it. No VM-Exit is 
> needed,

". No VM-Exit ..." -> ", no VM-Exit..."

> +the guest can handle this totally in non-root mode, thus avoiding all the 
> above
> +code flow.
> +
> +Posted-interrupt Introduction
> +
> +There are two components to the Posted-interrupt architecture:

"to the" -> "in the"


Thanks
Kevin

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v9 01/17] VT-d Posted-intterrupt (PI) design

[Wu, Feng]

> -Original Message-
> From: Tian, Kevin
> Sent: Tuesday, November 24, 2015 2:35 PM
> To: Wu, Feng ; xen-devel@lists.xen.org
> Cc: Zhang, Yang Z ; Jan Beulich
> ; Keir Fraser ; Andrew Cooper
> ; George Dunlap
> 
> Subject: RE: [PATCH v9 01/17] VT-d Posted-intterrupt (PI) design
> 
> > From: Wu, Feng
> > Sent: Tuesday, November 03, 2015 4:43 PM
> >
> >
> > diff --git a/docs/misc/vtd-pi.txt b/docs/misc/vtd-pi.txt
> > new file mode 100644
> > index 000..f9b4637
> > --- /dev/null
> > +++ b/docs/misc/vtd-pi.txt
> > @@ -0,0 +1,329 @@
> > +Authors: Feng Wu 
> > +
> > +VT-d Posted-interrupt (PI) design for XEN
> > +
> > +Background
> > +==
> > +With the development of virtualization, there are more and more device
> > +assignment requirements. However, today when a VM is running with
> > +assigned devices (such as, NIC), external interrupt handling for the
> assigned
> > +devices always needs VMM intervention.
> > +
> > +VT-d Posted-interrupt is a more enhanced method to handle interrupts
> > +in the virtualization environment. Interrupt posting is the process by
> > +which an interrupt request is recorded in a memory-resident
> > +posted-interrupt-descriptor structure by the root-complex, followed by
> > +an optional notification event issued to the CPU complex.
> 
> Some clarification required here. Is "Interrupt Posting" only used to
> represent the process of VT-d Posted-interrupt, or also for CPU-side
> posting through IPI? From above context, and later explanation about
> "Processor Support" and "Root-complex Support", looks the former
> is true. Then how do we call CPU-side posting?

Thanks for the comments. "Interrupt posting" in fact contains vt-d side
and cpu side. So I will reword this to make it clear.

Thanks,
Feng

> 
> I think a one-sentence definition about those terms in the start would
> be helpful.
> 
> > +
> > +With VT-d Posted-interrupt we can get the following advantages:
> > +- Direct delivery of external interrupts to running vCPUs without VMM
> > +intervention
> > +- Decrease the interrupt migration complexity. On vCPU migration,
> software
> > +can atomically co-migrate all interrupts targeting the migrating vCPU. For
> > +virtual machines with assigned devices, migrating a vCPU across pCPUs
> > +either incurs the overhead of forwarding interrupts in software (e.g. via
> VMM
> > +generated IPIs), or complexity to independently migrate each interrupt
> targeting
> > +the vCPU to the new pCPU. However, after enabling VT-d PI, the
> destination vCPU
> > +of an external interrupt from assigned devices is stored in the IRTE (i.e.
> > +Posted-interrupt Descriptor Address), when vCPU is migrated to another
> pCPU,
> > +we will set this new pCPU in the 'NDST' filed of Posted-interrupt
> descriptor, this
> > +make the interrupt migration automatic.
> > +
> > +Here is what Xen currently does for external interrupts from assigned
> devices:
> > +
> > +When a VM is running and an external interrupt from an assigned device
> occurs
> > +for it. VM-EXIT happens, then:
> > +
> > +vmx_do_extint() --> do_IRQ() --> __do_IRQ_guest() --> hvm_do_IRQ_dpci()
> -->
> > +raise_softirq_for(pirq_dpci) --> raise_softirq(HVM_DPCI_SOFTIRQ)
> > +
> > +softirq HVM_DPCI_SOFTIRQ is bound to dpci_softirq()
> > +
> > +dpci_softirq() --> hvm_dirq_assist() --> vmsi_deliver_pirq() -->
> vmsi_deliver() -->
> > +vmsi_inj_irq() --> vlapic_set_irq()
> > +
> > +vlapic_set_irq() does the following things:
> > +1. If CPU-side posted-interrupt is supported, call
> vmx_deliver_posted_intr() to deliver
> > +the virtual interrupt via posted-interrupt infrastructure.
> > +2. Else if CPU-side posted-interrupt is not supported, set the related vIRR
> in vLAPIC
> > +page and call vcpu_kick() to kick the related vCPU. Before VM-Entry,
> vmx_intr_assist()
> > +will help to inject the interrupt to guests.
> > +
> > +However, after VT-d PI is supported, when a guest is running in non-root
> and an
> > +external interrupt from an assigned device occurs for it. No VM-Exit is
> needed,
> 
> ". No VM-Exit ..." -> ", no VM-Exit..."
> 
> > +the guest can handle this totally in non-root mode, thus avoiding all the
> above
> > +code flow.
> > +
> > +Posted-interrupt Introduction
> > +
> > +There are two components to the Posted-interrupt architecture:
> 
> "to the" -> "in the"
> 
> 
> Thanks
> Kevin

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v9 01/17] VT-d Posted-intterrupt (PI) design

[Dario Faggioli]

On Tue, 2015-11-03 at 16:43 +0800, Feng Wu wrote:

> diff --git a/docs/misc/vtd-pi.txt b/docs/misc/vtd-pi.txt
> new file mode 100644
> index 000..f9b4637
> --- /dev/null
> +++ b/docs/misc/vtd-pi.txt

> +With VT-d Posted-interrupt we can get the following advantages:
> +- Direct delivery of external interrupts to running vCPUs without
> VMM
> +intervention
> +- Decrease the interrupt migration complexity. On vCPU migration,
> software
> +can atomically co-migrate all interrupts targeting the migrating
> vCPU. For
> +virtual machines with assigned devices, migrating a vCPU across
> pCPUs
> +either incurs the overhead of forwarding interrupts in software
> (e.g. via VMM
> +generated IPIs), or complexity to independently migrate each
> interrupt targeting
> +the vCPU to the new pCPU. However, after enabling VT-d PI, the
> destination vCPU
> +of an external interrupt from assigned devices is stored in the IRTE
> (i.e.
> +Posted-interrupt Descriptor Address), when vCPU is migrated to
> another pCPU,
> +we will set this new pCPU in the 'NDST' filed of Posted-interrupt

   ^field

> +More information about APICv and CPU-side Posted-interrupt, please
> refer
> +to Chapter 29, and Section 29.6 in the Intel SDM:
> +http://www.intel.com/content/dam/www/public/us/en/documents/manuals/
> 64-ia-32-architectures-software-developer-manual-325462.pdf
> +
ISTR Andrew said something about chapter names and numbers, and about
what to link in these cases, during v8 review.

> +- Update posted-interrupt descriptor during vCPU scheduling
> +
> +The basic idea here is:
> +1. When vCPU is running
> +- Set 'NV' to 'posted_intr_vector'.
> +- Clear 'SN' to accept posted-interrupts.
> +- Set 'NDST' to the pCPU on which the vCPU will be running.
> +2. When vCPU is blocked
> +- Set 'NV' to ' pi_wakeup_vector ', so we can wake up the
> +  related vCPU when posted-interrupt happens for it.
> +  Please refer to the above section about the new global
> vector.
> +- Clear 'SN' to accept posted-interrupts
> +3. When vCPU is preempted or sleeping
> +- Set 'SN' to suppress non-urgent interrupts
> +  (Currently, we only support non-urgent interrupts)
> + When vCPU is preempted or sleep, it doesn't need to accept
> + posted-interrupt notification event since we don't change
> the behavior
> + of scheduler when the interrupt occurs, we still need wait
> for the next
> + scheduling of the vCPU. When external interrupts from
> assigned devices occur,
> + the interrupts are recorded in PIR, and will be synced to
> IRR before VM-Entry.
> +- Set 'NV' to 'posted_intr_vector'.
> +
> +- How to wakeup blocked vCPU when an interrupt is posted for it
> (wakeup notification handler).
> +
>
> [...]
>
> +Here are what we do for the blocked vCPU:
> +1. Define a per-cpu list 'pi_blocked_vcpu', which stored the blocked
> +vCPU on the pCPU.
> +2. When the vCPU is going to block, insert the vCPU
> +to the per-cpu list belonging to the pCPU it was running.
> +3. When the vCPU is unblocked, remove the vCPU from the related pCPU
> list.
> +
> +In the handler of 'pi_wakeup_vector', we do:
> +1. Get the physical CPU.
> +2. Iterate the list 'pi_blocked_vcpu' of the current pCPU, if 'ON'
> is set,
> +we unblock the associated vCPU.
> +
This is indeed more general, and the fact that it does no longer
mentions RUNSTATEs, makes it more adherent to the actual implementation
(and hence more useful), so thanks for doing this.

Personally, I'd add a quick comment about how this, despite being
related to blocking and unblocking, happens actually inside VMX
handlers, i.e., (quickly) what is the relationship within these sets of
events (blocking/unblocking VMENTER/EXIT) and why it is ok to do things
like they are done.

I'm not too fussed about this, though. So, if others don't think
something like this is necessary, I'm fine with what you have here.

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)



signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel