Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
2015-01-20 23:04+0200, Nadav Amit: > Radim Kr?má? wrote: > > 2015-01-14 01:27+, Wu, Feng: > >>> the new > hardware even doesn't consider the TPR for lowest priority interrupts > >>> delivery. > >>> > >>> A bold move ... what hardware was the first to do so? > >> > >> I think it was starting with Nehalem. > > > > Thanks, (Could be that QPI can't inform about TPR changes anymore ...) > > > > I played with Linux's TPR on Haswell and found that is has no effect. > > Sorry for jumping into the discussion, but doesn’t it depend on > IA32_MISC_ENABLE[23]? This bit disables xTPR messages. On my machine it is > set (probably by the BIOS), but since there is no IA32_MISC_ENABLE is not > locked for changes, the OS can control it. Thanks, I didn't know about it. On Ivy Bridge EP (the only modern machine at hand), the bit was set by default. After clearing it, TPR still had no effect. The most relevant mention of xTPR I found is related to FSB [1]. [2] isn't enlightening, so there might be more from QPI-era ... --- 1: Intel® E7320 Memory Controller Hub (MCH) Datasheet http://www.intel.com/content/dam/doc/datasheet/e7320-memory-controller-hub-datasheet.pdf 5.2.2 System Bus Interrupts 2: Intel® Xeon® Processor E5 v2 Family: Datasheet, Vol. 2 http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v2-datasheet-vol-2.pdf 6.1.2 IntControl -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
Radim Kr?má? wrote: > 2015-01-14 01:27+, Wu, Feng: >>> the new hardware even doesn't consider the TPR for lowest priority interrupts >>> delivery. >>> >>> A bold move ... what hardware was the first to do so? >> >> I think it was starting with Nehalem. > > Thanks, (Could be that QPI can't inform about TPR changes anymore ...) > > I played with Linux's TPR on Haswell and found that is has no effect. Sorry for jumping into the discussion, but doesn’t it depend on IA32_MISC_ENABLE[23]? This bit disables xTPR messages. On my machine it is set (probably by the BIOS), but since there is no IA32_MISC_ENABLE is not locked for changes, the OS can control it. Nadav-- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
2015-01-14 01:27+, Wu, Feng: > > the new > > > hardware even doesn't consider the TPR for lowest priority interrupts > > delivery. > > > > A bold move ... what hardware was the first to do so? > > I think it was starting with Nehalem. Thanks, (Could be that QPI can't inform about TPR changes anymore ...) I played with Linux's TPR on Haswell and found that is has no effect. > > > As discussed with Paolo before, I will submit a patch to support lowest > > priority for PI > > > after this series is merged. > > > > Sure, I see only two good solutions though > > 1) don't optimize lowest priority with PI > > 2) don't balance lowest priority > > As discussed with Paolo before, as the first stage, we only support single-CPU > lowest priority for PI, since this is a new hardware feature enabling, Paolo > trends > to do simple things in the beginning. I agree, that is the best we can do without changing lowest priority. I wanted to avoid a future solution that would introduce two behaviors for lowest priority (round robin and something). Round robin (anything dynamic) can't be done with PI, hence the question if we can remove it. > Then we will support full lowest > priority for > it, such as, using vector hashing (this is one method of what hardware do for > lowest priority today), I need to get some detailed information about this > from > hardware guys before enabling it. I wasn't able to confirm hashing, is it a recent addition? I'm not sure we want it then: OS still has to take care of proper distribution, being predictable is better than uncertain gains, and we'll save code. The same should apply to hardware though ... do you know the reasons behind vector hashing? Thank you. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
On 14/01/2015 02:27, Wu, Feng wrote: > As discussed with Paolo before, as the first stage, we only support single-CPU > lowest priority for PI, since this is a new hardware feature enabling, Paolo > trends > to do simple things in the beginning. :) Nice way to sum it up! Paolo > Then we will support full lowest priority for > it, such as, using vector hashing (this is one method of what hardware do for > lowest priority today), I need to get some detailed information about this > from > hardware guys before enabling it. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
> -Original Message-
> From: Radim Kr?má? [mailto:rkrc...@redhat.com]
> Sent: Wednesday, January 14, 2015 12:17 AM
> To: Wu, Feng
> Cc: Paolo Bonzini; t...@linutronix.de; mi...@redhat.com; h...@zytor.com;
> x...@kernel.org; g...@kernel.org; dw...@infradead.org; j...@8bytes.org;
> alex.william...@redhat.com; jiang@linux.intel.com; eric.au...@linaro.org;
> linux-ker...@vger.kernel.org; io...@lists.linux-foundation.org;
> kvm@vger.kernel.org
> Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for
> VT-d PI
>
> 2015-01-13 00:27+, Wu, Feng:
> > > On 09/01/2015 15:54, Radim Krčmář wrote:
> > > > There are two points relevant to this patch in new KVM's implementation,
> > > > ("KVM: x86: amend APIC lowest priority arbitration",
> > > > https://lkml.org/lkml/2015/1/9/362)
> > > >
> > > > 1) lowest priority depends on TPR
> > > > 2) there is no need for balancing
> > > >
> > > > (1) has to be considered with PI as well.
> > >
> > > The chipset doesn't support it. :(
> > >
> > > > I kept (2) to avoid whining from people building on that behaviour, but
> > > > lowest priority backed by PI could be transparent without it.
> > > >
> > > > Patch below removes the balancing, but I am not sure this is a price we
> > > > allowed ourselves to pay ... what are your opinions?
> > >
> > > I wouldn't mind, but it requires a lot of benchmarking.
> >
> > In fact, the real hardware may do lowest priority in round robin way,
>
> Yes, but we won't emulate round robin with PI and I think it is wrong to
> have backends with significantly different guest-visible behaviors.
>
> >
> the new
> > hardware even doesn't consider the TPR for lowest priority interrupts
> delivery.
>
> A bold move ... what hardware was the first to do so?
I think it was starting with Nehalem.
>
> > As discussed with Paolo before, I will submit a patch to support lowest
> priority for PI
> > after this series is merged.
>
> Sure, I see only two good solutions though
> 1) don't optimize lowest priority with PI
> 2) don't balance lowest priority
As discussed with Paolo before, as the first stage, we only support single-CPU
lowest priority for PI, since this is a new hardware feature enabling, Paolo
trends
to do simple things in the beginning. Then we will support full lowest priority
for
it, such as, using vector hashing (this is one method of what hardware do for
lowest priority today), I need to get some detailed information about this from
hardware guys before enabling it.
Thanks,
Feng
N�r��yb�X��ǧv�^�){.n�+h����ܨ}���Ơz�&j:+v���zZ+��+zf���h���~i���z� �w���?�&�)ߢf
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
2015-01-13 00:27+, Wu, Feng:
> > On 09/01/2015 15:54, Radim Krčmář wrote:
> > > There are two points relevant to this patch in new KVM's implementation,
> > > ("KVM: x86: amend APIC lowest priority arbitration",
> > > https://lkml.org/lkml/2015/1/9/362)
> > >
> > > 1) lowest priority depends on TPR
> > > 2) there is no need for balancing
> > >
> > > (1) has to be considered with PI as well.
> >
> > The chipset doesn't support it. :(
> >
> > > I kept (2) to avoid whining from people building on that behaviour, but
> > > lowest priority backed by PI could be transparent without it.
> > >
> > > Patch below removes the balancing, but I am not sure this is a price we
> > > allowed ourselves to pay ... what are your opinions?
> >
> > I wouldn't mind, but it requires a lot of benchmarking.
>
> In fact, the real hardware may do lowest priority in round robin way,
Yes, but we won't emulate round robin with PI and I think it is wrong to
have backends with significantly different guest-visible behaviors.
> the new
> hardware even doesn't consider the TPR for lowest priority interrupts
> delivery.
A bold move ... what hardware was the first to do so?
> As discussed with Paolo before, I will submit a patch to support lowest
> priority for PI
> after this series is merged.
Sure, I see only two good solutions though
1) don't optimize lowest priority with PI
2) don't balance lowest priority
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
> -Original Message-
> From: Paolo Bonzini [mailto:pbonz...@redhat.com]
> Sent: Friday, January 09, 2015 10:56 PM
> To: Radim Krčmář; Wu, Feng
> Cc: t...@linutronix.de; mi...@redhat.com; h...@zytor.com; x...@kernel.org;
> g...@kernel.org; dw...@infradead.org; j...@8bytes.org;
> alex.william...@redhat.com; jiang@linux.intel.com; eric.au...@linaro.org;
> linux-ker...@vger.kernel.org; io...@lists.linux-foundation.org;
> kvm@vger.kernel.org
> Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for
> VT-d PI
>
>
>
> On 09/01/2015 15:54, Radim Krčmář wrote:
> > There are two points relevant to this patch in new KVM's implementation,
> > ("KVM: x86: amend APIC lowest priority arbitration",
> > https://lkml.org/lkml/2015/1/9/362)
> >
> > 1) lowest priority depends on TPR
> > 2) there is no need for balancing
> >
> > (1) has to be considered with PI as well.
>
> The chipset doesn't support it. :(
>
> > I kept (2) to avoid whining from people building on that behaviour, but
> > lowest priority backed by PI could be transparent without it.
> >
> > Patch below removes the balancing, but I am not sure this is a price we
> > allowed ourselves to pay ... what are your opinions?
>
> I wouldn't mind, but it requires a lot of benchmarking.
In fact, the real hardware may do lowest priority in round robin way, the new
hardware even doesn't consider the TPR for lowest priority interrupts delivery.
As discussed with Paolo before, I will submit a patch to support lowest
priority for PI
after this series is merged.
Thanks,
Feng
>
> Paolo
N�r��yb�X��ǧv�^�){.n�+h����ܨ}���Ơz�&j:+v���zZ+��+zf���h���~i���z� �w���?�&�)ߢf
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
2015-01-09 16:18+0100, Paolo Bonzini: > On 09/01/2015 16:12, Radim Krčmář wrote: > > > The chipset doesn't support it. :( > > > > I meant that we need to recompute PI entries for lowest priority > > interrupts every time guest's TPR changes. > > > > Luckily, Linux doesn't use TPR, but other OS might be a reason to drop > > lowest priority from PI optimizations. (Or make it more complicated.) > > Doing vector hashing is a possibility as well. I would like to know > what existing chipsets do in practice, then we can mimic it. When looking at /proc/interrupts from time to time, I have only seen interrupts landing on the first CPU of the set. We could also distinguish between AMD and Intel ... AMD should deliver to the highest APIC ID. (If we still need to decide after focus processor and APR checks.) I'll try to check using a more trustworthy approach. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
On 09/01/2015 16:12, Radim Krčmář wrote: > > The chipset doesn't support it. :( > > I meant that we need to recompute PI entries for lowest priority > interrupts every time guest's TPR changes. > > Luckily, Linux doesn't use TPR, but other OS might be a reason to drop > lowest priority from PI optimizations. (Or make it more complicated.) Doing vector hashing is a possibility as well. I would like to know what existing chipsets do in practice, then we can mimic it. Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
2015-01-09 15:56+0100, Paolo Bonzini:
>
>
> On 09/01/2015 15:54, Radim Krčmář wrote:
> > There are two points relevant to this patch in new KVM's implementation,
> > ("KVM: x86: amend APIC lowest priority arbitration",
> > https://lkml.org/lkml/2015/1/9/362)
> >
> > 1) lowest priority depends on TPR
> > 2) there is no need for balancing
> >
> > (1) has to be considered with PI as well.
>
> The chipset doesn't support it. :(
I meant that we need to recompute PI entries for lowest priority
interrupts every time guest's TPR changes.
Luckily, Linux doesn't use TPR, but other OS might be a reason to drop
lowest priority from PI optimizations. (Or make it more complicated.)
> > I kept (2) to avoid whining from people building on that behaviour, but
> > lowest priority backed by PI could be transparent without it.
> >
> > Patch below removes the balancing, but I am not sure this is a price we
> > allowed ourselves to pay ... what are your opinions?
>
> I wouldn't mind, but it requires a lot of benchmarking.
(I was afraid it would come to that.)
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
On 09/01/2015 15:54, Radim Krčmář wrote:
> There are two points relevant to this patch in new KVM's implementation,
> ("KVM: x86: amend APIC lowest priority arbitration",
> https://lkml.org/lkml/2015/1/9/362)
>
> 1) lowest priority depends on TPR
> 2) there is no need for balancing
>
> (1) has to be considered with PI as well.
The chipset doesn't support it. :(
> I kept (2) to avoid whining from people building on that behaviour, but
> lowest priority backed by PI could be transparent without it.
>
> Patch below removes the balancing, but I am not sure this is a price we
> allowed ourselves to pay ... what are your opinions?
I wouldn't mind, but it requires a lot of benchmarking.
Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
2014-12-12 23:14+0800, Feng Wu:
> This patch defines a new interface kvm_find_dest_vcpu for
> VT-d PI, which can returns the destination vCPU of the
> interrupt for guests.
>
> Since VT-d PI cannot handle broadcast/multicast interrupt,
> Here we only handle Fixed and Lowest priority interrupts.
>
> The current method of handling guest lowest priority interrtups
> is to use a counter 'apic_arb_prio' for each vCPU, we choose the
> vCPU with smallest 'apic_arb_prio' and then increase it by 1.
> However, for VT-d PI, we cannot re-use this, since we no longer
> have control to 'apic_arb_prio' with posted interrupt direct
> delivery by Hardware.
>
> Here, we introduce a similar way with 'apic_arb_prio' to handle
> guest lowest priority interrtups when VT-d PI is used. Here is the
> ideas:
> - Each vCPU has a counter 'round_robin_counter'.
> - When guests sets an interrupts to lowest priority, we choose
> the vCPU with smallest 'round_robin_counter' as the destination,
> then increase it.
There are two points relevant to this patch in new KVM's implementation,
("KVM: x86: amend APIC lowest priority arbitration",
https://lkml.org/lkml/2015/1/9/362)
1) lowest priority depends on TPR
2) there is no need for balancing
(1) has to be considered with PI as well.
I kept (2) to avoid whining from people building on that behaviour, but
lowest priority backed by PI could be transparent without it.
Patch below removes the balancing, but I am not sure this is a price we
allowed ourselves to pay ... what are your opinions?
---8<---
KVM: x86: don't balance lowest priority interrupts
Balancing is not mandated by specification and real hardware most likely
doesn't do it. We break backward compatibility to allow optimizations.
(Posted interrupts can deliver to only one fixed destination.)
Signed-off-by: Radim Krčmář
---
arch/x86/include/asm/kvm_host.h | 1 -
arch/x86/kvm/lapic.c| 8 ++--
2 files changed, 2 insertions(+), 7 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 97a5dd0222c8..aa4bd8286232 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -370,7 +370,6 @@ struct kvm_vcpu_arch {
u64 apic_base;
struct kvm_lapic *apic;/* kernel irqchip context */
unsigned long apic_attention;
- int32_t apic_arb_prio;
int mp_state;
u64 ia32_misc_enable_msr;
bool tpr_access_reporting;
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 5b9d8c589bba..eb85af8e8fc0 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -749,7 +749,6 @@ static int __apic_accept_irq(struct kvm_lapic *apic, int
delivery_mode,
trig_mode, vector);
switch (delivery_mode) {
case APIC_DM_LOWEST:
- vcpu->arch.apic_arb_prio++;
case APIC_DM_FIXED:
/* FIXME add logic for vcpu on reset */
if (unlikely(!apic_enabled(apic)))
@@ -837,11 +836,9 @@ int kvm_apic_compare_prio(struct kvm_vcpu *vcpu1, struct
kvm_vcpu *vcpu2)
* - uses the APR register (which also considers ISR and IRR),
* - chooses the highest APIC ID when APRs are identical,
* - and allows a focus processor.
-* XXX: pseudo-balancing with apic_arb_prio is a KVM-specific feature
*/
- int tpr = kvm_apic_get_reg(vcpu1->arch.apic, APIC_TASKPRI) -
- kvm_apic_get_reg(vcpu2->arch.apic, APIC_TASKPRI);
- return tpr ? : vcpu1->arch.apic_arb_prio - vcpu2->arch.apic_arb_prio;
+ return kvm_apic_get_reg(vcpu1->arch.apic, APIC_TASKPRI) -
+ kvm_apic_get_reg(vcpu2->arch.apic, APIC_TASKPRI);
}
static void kvm_ioapic_send_eoi(struct kvm_lapic *apic, int vector)
@@ -1595,7 +1592,6 @@ void kvm_lapic_reset(struct kvm_vcpu *vcpu)
vcpu->arch.pv_eoi.msr_val = 0;
apic_update_ppr(apic);
- vcpu->arch.apic_arb_prio = 0;
vcpu->arch.apic_attention = 0;
apic_debug("%s: vcpu=%p, id=%d, base_msr="
--
2.2.0
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
Paolo Bonzini wrote on 2014-12-23:
>> The problem is we still need to support PI with lowest priority
>> delivery mode
> even if guest does not configure irq affinity via /proc/irq/. Don't we?
>
> Yes, but we can get the basic support working first.
>
> I and Feng talked on irc and agreed to start with a simple
> implementation. Then he can add support for vector hashing for all
> interrupts, both vt-d pi and normal LAPIC interrupts.
Agree. Obviously, current PI has some limitations to achieve highest
performance. We can start with a simple implementation but must ensure we don't
change hardware's behavior(From guest's point). Feng's current implementation
or use the way I suggested are both ok since both of them cannot solve the
problem well.
>
> Paolo
>
Best regards,
Yang
N�r��yb�X��ǧv�^�){.n�+h����ܨ}���Ơz�&j:+v���zZ+��+zf���h���~i���z� �w���?�&�)ߢf
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
On Mon, Dec 22, 2014 at 12:23:36PM +0100, Paolo Bonzini wrote: > > > On 22/12/2014 12:17, Wu, Feng wrote: > >> Yes, and those need not be accelerated. But what if you set > >> affinity to a single CPU? > > > > How do I set affinity to a single CPU if guest configure a lowest > > priority interrupt? Thanks a lot! > > I mean if the guest (via irqbalance and /proc/irq/) configures affinity > to a single vCPU. In that case, you can use PI. > The problem is we still need to support PI with lowest priority delivery mode even if guest does not configure irq affinity via /proc/irq/. Don't we? Thanks -Yong -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
On 22/12/2014 12:17, Wu, Feng wrote: >> Yes, and those need not be accelerated. But what if you set >> affinity to a single CPU? > > How do I set affinity to a single CPU if guest configure a lowest > priority interrupt? Thanks a lot! I mean if the guest (via irqbalance and /proc/irq/) configures affinity to a single vCPU. In that case, you can use PI. Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
> -Original Message- > From: Paolo Bonzini [mailto:pbonz...@redhat.com] > Sent: Monday, December 22, 2014 7:07 PM > To: Wu, Feng; Zhang, Yang Z; Thomas Gleixner; Ingo Molnar; H. Peter Anvin; > x...@kernel.org; Gleb Natapov; dw...@infradead.org; j...@8bytes.org; > Alex Williamson; Jiang Liu > Cc: io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list; > Eric Auger > Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for > VT-d PI > > > > On 22/12/2014 12:04, Wu, Feng wrote: > > > Can you support them only if the destination is a single CPU? > > > > Sorry, I am not quite understand this. I still don't understand the "single > > CPU" > here. > > Lowest priority interrupts always have a cpumask which contains multiple > CPU. > > Yes, and those need not be accelerated. But what if you set affinity to > a single CPU? How do I set affinity to a single CPU if guest configure a lowest priority interrupt? Thanks a lot! Thanks, Feng > > Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
On 22/12/2014 12:04, Wu, Feng wrote: > > Can you support them only if the destination is a single CPU? > > Sorry, I am not quite understand this. I still don't understand the "single > CPU" here. > Lowest priority interrupts always have a cpumask which contains multiple CPU. Yes, and those need not be accelerated. But what if you set affinity to a single CPU? Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
> -Original Message- > From: Paolo Bonzini [mailto:pbonz...@redhat.com] > Sent: Monday, December 22, 2014 5:28 PM > To: Wu, Feng; Zhang, Yang Z; Thomas Gleixner; Ingo Molnar; H. Peter Anvin; > x...@kernel.org; Gleb Natapov; dw...@infradead.org; j...@8bytes.org; > Alex Williamson; Jiang Liu > Cc: io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list; > Eric Auger > Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for > VT-d PI > > > > On 22/12/2014 05:48, Wu, Feng wrote: > > Do you mean we don't support Lowest priority interrupts? As I mentioned > before, > > Lowest priority interrupts is widely used in Linux, so I think supporting > > lowest > priority > > interrupts is very important for Linux guest OS. Do you have any > ideas/suggestions about > > how to support Lowest priority interrupts for PI? Thanks a lot! > > Can you support them only if the destination is a single CPU? Sorry, I am not quite understand this. I still don't understand the "single CPU" here. Lowest priority interrupts always have a cpumask which contains multiple CPU. Thanks, Feng > > Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
On 22/12/2014 05:48, Wu, Feng wrote: > Do you mean we don't support Lowest priority interrupts? As I mentioned > before, > Lowest priority interrupts is widely used in Linux, so I think supporting > lowest priority > interrupts is very important for Linux guest OS. Do you have any > ideas/suggestions about > how to support Lowest priority interrupts for PI? Thanks a lot! Can you support them only if the destination is a single CPU? Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
> -Original Message- > From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo > Bonzini > Sent: Saturday, December 20, 2014 9:17 PM > To: Wu, Feng; Paolo Bonzini; Zhang, Yang Z; Thomas Gleixner; Ingo Molnar; H. > Peter Anvin; x...@kernel.org; Gleb Natapov; dw...@infradead.org; > j...@8bytes.org; Alex Williamson; Jiang Liu > Cc: io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list; > Eric Auger > Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for > VT-d PI > > > > On 20/12/2014 00:48, Wu, Feng wrote: > > In my understanding, lowest priority interrupts are always delivered to a > > Single CPU, we need to find the right destination CPU from the cpumask. > > Yes, but which CPU however differs every time the interrupt is > delivered. So the emulation here is a bit poor. For now, please limit > PI to fixed interrupts. Do you mean we don't support Lowest priority interrupts? As I mentioned before, Lowest priority interrupts is widely used in Linux, so I think supporting lowest priority interrupts is very important for Linux guest OS. Do you have any ideas/suggestions about how to support Lowest priority interrupts for PI? Thanks a lot! Thanks, Feng > > > Actually, we don't support posting broadcast/multicast interrupts, because > > the interrupt is associated with one Posted-interrupts descriptor, then one > > vCPU. > > Understood. > > Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
On 20/12/2014 00:48, Wu, Feng wrote: > In my understanding, lowest priority interrupts are always delivered to a > Single CPU, we need to find the right destination CPU from the cpumask. Yes, but which CPU however differs every time the interrupt is delivered. So the emulation here is a bit poor. For now, please limit PI to fixed interrupts. > Actually, we don't support posting broadcast/multicast interrupts, because > the interrupt is associated with one Posted-interrupts descriptor, then one > vCPU. Understood. Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
> -Original Message- > From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo > Bonzini > Sent: Friday, December 19, 2014 7:59 PM > To: Wu, Feng; Paolo Bonzini; Zhang, Yang Z; Thomas Gleixner; Ingo Molnar; H. > Peter Anvin; x...@kernel.org; Gleb Natapov; dw...@infradead.org; > j...@8bytes.org; Alex Williamson; Jiang Liu > Cc: io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list; > Eric Auger > Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for > VT-d PI > > > > On 19/12/2014 02:30, Wu, Feng wrote: > >>> How this can work well? All subsequent interrupts are delivered to > >>> one vCPU? It shouldn't be the best solution, need more consideration. > >> > >> Well, it's a hardware limitation. The alternative (which is easy to > >> implement) is to only do PI for single-CPU interrupts. This should work > >> well for multiqueue NICs (and of course for UP guests :)), so perhaps > >> it's a good idea to only support that as a first attempt. > >> > >> Paolo > > > > Paolo, what do you mean by "single-CPU interrupts"? Do you mean we don't > > support lowest priority interrupts for PI? But Linux OS uses lowest priority > > for most of the case? If so, we can hardly get benefit from this feature for > > Linux guest OS. > > You can post lowest priority interrupts if they are delivered to a > single CPU, in which case they are effectively fixed priority. > > If they are broadcast to multiple CPUs, do not post them. > > Paolo In my understanding, lowest priority interrupts are always delivered to a Single CPU, we need to find the right destination CPU from the cpumask. This is what I do in this patch. Did I misunderstanding something in your Comments? Thanks a lot! Actually, we don't support posting broadcast/multicast interrupts, because the interrupt is associated with one Posted-interrupts descriptor, then one vCPU. Thanks, Feng -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
On 19/12/2014 02:30, Wu, Feng wrote: >>> How this can work well? All subsequent interrupts are delivered to >>> one vCPU? It shouldn't be the best solution, need more consideration. >> >> Well, it's a hardware limitation. The alternative (which is easy to >> implement) is to only do PI for single-CPU interrupts. This should work >> well for multiqueue NICs (and of course for UP guests :)), so perhaps >> it's a good idea to only support that as a first attempt. >> >> Paolo > > Paolo, what do you mean by "single-CPU interrupts"? Do you mean we don't > support lowest priority interrupts for PI? But Linux OS uses lowest priority > for most of the case? If so, we can hardly get benefit from this feature for > Linux guest OS. You can post lowest priority interrupts if they are delivered to a single CPU, in which case they are effectively fixed priority. If they are broadcast to multiple CPUs, do not post them. Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
Wu, Feng wrote on 2014-12-19: > > > Paolo Bonzini wrote on 2014-12-19: >> jiang@linux.intel.com >> Cc: eric.au...@linaro.org; linux-ker...@vger.kernel.org; >> io...@lists.linux-foundation.org; kvm@vger.kernel.org >> Subject: Re: [v3 13/26] KVM: Define a new interface >> kvm_find_dest_vcpu() for VT-d PI >> >> >> >> On 18/12/2014 15:49, Zhang, Yang Z wrote: >>>>> Here, we introduce a similar way with 'apic_arb_prio' to handle >>>>> guest lowest priority interrtups when VT-d PI is used. Here is >>>>> the >>>>> ideas: - Each vCPU has a counter 'round_robin_counter'. - When >>>>> guests sets an interrupts to lowest priority, we choose the vCPU >>>>> with smallest 'round_robin_counter' as the destination, then >>>>> increase it. >>> >>> How this can work well? All subsequent interrupts are delivered to >>> one vCPU? It shouldn't be the best solution, need more consideration. >> >> Well, it's a hardware limitation. The alternative (which is easy to >> implement) is to only do PI for single-CPU interrupts. This should >> work well for multiqueue NICs (and of course for UP guests :)), so >> perhaps it's a good idea to only support that as a first attempt. >> >> Paolo > > Paolo, what do you mean by "single-CPU interrupts"? Do you mean we It should be same idea as I mentioned on another thread: deliver the interrupt to a single CPU(maybe the first matched VCPU?) > don't support lowest priority interrupts for PI? But Linux OS uses > lowest priority for most of the case? If so, we can hardly get benefit > from this feature for Linux guest OS. > > Thanks, > Feng > >> >>> Also, I think you should take the apic_arb_prio into consider >>> since the priority is for the whole vCPU not for one interrupt. Best regards, Yang -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
> -Original Message- > From: Zhang, Yang Z > Sent: Friday, December 19, 2014 9:14 AM > To: Paolo Bonzini; Wu, Feng; t...@linutronix.de; mi...@redhat.com; > h...@zytor.com; x...@kernel.org; g...@kernel.org; dw...@infradead.org; > j...@8bytes.org; alex.william...@redhat.com; jiang@linux.intel.com > Cc: eric.au...@linaro.org; linux-ker...@vger.kernel.org; > io...@lists.linux-foundation.org; kvm@vger.kernel.org > Subject: RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for > VT-d PI > > Paolo Bonzini wrote on 2014-12-19: > > > > > > On 18/12/2014 15:49, Zhang, Yang Z wrote: > >>>> Here, we introduce a similar way with 'apic_arb_prio' to handle > >>>> guest lowest priority interrtups when VT-d PI is used. Here is the > >>>> ideas: - Each vCPU has a counter 'round_robin_counter'. - When > >>>> guests sets an interrupts to lowest priority, we choose the vCPU > >>>> with smallest 'round_robin_counter' as the destination, then > >>>> increase it. > >> > >> How this can work well? All subsequent interrupts are delivered to > >> one vCPU? It shouldn't be the best solution, need more consideration. > > > > Well, it's a hardware limitation. The alternative (which is easy to > > Agree, it is limited by hardware. But lowest priority distributes the > interrupt > more efficient than fixed mode. And current implementation more likes to > switch the lowest priority mode to fixed mode. In case of interrupt intensive > environment, this may be a bottleneck and VM may not benefit greatly from > VT-d PI. But agree again, it is really a hardware limitation. > > > implement) is to only do PI for single-CPU interrupts. This should > > work well for multiqueue NICs (and of course for UP guests :)), so > > perhaps it's a good idea to only support that as a first attempt. > > The more easy way is to deliver the interrupt to the first matched VCPU we > find. > The round_robin_counter really helps nothing here since the interrupt is > delivered by hardware directly. > > > > > Paolo > > > >> Also, I think you should take the apic_arb_prio into consider since > >> the priority is for the whole vCPU not for one interrupt. > > > Best regards, > Yang In fact, the current solution was discussed with Rajesh in the cc List, here is Rajesh's original words: "When you see a guest requesting a lowest priority interrupts (by programming the virtual IOAPIC, or by programming the virtual MSI/MSI-X registers), have KVM associate it to a vCPU. Or, put another way, use the 'apic_arb_prio' method you describe below, but instead of using it at time of interrupt (which you no longer have control with posted interrupt direct delivery), do it at time of initializing the interrupt resource. This way, if the guest asks for 4 lowest priority interrupts, and say you a guest with two vCPUs, the first interrupt request will be serviced by KVM by assigning it through posting to vCPU0, the next one goes to vCPU1, the next one would go back to vCPU0, and so forth.. You could also choose to do this based on vector hashing instead of round-robin." Thanks, Feng > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
> -Original Message- > From: Paolo Bonzini [mailto:pbonz...@redhat.com] > Sent: Friday, December 19, 2014 12:58 AM > To: Zhang, Yang Z; Wu, Feng; t...@linutronix.de; mi...@redhat.com; > h...@zytor.com; x...@kernel.org; g...@kernel.org; dw...@infradead.org; > j...@8bytes.org; alex.william...@redhat.com; jiang@linux.intel.com > Cc: eric.au...@linaro.org; linux-ker...@vger.kernel.org; > io...@lists.linux-foundation.org; kvm@vger.kernel.org > Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for > VT-d PI > > > > On 18/12/2014 15:49, Zhang, Yang Z wrote: > >>> Here, we introduce a similar way with 'apic_arb_prio' to handle > >>> guest lowest priority interrtups when VT-d PI is used. Here is > >>> the ideas: - Each vCPU has a counter 'round_robin_counter'. - > >>> When guests sets an interrupts to lowest priority, we choose the > >>> vCPU with smallest 'round_robin_counter' as the destination, then > >>> increase it. > > > > How this can work well? All subsequent interrupts are delivered to > > one vCPU? It shouldn't be the best solution, need more consideration. > > Well, it's a hardware limitation. The alternative (which is easy to > implement) is to only do PI for single-CPU interrupts. This should work > well for multiqueue NICs (and of course for UP guests :)), so perhaps > it's a good idea to only support that as a first attempt. > > Paolo Paolo, what do you mean by "single-CPU interrupts"? Do you mean we don't support lowest priority interrupts for PI? But Linux OS uses lowest priority for most of the case? If so, we can hardly get benefit from this feature for Linux guest OS. Thanks, Feng > > > Also, I think you should take the apic_arb_prio into consider since > > the priority is for the whole vCPU not for one interrupt. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
Paolo Bonzini wrote on 2014-12-19: > > > On 18/12/2014 15:49, Zhang, Yang Z wrote: Here, we introduce a similar way with 'apic_arb_prio' to handle guest lowest priority interrtups when VT-d PI is used. Here is the ideas: - Each vCPU has a counter 'round_robin_counter'. - When guests sets an interrupts to lowest priority, we choose the vCPU with smallest 'round_robin_counter' as the destination, then increase it. >> >> How this can work well? All subsequent interrupts are delivered to >> one vCPU? It shouldn't be the best solution, need more consideration. > > Well, it's a hardware limitation. The alternative (which is easy to Agree, it is limited by hardware. But lowest priority distributes the interrupt more efficient than fixed mode. And current implementation more likes to switch the lowest priority mode to fixed mode. In case of interrupt intensive environment, this may be a bottleneck and VM may not benefit greatly from VT-d PI. But agree again, it is really a hardware limitation. > implement) is to only do PI for single-CPU interrupts. This should > work well for multiqueue NICs (and of course for UP guests :)), so > perhaps it's a good idea to only support that as a first attempt. The more easy way is to deliver the interrupt to the first matched VCPU we find. The round_robin_counter really helps nothing here since the interrupt is delivered by hardware directly. > > Paolo > >> Also, I think you should take the apic_arb_prio into consider since >> the priority is for the whole vCPU not for one interrupt. Best regards, Yang -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
On 18/12/2014 15:49, Zhang, Yang Z wrote: >>> Here, we introduce a similar way with 'apic_arb_prio' to handle >>> guest lowest priority interrtups when VT-d PI is used. Here is >>> the ideas: - Each vCPU has a counter 'round_robin_counter'. - >>> When guests sets an interrupts to lowest priority, we choose the >>> vCPU with smallest 'round_robin_counter' as the destination, then >>> increase it. > > How this can work well? All subsequent interrupts are delivered to > one vCPU? It shouldn't be the best solution, need more consideration. Well, it's a hardware limitation. The alternative (which is easy to implement) is to only do PI for single-CPU interrupts. This should work well for multiqueue NICs (and of course for UP guests :)), so perhaps it's a good idea to only support that as a first attempt. Paolo > Also, I think you should take the apic_arb_prio into consider since > the priority is for the whole vCPU not for one interrupt. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
Feng Wu wrote on 2014-12-12: > This patch defines a new interface kvm_find_dest_vcpu for > VT-d PI, which can returns the destination vCPU of the > interrupt for guests. > > Since VT-d PI cannot handle broadcast/multicast interrupt, > Here we only handle Fixed and Lowest priority interrupts. > > The current method of handling guest lowest priority interrtups > is to use a counter 'apic_arb_prio' for each vCPU, we choose the > vCPU with smallest 'apic_arb_prio' and then increase it by 1. > However, for VT-d PI, we cannot re-use this, since we no longer > have control to 'apic_arb_prio' with posted interrupt direct > delivery by Hardware. > > Here, we introduce a similar way with 'apic_arb_prio' to handle guest > lowest priority interrtups when VT-d PI is used. Here is the ideas: - > Each vCPU has a counter 'round_robin_counter'. - When guests sets an > interrupts to lowest priority, we choose the vCPU with smallest > 'round_robin_counter' as the destination, then increase it. How this can work well? All subsequent interrupts are delivered to one vCPU? It shouldn't be the best solution, need more consideration. Also, I think you should take the apic_arb_prio into consider since the priority is for the whole vCPU not for one interrupt. Best regards, Yang -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI
This patch defines a new interface kvm_find_dest_vcpu for
VT-d PI, which can returns the destination vCPU of the
interrupt for guests.
Since VT-d PI cannot handle broadcast/multicast interrupt,
Here we only handle Fixed and Lowest priority interrupts.
The current method of handling guest lowest priority interrtups
is to use a counter 'apic_arb_prio' for each vCPU, we choose the
vCPU with smallest 'apic_arb_prio' and then increase it by 1.
However, for VT-d PI, we cannot re-use this, since we no longer
have control to 'apic_arb_prio' with posted interrupt direct
delivery by Hardware.
Here, we introduce a similar way with 'apic_arb_prio' to handle
guest lowest priority interrtups when VT-d PI is used. Here is the
ideas:
- Each vCPU has a counter 'round_robin_counter'.
- When guests sets an interrupts to lowest priority, we choose
the vCPU with smallest 'round_robin_counter' as the destination,
then increase it.
Signed-off-by: Feng Wu
---
arch/x86/include/asm/kvm_host.h | 4
virt/kvm/irq_comm.c | 41 +
2 files changed, 45 insertions(+)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 6ed0c30..7a41808 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -358,6 +358,7 @@ struct kvm_vcpu_arch {
struct kvm_lapic *apic;/* kernel irqchip context */
unsigned long apic_attention;
int32_t apic_arb_prio;
+ int32_t round_robin_counter;
int mp_state;
u64 ia32_misc_enable_msr;
bool tpr_access_reporting;
@@ -1093,4 +1094,7 @@ int kvm_pmu_read_pmc(struct kvm_vcpu *vcpu, unsigned pmc,
u64 *data);
void kvm_handle_pmu_event(struct kvm_vcpu *vcpu);
void kvm_deliver_pmi(struct kvm_vcpu *vcpu);
+bool kvm_find_dest_vcpu(struct kvm *kvm, struct kvm_lapic_irq *irq,
+ struct kvm_vcpu **dest_vcpu);
+
#endif /* _ASM_X86_KVM_HOST_H */
diff --git a/virt/kvm/irq_comm.c b/virt/kvm/irq_comm.c
index 963b899..f3c5d69 100644
--- a/virt/kvm/irq_comm.c
+++ b/virt/kvm/irq_comm.c
@@ -317,6 +317,47 @@ out:
return r;
}
+int kvm_compare_rr_counter(struct kvm_vcpu *vcpu1, struct kvm_vcpu *vcpu2)
+{
+ return vcpu1->arch.round_robin_counter -
+ vcpu2->arch.round_robin_counter;
+}
+
+bool kvm_find_dest_vcpu(struct kvm *kvm, struct kvm_lapic_irq *irq,
+ struct kvm_vcpu **dest_vcpu)
+{
+ int i, r = 0;
+ struct kvm_vcpu *vcpu, *dest = NULL;
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ if (!kvm_apic_present(vcpu))
+ continue;
+
+ if (!kvm_apic_match_dest(vcpu, NULL, irq->shorthand,
+ irq->dest_id, irq->dest_mode))
+ continue;
+
+ if (!kvm_is_dm_lowest_prio(irq)) {
+ r++;
+ *dest_vcpu = vcpu;
+ } else if (kvm_lapic_enabled(vcpu)) {
+ if (!dest)
+ dest = vcpu;
+ else if (kvm_compare_rr_counter(vcpu, dest) < 0)
+ dest = vcpu;
+ }
+ }
+
+ if (dest) {
+ dest->arch.round_robin_counter++;
+ *dest_vcpu = dest;
+ return true;
+ } else if (r == 1)
+ return true;
+
+ return false;
+}
+
#define IOAPIC_ROUTING_ENTRY(irq) \
{ .gsi = irq, .type = KVM_IRQ_ROUTING_IRQCHIP, \
.u.irqchip = { .irqchip = KVM_IRQCHIP_IOAPIC, .pin = (irq) } }
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
