Re: [PATCH 2/2] armv7 initial device passthrough support
On 6/25/2013 12:27 AM, Stuart Yoder wrote: >> >> We should avoid creating an environment with important functionality >> outside of the main tree, if at all possible. > > Also, as we architect that generic infrastructure we need to keep in mind that > there are important use cases for doing I/O in user space that are not > KVM guests-- just normal applications that need direct device > access. Yes that's a good point especially data plane NE, also LTE has these use cases at the radio side. > > Stuart > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] armv7 initial device passthrough support
On 6/24/2013 10:01 PM, Christoffer Dall wrote: >> There are many other latency/perf. reqs for NFV related to RT, >> essentially Guest must run near native. In the end it may turn out this >> may need to be outside of main tree we'll see. >> > It doesn't sound like this will be the end result. Everything that you > try to do in your patch set can be accomplished using VFIO and a more > generic infrastructure for virtual IRQ integration with KVM and user > space. I mentioned in previous email we will pursue VFIO, but even at that VFIO is a starting point for NFV. > > We should avoid creating an environment with important functionality > outside of the main tree, if at all possible. Of course that would be ideal but with NFV it may be more involved. This is similar Linux and TEM adaption around 04/05. We wanted to adapt Linux but it lacked required features that's when CGL specifications came into play to provide guidance a lot of features (TIPC, OpenIMPI, preempt_rt, AEM) lived outside mainline, supported by OS vendors delivering CGL compliant distro, while others decided to stick with IT, penetrating some applications like HLR. With NFV a likely scenario may evolve, TEMs need to start demonstrating to operators fixed and wireless virtualization use cases. The only significant difference is that unlike CGL for Linux, KVM has nor real representation and understanding of NFV reqs (as opposed to proprietary vendors). I can't speak for all TEMs but it's likely they will go off on their own to demo/proto-type and worry about Open Source acceptance later. > > -Christoffer > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] armv7 initial device passthrough support
On Mon, Jun 24, 2013 at 3:01 PM, Christoffer Dall wrote: > On Mon, Jun 24, 2013 at 10:08:08AM +0200, Mario Smarduch wrote: >> >> >> On 6/15/2013 5:47 PM, Paolo Bonzini wrote: >> > Il 13/06/2013 11:19, Mario Smarduch ha scritto: >> >> Updated Device Passthrough Patch. >> >> - optimized IRQ->CPU->vCPU binding, irq is installed once >> >> - added dynamic IRQ affinity on schedule in >> >> - added documentation and few other coding recommendations. >> >> >> >> Per earlier discussion VFIO is our target but we like >> >> something earlier to work with to tackle performance >> >> latency issue (some ARM related) for device passthrough >> >> while we migrate towards VFIO. >> > >> > I don't think this is acceptable upstream, unfortunately. KVM device >> > assignment is deprecated and we should not add more users. >> That's fine we'll work our way towards dev-tree VFIO reusing what we can >> working with the community. >> >> At this point we're more concerned with numbers and best practices as >> opposed to mechanism this part will be time consuming. >> VFIO will be more background for us. >> >> > >> > What are the latency issues you have? >> >> Our focus now is on IRQ latency and throughput. Right now it appears lowest >> latency >> is 2x + exit/enter + IRQ injection overhead. We can't tolerate additional >> IPIs or deferred IRQ injection approaches. We're looking for numbers closer >> to what IBMs ELI managed. Also high res timers which ARM Virt. Ext supports >> very well. Exitless interrupts which ARM handles very well too. There are >> some future hw ARM interrupt enhancements coming up which may help a lot as >> well. >> >> There are many other latency/perf. reqs for NFV related to RT, >> essentially Guest must run near native. In the end it may turn out this >> may need to be outside of main tree we'll see. >> > It doesn't sound like this will be the end result. Everything that you > try to do in your patch set can be accomplished using VFIO and a more > generic infrastructure for virtual IRQ integration with KVM and user > space. > > We should avoid creating an environment with important functionality > outside of the main tree, if at all possible. Also, as we architect that generic infrastructure we need to keep in mind that there are important use cases for doing I/O in user space that are not KVM guests-- just normal applications that need direct device access. Stuart -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] armv7 initial device passthrough support
On Mon, Jun 24, 2013 at 10:08:08AM +0200, Mario Smarduch wrote: > > > On 6/15/2013 5:47 PM, Paolo Bonzini wrote: > > Il 13/06/2013 11:19, Mario Smarduch ha scritto: > >> Updated Device Passthrough Patch. > >> - optimized IRQ->CPU->vCPU binding, irq is installed once > >> - added dynamic IRQ affinity on schedule in > >> - added documentation and few other coding recommendations. > >> > >> Per earlier discussion VFIO is our target but we like > >> something earlier to work with to tackle performance > >> latency issue (some ARM related) for device passthrough > >> while we migrate towards VFIO. > > > > I don't think this is acceptable upstream, unfortunately. KVM device > > assignment is deprecated and we should not add more users. > That's fine we'll work our way towards dev-tree VFIO reusing what we can > working with the community. > > At this point we're more concerned with numbers and best practices as > opposed to mechanism this part will be time consuming. > VFIO will be more background for us. > > > > > What are the latency issues you have? > > Our focus now is on IRQ latency and throughput. Right now it appears lowest > latency > is 2x + exit/enter + IRQ injection overhead. We can't tolerate additional > IPIs or deferred IRQ injection approaches. We're looking for numbers closer > to what IBMs ELI managed. Also high res timers which ARM Virt. Ext supports > very well. Exitless interrupts which ARM handles very well too. There are > some future hw ARM interrupt enhancements coming up which may help a lot as > well. > > There are many other latency/perf. reqs for NFV related to RT, > essentially Guest must run near native. In the end it may turn out this > may need to be outside of main tree we'll see. > It doesn't sound like this will be the end result. Everything that you try to do in your patch set can be accomplished using VFIO and a more generic infrastructure for virtual IRQ integration with KVM and user space. We should avoid creating an environment with important functionality outside of the main tree, if at all possible. -Christoffer -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] armv7 initial device passthrough support
On 6/15/2013 5:47 PM, Paolo Bonzini wrote: > Il 13/06/2013 11:19, Mario Smarduch ha scritto: >> Updated Device Passthrough Patch. >> - optimized IRQ->CPU->vCPU binding, irq is installed once >> - added dynamic IRQ affinity on schedule in >> - added documentation and few other coding recommendations. >> >> Per earlier discussion VFIO is our target but we like >> something earlier to work with to tackle performance >> latency issue (some ARM related) for device passthrough >> while we migrate towards VFIO. > > I don't think this is acceptable upstream, unfortunately. KVM device > assignment is deprecated and we should not add more users. That's fine we'll work our way towards dev-tree VFIO reusing what we can working with the community. At this point we're more concerned with numbers and best practices as opposed to mechanism this part will be time consuming. VFIO will be more background for us. > > What are the latency issues you have? Our focus now is on IRQ latency and throughput. Right now it appears lowest latency is 2x + exit/enter + IRQ injection overhead. We can't tolerate additional IPIs or deferred IRQ injection approaches. We're looking for numbers closer to what IBMs ELI managed. Also high res timers which ARM Virt. Ext supports very well. Exitless interrupts which ARM handles very well too. There are some future hw ARM interrupt enhancements coming up which may help a lot as well. There are many other latency/perf. reqs for NFV related to RT, essentially Guest must run near native. In the end it may turn out this may need to be outside of main tree we'll see. - Mario > > Paolo > >> - Mario -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] armv7 initial device passthrough support
Il 13/06/2013 11:19, Mario Smarduch ha scritto:
> Updated Device Passthrough Patch.
> - optimized IRQ->CPU->vCPU binding, irq is installed once
> - added dynamic IRQ affinity on schedule in
> - added documentation and few other coding recommendations.
>
> Per earlier discussion VFIO is our target but we like
> something earlier to work with to tackle performance
> latency issue (some ARM related) for device passthrough
> while we migrate towards VFIO.
I don't think this is acceptable upstream, unfortunately. KVM device
assignment is deprecated and we should not add more users.
What are the latency issues you have?
Paolo
> - Mario
>
>
> Signed-off-by: Mario Smarduch
> ---
> arch/arm/include/asm/kvm_host.h | 31 +
> arch/arm/include/asm/kvm_vgic.h | 10 ++
> arch/arm/kvm/Makefile |1 +
> arch/arm/kvm/arm.c | 80 +
> arch/arm/kvm/assign-dev.c | 248
> +++
> arch/arm/kvm/vgic.c | 134 +
> include/linux/irqchip/arm-gic.h |1 +
> include/uapi/linux/kvm.h| 33 ++
> 8 files changed, 538 insertions(+)
> create mode 100644 arch/arm/kvm/assign-dev.c
>
> diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
> index 57cb786..c85c3a0 100644
> --- a/arch/arm/include/asm/kvm_host.h
> +++ b/arch/arm/include/asm/kvm_host.h
> @@ -67,6 +67,10 @@ struct kvm_arch {
>
> /* Interrupt controller */
> struct vgic_distvgic;
> +
> + /* Device Passthrough Fields */
> + struct list_headassigned_dev_head;
> + struct mutexdev_passthrough_lock;
> };
>
> #define KVM_NR_MEM_OBJS 40
> @@ -146,6 +150,13 @@ struct kvm_vcpu_stat {
> u32 halt_wakeup;
> };
>
> +struct kvm_arm_assigned_dev_kernel {
> + struct list_head list;
> + struct kvm_arm_assigned_device dev;
> + irqreturn_t (*irq_handler)(int, void *);
> + unsigned long vcpuid_irq_arg;
> +};
> +
> struct kvm_vcpu_init;
> int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
> const struct kvm_vcpu_init *init);
> @@ -157,6 +168,26 @@ int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct
> kvm_one_reg *reg);
> u64 kvm_call_hyp(void *hypfn, ...);
> void force_vm_exit(const cpumask_t *mask);
>
> +#ifdef CONFIG_KVM_ARM_INT_PRIO_DROP
> +int kvm_arm_get_device_resources(struct kvm *,
> + struct kvm_arm_get_device_resources *);
> +int kvm_arm_assign_device(struct kvm *, struct kvm_arm_assigned_device *);
> +void kvm_arm_setdev_irq_affinity(struct kvm_vcpu *vcpu, int cpu);
> +#else
> +static inline int kvm_arm_get_device_resources(struct kvm *k, struct
> kvm_arm_get_device_resources *r)
> +{
> + return -1;
> +}
> +static inline int kvm_arm_assign_device(struct kvm *k, struct
> kvm_arm_assigned_device *d)
> +{
> + return -1;
> +}
> +
> +static inline void kvm_arm_setdev_irq_affinity(struct kvm_vcpu *vcpu, int
> cpu)
> +{
> +}
> +#endif
> +
> #define KVM_ARCH_WANT_MMU_NOTIFIER
> struct kvm;
> int kvm_unmap_hva(struct kvm *kvm, unsigned long hva);
> diff --git a/arch/arm/include/asm/kvm_vgic.h b/arch/arm/include/asm/kvm_vgic.h
> index 343744e..fb6afd2 100644
> --- a/arch/arm/include/asm/kvm_vgic.h
> +++ b/arch/arm/include/asm/kvm_vgic.h
> @@ -107,6 +107,16 @@ struct vgic_dist {
>
> /* Bitmap indicating which CPU has something pending */
> unsigned long irq_pending_on_cpu;
> +
> + /* Device passthrough fields */
> + /* Host irq to guest irq mapping */
> + u8 guest_irq[VGIC_NR_SHARED_IRQS];
> +
> + /* Pending passthruogh irq */
> + struct vgic_bitmap passthrough_spi_pending;
> +
> + /* At least one passthrough IRQ pending for some vCPU */
> + u32 passthrough_pending;
> #endif
> };
>
> diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
> index 53c5ed8..823fc38 100644
> --- a/arch/arm/kvm/Makefile
> +++ b/arch/arm/kvm/Makefile
> @@ -21,3 +21,4 @@ obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
> obj-y += coproc.o coproc_a15.o mmio.o psci.o perf.o
> obj-$(CONFIG_KVM_ARM_VGIC) += vgic.o
> obj-$(CONFIG_KVM_ARM_TIMER) += arch_timer.o
> +obj-$(CONFIG_KVM_ARM_INT_PRIO_DROP) += assign-dev.o
> diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
> index 37d216d..ba54c64 100644
> --- a/arch/arm/kvm/arm.c
> +++ b/arch/arm/kvm/arm.c
> @@ -26,6 +26,8 @@
> #include
> #include
> #include
> +#include
> +#include
> #include
>
> #define CREATE_TRACE_POINTS
> @@ -43,6 +45,7 @@
> #include
> #include
> #include
> +#include
>
> #ifdef REQUIRES_VIRT
> __asm__(".arch_extension virt");
> @@ -139,6 +142,11 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
>
> /* Mark the initial VMID generation invalid */
> kvm->arch.vmid_gen = 0;
> + /*
> + * Initialize Dev Passthrough Fields
> + */
> + INIT_LIST_HEAD(&kvm->arch.assigned_dev_head);
[PATCH 2/2] armv7 initial device passthrough support
Updated Device Passthrough Patch.
- optimized IRQ->CPU->vCPU binding, irq is installed once
- added dynamic IRQ affinity on schedule in
- added documentation and few other coding recommendations.
Per earlier discussion VFIO is our target but we like
something earlier to work with to tackle performance
latency issue (some ARM related) for device passthrough
while we migrate towards VFIO.
- Mario
Signed-off-by: Mario Smarduch
---
arch/arm/include/asm/kvm_host.h | 31 +
arch/arm/include/asm/kvm_vgic.h | 10 ++
arch/arm/kvm/Makefile |1 +
arch/arm/kvm/arm.c | 80 +
arch/arm/kvm/assign-dev.c | 248 +++
arch/arm/kvm/vgic.c | 134 +
include/linux/irqchip/arm-gic.h |1 +
include/uapi/linux/kvm.h| 33 ++
8 files changed, 538 insertions(+)
create mode 100644 arch/arm/kvm/assign-dev.c
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 57cb786..c85c3a0 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -67,6 +67,10 @@ struct kvm_arch {
/* Interrupt controller */
struct vgic_distvgic;
+
+ /* Device Passthrough Fields */
+ struct list_headassigned_dev_head;
+ struct mutexdev_passthrough_lock;
};
#define KVM_NR_MEM_OBJS 40
@@ -146,6 +150,13 @@ struct kvm_vcpu_stat {
u32 halt_wakeup;
};
+struct kvm_arm_assigned_dev_kernel {
+ struct list_head list;
+ struct kvm_arm_assigned_device dev;
+ irqreturn_t (*irq_handler)(int, void *);
+ unsigned long vcpuid_irq_arg;
+};
+
struct kvm_vcpu_init;
int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
const struct kvm_vcpu_init *init);
@@ -157,6 +168,26 @@ int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct
kvm_one_reg *reg);
u64 kvm_call_hyp(void *hypfn, ...);
void force_vm_exit(const cpumask_t *mask);
+#ifdef CONFIG_KVM_ARM_INT_PRIO_DROP
+int kvm_arm_get_device_resources(struct kvm *,
+ struct kvm_arm_get_device_resources *);
+int kvm_arm_assign_device(struct kvm *, struct kvm_arm_assigned_device *);
+void kvm_arm_setdev_irq_affinity(struct kvm_vcpu *vcpu, int cpu);
+#else
+static inline int kvm_arm_get_device_resources(struct kvm *k, struct
kvm_arm_get_device_resources *r)
+{
+ return -1;
+}
+static inline int kvm_arm_assign_device(struct kvm *k, struct
kvm_arm_assigned_device *d)
+{
+ return -1;
+}
+
+static inline void kvm_arm_setdev_irq_affinity(struct kvm_vcpu *vcpu, int cpu)
+{
+}
+#endif
+
#define KVM_ARCH_WANT_MMU_NOTIFIER
struct kvm;
int kvm_unmap_hva(struct kvm *kvm, unsigned long hva);
diff --git a/arch/arm/include/asm/kvm_vgic.h b/arch/arm/include/asm/kvm_vgic.h
index 343744e..fb6afd2 100644
--- a/arch/arm/include/asm/kvm_vgic.h
+++ b/arch/arm/include/asm/kvm_vgic.h
@@ -107,6 +107,16 @@ struct vgic_dist {
/* Bitmap indicating which CPU has something pending */
unsigned long irq_pending_on_cpu;
+
+ /* Device passthrough fields */
+ /* Host irq to guest irq mapping */
+ u8 guest_irq[VGIC_NR_SHARED_IRQS];
+
+ /* Pending passthruogh irq */
+ struct vgic_bitmap passthrough_spi_pending;
+
+ /* At least one passthrough IRQ pending for some vCPU */
+ u32 passthrough_pending;
#endif
};
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index 53c5ed8..823fc38 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -21,3 +21,4 @@ obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
obj-y += coproc.o coproc_a15.o mmio.o psci.o perf.o
obj-$(CONFIG_KVM_ARM_VGIC) += vgic.o
obj-$(CONFIG_KVM_ARM_TIMER) += arch_timer.o
+obj-$(CONFIG_KVM_ARM_INT_PRIO_DROP) += assign-dev.o
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 37d216d..ba54c64 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -26,6 +26,8 @@
#include
#include
#include
+#include
+#include
#include
#define CREATE_TRACE_POINTS
@@ -43,6 +45,7 @@
#include
#include
#include
+#include
#ifdef REQUIRES_VIRT
__asm__(".arch_extension virt");
@@ -139,6 +142,11 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
/* Mark the initial VMID generation invalid */
kvm->arch.vmid_gen = 0;
+ /*
+* Initialize Dev Passthrough Fields
+*/
+ INIT_LIST_HEAD(&kvm->arch.assigned_dev_head);
+ mutex_init(&kvm->arch.dev_passthrough_lock);
return ret;
out_free_stage2_pgd:
@@ -169,6 +177,40 @@ int kvm_arch_create_memslot(struct kvm_memory_slot *slot,
unsigned long npages)
void kvm_arch_destroy_vm(struct kvm *kvm)
{
int i;
+ struct list_head *dev_list_ptr = &kvm->arch.assigned_dev_head;
+ struct list_head *ptr, *q;
+ struct kvm_arm_assigned_dev_kernel *assigned_dev = NULL;
+ u64 hpa;
+
