[Bug 77871] New: USB device passthrough to Virtual GUEST system from Linux
https://bugzilla.kernel.org/show_bug.cgi?id=77871 Bug ID: 77871 Summary: USB device passthrough to Virtual GUEST system from Linux Product: Virtualization Version: unspecified Kernel Version: 3.14.6 or 3.14.7 Hardware: All OS: Linux Tree: Fedora Status: NEW Severity: high Priority: P1 Component: kvm Assignee: virtualization_...@kernel-bugs.osdl.org Reporter: ertaner...@gmail.com Regression: No Created attachment 139701 -- https://bugzilla.kernel.org/attachment.cgi?id=139701action=edit Full log file. After kernel 3.14.6 and 3.14.7 upgrade at Fedora when I try passthrough any USB device to Virtual GUEST then kerel create log some time after rsyslog demon top collect log due rate limit. I try that problem with Qemu KVM and Virtualbox. Also I add full log file to this ticket. I get that log from Fedora 20 when I use Virtualbox but also KVM have same problem. -##- Jun 13 10:44:58 homless kernel: WARNING: CPU: 0 PID: 15753 at drivers/usb/core/urb.c:476 usb_submit_urb+0x28d/0x5c0() Jun 13 10:44:58 homless kernel: usb 3-2.2: BOGUS urb flags, 1 -- 0 Jun 13 10:44:58 homless kernel: Modules linked in: ax88179_178a usbnet rt2800usb rt2x00usb rt2800lib rt2x00lib crc_ccitt rtl8187 eeprom_93cx6 nls_utf8 udf crc_itu_t usb_storage fuse ccm nf_conntrack_tftp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter bnep bluetooth 6lowpan_iphc ip6t_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle vboxpci(OF) vboxnetadp(OF) iptable_security vboxnetflt(OF) iptable_raw vboxdrv(OF) bbswitch(OF) iTCO_wdt rtsx_pci_sdmmc iTCO_vendor_support rtsx_pci_ms mmc_core memstick arc4 uvcvideo videobuf2_vmalloc videobuf2_memops Jun 13 10:44:58 homless kernel: videobuf2_core videodev media snd_hda_codec_hdmi snd_hda_codec_via snd_hda_codec_generic x86_pkg_temp_thermal snd_hda_intel coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ath9k ghash_clmulni_intel ath9k_common snd_hda_codec snd_hwdep ath9k_hw snd_seq snd_seq_device ath snd_pcm mac80211 microcode joydev serio_raw cfg80211 r8169 rtsx_pci mii snd_timer lpc_ich snd mfd_core rfkill i2c_i801 soundcore mei_me shpchp mei tpm_infineon tpm_tis tpm acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd binfmt_misc sunrpc mxm_wmi i915 i2c_algo_bit drm_kms_helper drm i2c_core video wmi [last unloaded: nvidia] Jun 13 10:44:58 homless kernel: CPU: 0 PID: 15753 Comm: EMT-1 Tainted: PF W O 3.14.7-200.fc20.x86_64 #1 Jun 13 10:44:58 homless kernel: Hardware name: CLEVO CO. W110ER /W110ER , BIOS 1.02.11 PM v3 07/01/20 Jun 13 10:44:58 homless kernel: bf4461af 8802a0239ca8 816f04b2 Jun 13 10:44:58 homless kernel: 8802a0239cf0 8802a0239ce0 8108a1cd 8802e99e79c0 Jun 13 10:44:58 homless kernel: 88036a43e800 0002 0001 0020 Jun 13 10:44:58 homless kernel: Call Trace: Jun 13 10:44:58 homless kernel: [816f04b2] dump_stack+0x45/0x56 Jun 13 10:44:58 homless kernel: [8108a1cd] warn_slowpath_common+0x7d/0xa0 Jun 13 10:44:58 homless kernel: [8108a24c] warn_slowpath_fmt+0x5c/0x80 Jun 13 10:44:58 homless kernel: [814e4d4d] usb_submit_urb+0x28d/0x5c0 Jun 13 10:44:58 homless kernel: [814f0e0d] proc_do_submiturb+0xc6d/0xcc0 Jun 13 10:44:58 homless kernel: [811cc265] ? __kmalloc+0x55/0x240 Jun 13 10:44:58 homless kernel: [a10ace8d] ? rtR0MemAllocEx+0x19d/0x280 [vboxdrv] Jun 13 10:44:58 homless kernel: [814f130f] usbdev_do_ioctl+0x4af/0x1080 Jun 13 10:44:58 homless kernel: [a10acfca] ? rtR0MemFree+0x5a/0x70 [vboxdrv] Jun 13 10:44:58 homless kernel: [814f1f0e] usbdev_ioctl+0xe/0x20 Jun 13 10:44:58 homless kernel: [811fd0f0] do_vfs_ioctl+0x2e0/0x4a0 Jun 13 10:44:58 homless kernel: [811fd331] SyS_ioctl+0x81/0xa0 Jun 13 10:44:58 homless kernel: [81121eb6] ? __audit_syscall_exit+0x1f6/0x2a0 Jun 13 10:44:58 homless kernel: [81700869] system_call_fastpath+0x16/0x1b Jun 13 10:44:58 homless kernel: ---[ end trace 0fe71e2b17d4081e ]--- Jun 13 10:44:58 homless kernel: xhci_hcd :00:14.0: WARN Event TRB for slot 4 ep 2 with no TDs queued? Jun 13 10:44:58 homless kernel: xhci_hcd :00:14.0: WARN Event TRB for slot 4 ep 2 with no TDs queued? Jun 13 10:44:58 homless kernel: xhci_hcd :00:14.0: WARN Event TRB for slot 4 ep 2 with no TDs queued? Jun 13 10:44:58 homless kernel: xhci_hcd :00:14.0: WARN Event TRB for slot 4 ep 2 with no TDs
Re: [PATCH 2/2] armv7 initial device passthrough support
On 6/24/2013 10:01 PM, Christoffer Dall wrote: There are many other latency/perf. reqs for NFV related to RT, essentially Guest must run near native. In the end it may turn out this may need to be outside of main tree we'll see. It doesn't sound like this will be the end result. Everything that you try to do in your patch set can be accomplished using VFIO and a more generic infrastructure for virtual IRQ integration with KVM and user space. I mentioned in previous email we will pursue VFIO, but even at that VFIO is a starting point for NFV. We should avoid creating an environment with important functionality outside of the main tree, if at all possible. Of course that would be ideal but with NFV it may be more involved. This is similar Linux and TEM adaption around 04/05. We wanted to adapt Linux but it lacked required features that's when CGL specifications came into play to provide guidance a lot of features (TIPC, OpenIMPI, preempt_rt, AEM) lived outside mainline, supported by OS vendors delivering CGL compliant distro, while others decided to stick with IT, penetrating some applications like HLR. With NFV a likely scenario may evolve, TEMs need to start demonstrating to operators fixed and wireless virtualization use cases. The only significant difference is that unlike CGL for Linux, KVM has nor real representation and understanding of NFV reqs (as opposed to proprietary vendors). I can't speak for all TEMs but it's likely they will go off on their own to demo/proto-type and worry about Open Source acceptance later. -Christoffer -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] armv7 initial device passthrough support
On 6/25/2013 12:27 AM, Stuart Yoder wrote: We should avoid creating an environment with important functionality outside of the main tree, if at all possible. Also, as we architect that generic infrastructure we need to keep in mind that there are important use cases for doing I/O in user space that are not KVM guests-- just normal applications that need direct device access. Yes that's a good point especially data plane NE, also LTE has these use cases at the radio side. Stuart -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] armv7 initial device passthrough support
On 6/15/2013 5:47 PM, Paolo Bonzini wrote: Il 13/06/2013 11:19, Mario Smarduch ha scritto: Updated Device Passthrough Patch. - optimized IRQ-CPU-vCPU binding, irq is installed once - added dynamic IRQ affinity on schedule in - added documentation and few other coding recommendations. Per earlier discussion VFIO is our target but we like something earlier to work with to tackle performance latency issue (some ARM related) for device passthrough while we migrate towards VFIO. I don't think this is acceptable upstream, unfortunately. KVM device assignment is deprecated and we should not add more users. That's fine we'll work our way towards dev-tree VFIO reusing what we can working with the community. At this point we're more concerned with numbers and best practices as opposed to mechanism this part will be time consuming. VFIO will be more background for us. What are the latency issues you have? Our focus now is on IRQ latency and throughput. Right now it appears lowest latency is 2x + exit/enter + IRQ injection overhead. We can't tolerate additional IPIs or deferred IRQ injection approaches. We're looking for numbers closer to what IBMs ELI managed. Also high res timers which ARM Virt. Ext supports very well. Exitless interrupts which ARM handles very well too. There are some future hw ARM interrupt enhancements coming up which may help a lot as well. There are many other latency/perf. reqs for NFV related to RT, essentially Guest must run near native. In the end it may turn out this may need to be outside of main tree we'll see. - Mario Paolo - Mario -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] armv7 initial device passthrough support
On Mon, Jun 24, 2013 at 10:08:08AM +0200, Mario Smarduch wrote: On 6/15/2013 5:47 PM, Paolo Bonzini wrote: Il 13/06/2013 11:19, Mario Smarduch ha scritto: Updated Device Passthrough Patch. - optimized IRQ-CPU-vCPU binding, irq is installed once - added dynamic IRQ affinity on schedule in - added documentation and few other coding recommendations. Per earlier discussion VFIO is our target but we like something earlier to work with to tackle performance latency issue (some ARM related) for device passthrough while we migrate towards VFIO. I don't think this is acceptable upstream, unfortunately. KVM device assignment is deprecated and we should not add more users. That's fine we'll work our way towards dev-tree VFIO reusing what we can working with the community. At this point we're more concerned with numbers and best practices as opposed to mechanism this part will be time consuming. VFIO will be more background for us. What are the latency issues you have? Our focus now is on IRQ latency and throughput. Right now it appears lowest latency is 2x + exit/enter + IRQ injection overhead. We can't tolerate additional IPIs or deferred IRQ injection approaches. We're looking for numbers closer to what IBMs ELI managed. Also high res timers which ARM Virt. Ext supports very well. Exitless interrupts which ARM handles very well too. There are some future hw ARM interrupt enhancements coming up which may help a lot as well. There are many other latency/perf. reqs for NFV related to RT, essentially Guest must run near native. In the end it may turn out this may need to be outside of main tree we'll see. It doesn't sound like this will be the end result. Everything that you try to do in your patch set can be accomplished using VFIO and a more generic infrastructure for virtual IRQ integration with KVM and user space. We should avoid creating an environment with important functionality outside of the main tree, if at all possible. -Christoffer -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] armv7 initial device passthrough support
On Mon, Jun 24, 2013 at 3:01 PM, Christoffer Dall christoffer.d...@linaro.org wrote: On Mon, Jun 24, 2013 at 10:08:08AM +0200, Mario Smarduch wrote: On 6/15/2013 5:47 PM, Paolo Bonzini wrote: Il 13/06/2013 11:19, Mario Smarduch ha scritto: Updated Device Passthrough Patch. - optimized IRQ-CPU-vCPU binding, irq is installed once - added dynamic IRQ affinity on schedule in - added documentation and few other coding recommendations. Per earlier discussion VFIO is our target but we like something earlier to work with to tackle performance latency issue (some ARM related) for device passthrough while we migrate towards VFIO. I don't think this is acceptable upstream, unfortunately. KVM device assignment is deprecated and we should not add more users. That's fine we'll work our way towards dev-tree VFIO reusing what we can working with the community. At this point we're more concerned with numbers and best practices as opposed to mechanism this part will be time consuming. VFIO will be more background for us. What are the latency issues you have? Our focus now is on IRQ latency and throughput. Right now it appears lowest latency is 2x + exit/enter + IRQ injection overhead. We can't tolerate additional IPIs or deferred IRQ injection approaches. We're looking for numbers closer to what IBMs ELI managed. Also high res timers which ARM Virt. Ext supports very well. Exitless interrupts which ARM handles very well too. There are some future hw ARM interrupt enhancements coming up which may help a lot as well. There are many other latency/perf. reqs for NFV related to RT, essentially Guest must run near native. In the end it may turn out this may need to be outside of main tree we'll see. It doesn't sound like this will be the end result. Everything that you try to do in your patch set can be accomplished using VFIO and a more generic infrastructure for virtual IRQ integration with KVM and user space. We should avoid creating an environment with important functionality outside of the main tree, if at all possible. Also, as we architect that generic infrastructure we need to keep in mind that there are important use cases for doing I/O in user space that are not KVM guests-- just normal applications that need direct device access. Stuart -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] armv7 initial device passthrough support
Il 13/06/2013 11:19, Mario Smarduch ha scritto:
Updated Device Passthrough Patch.
- optimized IRQ-CPU-vCPU binding, irq is installed once
- added dynamic IRQ affinity on schedule in
- added documentation and few other coding recommendations.
Per earlier discussion VFIO is our target but we like
something earlier to work with to tackle performance
latency issue (some ARM related) for device passthrough
while we migrate towards VFIO.
I don't think this is acceptable upstream, unfortunately. KVM device
assignment is deprecated and we should not add more users.
What are the latency issues you have?
Paolo
- Mario
Signed-off-by: Mario Smarduch mario.smard...@huawei.com
---
arch/arm/include/asm/kvm_host.h | 31 +
arch/arm/include/asm/kvm_vgic.h | 10 ++
arch/arm/kvm/Makefile |1 +
arch/arm/kvm/arm.c | 80 +
arch/arm/kvm/assign-dev.c | 248
+++
arch/arm/kvm/vgic.c | 134 +
include/linux/irqchip/arm-gic.h |1 +
include/uapi/linux/kvm.h| 33 ++
8 files changed, 538 insertions(+)
create mode 100644 arch/arm/kvm/assign-dev.c
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 57cb786..c85c3a0 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -67,6 +67,10 @@ struct kvm_arch {
/* Interrupt controller */
struct vgic_distvgic;
+
+ /* Device Passthrough Fields */
+ struct list_headassigned_dev_head;
+ struct mutexdev_passthrough_lock;
};
#define KVM_NR_MEM_OBJS 40
@@ -146,6 +150,13 @@ struct kvm_vcpu_stat {
u32 halt_wakeup;
};
+struct kvm_arm_assigned_dev_kernel {
+ struct list_head list;
+ struct kvm_arm_assigned_device dev;
+ irqreturn_t (*irq_handler)(int, void *);
+ unsigned long vcpuid_irq_arg;
+};
+
struct kvm_vcpu_init;
int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
const struct kvm_vcpu_init *init);
@@ -157,6 +168,26 @@ int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct
kvm_one_reg *reg);
u64 kvm_call_hyp(void *hypfn, ...);
void force_vm_exit(const cpumask_t *mask);
+#ifdef CONFIG_KVM_ARM_INT_PRIO_DROP
+int kvm_arm_get_device_resources(struct kvm *,
+ struct kvm_arm_get_device_resources *);
+int kvm_arm_assign_device(struct kvm *, struct kvm_arm_assigned_device *);
+void kvm_arm_setdev_irq_affinity(struct kvm_vcpu *vcpu, int cpu);
+#else
+static inline int kvm_arm_get_device_resources(struct kvm *k, struct
kvm_arm_get_device_resources *r)
+{
+ return -1;
+}
+static inline int kvm_arm_assign_device(struct kvm *k, struct
kvm_arm_assigned_device *d)
+{
+ return -1;
+}
+
+static inline void kvm_arm_setdev_irq_affinity(struct kvm_vcpu *vcpu, int
cpu)
+{
+}
+#endif
+
#define KVM_ARCH_WANT_MMU_NOTIFIER
struct kvm;
int kvm_unmap_hva(struct kvm *kvm, unsigned long hva);
diff --git a/arch/arm/include/asm/kvm_vgic.h b/arch/arm/include/asm/kvm_vgic.h
index 343744e..fb6afd2 100644
--- a/arch/arm/include/asm/kvm_vgic.h
+++ b/arch/arm/include/asm/kvm_vgic.h
@@ -107,6 +107,16 @@ struct vgic_dist {
/* Bitmap indicating which CPU has something pending */
unsigned long irq_pending_on_cpu;
+
+ /* Device passthrough fields */
+ /* Host irq to guest irq mapping */
+ u8 guest_irq[VGIC_NR_SHARED_IRQS];
+
+ /* Pending passthruogh irq */
+ struct vgic_bitmap passthrough_spi_pending;
+
+ /* At least one passthrough IRQ pending for some vCPU */
+ u32 passthrough_pending;
#endif
};
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index 53c5ed8..823fc38 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -21,3 +21,4 @@ obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
obj-y += coproc.o coproc_a15.o mmio.o psci.o perf.o
obj-$(CONFIG_KVM_ARM_VGIC) += vgic.o
obj-$(CONFIG_KVM_ARM_TIMER) += arch_timer.o
+obj-$(CONFIG_KVM_ARM_INT_PRIO_DROP) += assign-dev.o
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 37d216d..ba54c64 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -26,6 +26,8 @@
#include linux/mman.h
#include linux/sched.h
#include linux/kvm.h
+#include linux/interrupt.h
+#include linux/ioport.h
#include trace/events/kvm.h
#define CREATE_TRACE_POINTS
@@ -43,6 +45,7 @@
#include asm/kvm_emulate.h
#include asm/kvm_coproc.h
#include asm/kvm_psci.h
+#include asm/kvm_host.h
#ifdef REQUIRES_VIRT
__asm__(.arch_extension virt);
@@ -139,6 +142,11 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
/* Mark the initial VMID generation invalid */
kvm-arch.vmid_gen = 0;
+ /*
+ * Initialize Dev Passthrough Fields
+ */
+ INIT_LIST_HEAD(kvm
[PATCH 2/2] armv7 initial device passthrough support
Updated Device Passthrough Patch.
- optimized IRQ-CPU-vCPU binding, irq is installed once
- added dynamic IRQ affinity on schedule in
- added documentation and few other coding recommendations.
Per earlier discussion VFIO is our target but we like
something earlier to work with to tackle performance
latency issue (some ARM related) for device passthrough
while we migrate towards VFIO.
- Mario
Signed-off-by: Mario Smarduch mario.smard...@huawei.com
---
arch/arm/include/asm/kvm_host.h | 31 +
arch/arm/include/asm/kvm_vgic.h | 10 ++
arch/arm/kvm/Makefile |1 +
arch/arm/kvm/arm.c | 80 +
arch/arm/kvm/assign-dev.c | 248 +++
arch/arm/kvm/vgic.c | 134 +
include/linux/irqchip/arm-gic.h |1 +
include/uapi/linux/kvm.h| 33 ++
8 files changed, 538 insertions(+)
create mode 100644 arch/arm/kvm/assign-dev.c
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 57cb786..c85c3a0 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -67,6 +67,10 @@ struct kvm_arch {
/* Interrupt controller */
struct vgic_distvgic;
+
+ /* Device Passthrough Fields */
+ struct list_headassigned_dev_head;
+ struct mutexdev_passthrough_lock;
};
#define KVM_NR_MEM_OBJS 40
@@ -146,6 +150,13 @@ struct kvm_vcpu_stat {
u32 halt_wakeup;
};
+struct kvm_arm_assigned_dev_kernel {
+ struct list_head list;
+ struct kvm_arm_assigned_device dev;
+ irqreturn_t (*irq_handler)(int, void *);
+ unsigned long vcpuid_irq_arg;
+};
+
struct kvm_vcpu_init;
int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
const struct kvm_vcpu_init *init);
@@ -157,6 +168,26 @@ int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct
kvm_one_reg *reg);
u64 kvm_call_hyp(void *hypfn, ...);
void force_vm_exit(const cpumask_t *mask);
+#ifdef CONFIG_KVM_ARM_INT_PRIO_DROP
+int kvm_arm_get_device_resources(struct kvm *,
+ struct kvm_arm_get_device_resources *);
+int kvm_arm_assign_device(struct kvm *, struct kvm_arm_assigned_device *);
+void kvm_arm_setdev_irq_affinity(struct kvm_vcpu *vcpu, int cpu);
+#else
+static inline int kvm_arm_get_device_resources(struct kvm *k, struct
kvm_arm_get_device_resources *r)
+{
+ return -1;
+}
+static inline int kvm_arm_assign_device(struct kvm *k, struct
kvm_arm_assigned_device *d)
+{
+ return -1;
+}
+
+static inline void kvm_arm_setdev_irq_affinity(struct kvm_vcpu *vcpu, int cpu)
+{
+}
+#endif
+
#define KVM_ARCH_WANT_MMU_NOTIFIER
struct kvm;
int kvm_unmap_hva(struct kvm *kvm, unsigned long hva);
diff --git a/arch/arm/include/asm/kvm_vgic.h b/arch/arm/include/asm/kvm_vgic.h
index 343744e..fb6afd2 100644
--- a/arch/arm/include/asm/kvm_vgic.h
+++ b/arch/arm/include/asm/kvm_vgic.h
@@ -107,6 +107,16 @@ struct vgic_dist {
/* Bitmap indicating which CPU has something pending */
unsigned long irq_pending_on_cpu;
+
+ /* Device passthrough fields */
+ /* Host irq to guest irq mapping */
+ u8 guest_irq[VGIC_NR_SHARED_IRQS];
+
+ /* Pending passthruogh irq */
+ struct vgic_bitmap passthrough_spi_pending;
+
+ /* At least one passthrough IRQ pending for some vCPU */
+ u32 passthrough_pending;
#endif
};
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index 53c5ed8..823fc38 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -21,3 +21,4 @@ obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
obj-y += coproc.o coproc_a15.o mmio.o psci.o perf.o
obj-$(CONFIG_KVM_ARM_VGIC) += vgic.o
obj-$(CONFIG_KVM_ARM_TIMER) += arch_timer.o
+obj-$(CONFIG_KVM_ARM_INT_PRIO_DROP) += assign-dev.o
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 37d216d..ba54c64 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -26,6 +26,8 @@
#include linux/mman.h
#include linux/sched.h
#include linux/kvm.h
+#include linux/interrupt.h
+#include linux/ioport.h
#include trace/events/kvm.h
#define CREATE_TRACE_POINTS
@@ -43,6 +45,7 @@
#include asm/kvm_emulate.h
#include asm/kvm_coproc.h
#include asm/kvm_psci.h
+#include asm/kvm_host.h
#ifdef REQUIRES_VIRT
__asm__(.arch_extension virt);
@@ -139,6 +142,11 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
/* Mark the initial VMID generation invalid */
kvm-arch.vmid_gen = 0;
+ /*
+* Initialize Dev Passthrough Fields
+*/
+ INIT_LIST_HEAD(kvm-arch.assigned_dev_head);
+ mutex_init(kvm-arch.dev_passthrough_lock);
return ret;
out_free_stage2_pgd:
@@ -169,6 +177,40 @@ int kvm_arch_create_memslot(struct kvm_memory_slot *slot,
unsigned long npages)
void kvm_arch_destroy_vm(struct kvm *kvm)
{
int i;
+ struct list_head
[PATCH 2/2] KVM: IOMMU: Use iommu_commit() in device-passthrough code
Use the new iommu_commit() function in the KVM device passthrough code. Signed-off-by: Joerg Roedel joerg.roe...@amd.com --- virt/kvm/iommu.c |8 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/virt/kvm/iommu.c b/virt/kvm/iommu.c index 62a9caf..87ef85a 100644 --- a/virt/kvm/iommu.c +++ b/virt/kvm/iommu.c @@ -187,6 +187,8 @@ int kvm_assign_device(struct kvm *kvm, PCI_SLOT(assigned_dev-host_devfn), PCI_FUNC(assigned_dev-host_devfn)); + iommu_commit(domain); + return 0; out_unmap: kvm_iommu_unmap_memslots(kvm); @@ -215,6 +217,8 @@ int kvm_deassign_device(struct kvm *kvm, PCI_SLOT(assigned_dev-host_devfn), PCI_FUNC(assigned_dev-host_devfn)); + iommu_commit(domain); + return 0; } @@ -235,6 +239,8 @@ int kvm_iommu_map_guest(struct kvm *kvm) if (r) goto out_unmap; + iommu_commit(kvm-arch.iommu_domain); + return 0; out_unmap: @@ -283,6 +289,8 @@ static void kvm_iommu_put_pages(struct kvm *kvm, gfn += unmap_pages; } + + iommu_commit(domain); } static int kvm_iommu_unmap_memslots(struct kvm *kvm) -- 1.7.4.1 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[Bug 29232] [VT-d] VT-d device passthrough fail to guest
https://bugzilla.kernel.org/show_bug.cgi?id=29232 Florian Mickler flor...@mickler.org changed: What|Removed |Added Status|NEW |RESOLVED Resolution||CODE_FIX --- Comment #3 from Florian Mickler flor...@mickler.org 2011-03-04 20:33:57 --- merged for v2.6.38-rc5: commit f00eaeea7a42b5ea327e9ce8839cb0b53d3bdb4e Author: Linus Torvalds torva...@linux-foundation.org Date: Sun Feb 13 07:50:50 2011 -0800 Revert pci: use security_capable() when checking capablities during config space read -- Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are watching the assignee of the bug. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[Bug 29232] [VT-d] VT-d device passthrough fail to guest
https://bugzilla.kernel.org/show_bug.cgi?id=29232 Florian Mickler flor...@mickler.org changed: What|Removed |Added Status|RESOLVED|CLOSED -- Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are watching the assignee of the bug. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[Bug 29232] [VT-d] VT-d device passthrough fail to guest
https://bugzilla.kernel.org/show_bug.cgi?id=29232 Jay Ren yongjie@intel.com changed: What|Removed |Added CC||yongjie@intel.com --- Comment #2 from Jay Ren yongjie@intel.com 2011-03-04 02:28:49 --- This bug has been verified. It's OK with b35049715 kvm.git. -- Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are watching the assignee of the bug. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[Bug 29232] [VT-d] VT-d device passthrough fail to guest
https://bugzilla.kernel.org/show_bug.cgi?id=29232 Rafael J. Wysocki r...@sisk.pl changed: What|Removed |Added CC||flor...@mickler.org, ||maciej.rute...@gmail.com, ||r...@sisk.pl Kernel Version||2.6.38-rc4 Version|unspecified |2.5 Blocks||27352 -- Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are watching the assignee of the bug. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[Bug 29232] New: [VT-d] VT-d device passthrough fail to guest
https://bugzilla.kernel.org/show_bug.cgi?id=29232 Summary: [VT-d] VT-d device passthrough fail to guest Product: Virtualization Version: unspecified Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: high Priority: P1 Component: kvm AssignedTo: virtualization_...@kernel-bugs.osdl.org ReportedBy: xudong@intel.com CC: a...@redhat.com Regression: Yes Environment: Host OS : IA32E Guest OS : IA32E kvm.git Commit: a685b38e272587e644fedd37269ddb82df21c052 qemu-kvm Commit: 671d89d6411655bb4f8058ce6eb86bb0bb8ec978 Host Kernel Version: 2.6.38-rc4 Bug detailed description: -- When assign one PCI/e device to guest, qemu report error and guest can not boot up. If do hot-add device to guest, qemu will be killed. [root@vt-nhm9 qemu-kvm]# qemu-system-x86_64 -m 512 -smp 2 -device pci-assign,host=13:00.0 -net none -hda /path/rhel5.img assigned_dev_pci_read: pread failed, ret = 0 errno = 22 Reproduce steps: 1) install latest kvm and qemu-kvm 2) qemu-system-x86_64 -m 512 -smp 2 -device pci-assign,host=13:00.0 -net none -hda /path/rhel5.img 3) there will be a qemu error -- Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are watching the assignee of the bug. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[Bug 29232] [VT-d] VT-d device passthrough fail to guest
https://bugzilla.kernel.org/show_bug.cgi?id=29232 alex.william...@redhat.com changed: What|Removed |Added CC||alex.william...@redhat.com --- Comment #1 from alex.william...@redhat.com 2011-02-16 16:43:26 --- This will be fixed with the 2.6.38-rc5 merge of kvm.git. Commit 47970b1b was bad and later reverted in f00eaeea. A corrected version was added in 683034fca. -- Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are watching the assignee of the bug. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 0/9] KVM: Improve IRQ assignment for device passthrough
On 11/16/2010 08:26 PM, Jan Kiszka wrote: Am 16.11.2010 17:55, Marcelo Tosatti wrote: On Mon, Nov 08, 2010 at 12:21:44PM +0100, Jan Kiszka wrote: Nine patches (yeah, it's getting more and more) to improve classic device assigment /wrt IRQs. Highlight is the last one that resolves the host IRQ sharing issue for all PCI 2.3 devices. Quite essential when passing non-MSI-ready devices like many USB host controllers. As there were concerns regarding the overhead of IRQ masking via the PCI config space, I did some micro-benchmarks. Well, the concerns are valid: disable_irq_nosync: ~600 cycles pci_2_3_irq_check_and_mask: ~6000 cycles (EHCI) ~22000 cycles (AR9287, with peaks10) Specifically the varying impact of the device like in the Atheros case is worrying (this device is actually known to cause horrible latencies to the host, but who knows what other devices do). So I decided to go with PCI-2.3 masking as default off in the to-be-sent qemu-kvm patch. Maybe something to consider vor VFIO as well. Looks fine to me. Michael, Alex? Also needs a rebase. I think Avi wanted to explore possibilities to avoid host_pci_2_3=on|off by switching between both modes on demand. I've some ideas, but still need to look into design details. Given that this would have impact on the ABI, I guess we better wait with merging 9/9. Indeed. Moreover, Avi preferred to skip the srcu conversion and run with a simply lock across kvm_set_irq (just like we do now). Yes. It was predicated on agreeing that deferring broadcast/multicast IPIs to a workqueue is possible. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5-light 0/6] KVM: Improve IRQ assignment for device passthrough
On 11/16/2010 11:30 PM, Jan Kiszka wrote: This is the rebased light version of the previous series, i.e. without PCI-2.3-based IRQ masking or any SRCU conversion. PCI-2.3 support is under rework to explore options for automatic mode switches. Looks good, especially the threaded irq - a real winner. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5-light 0/6] KVM: Improve IRQ assignment for device passthrough
On Tue, Nov 16, 2010 at 10:30:01PM +0100, Jan Kiszka wrote: This is the rebased light version of the previous series, i.e. without PCI-2.3-based IRQ masking or any SRCU conversion. PCI-2.3 support is under rework to explore options for automatic mode switches. Jan Kiszka (6): KVM: Clear assigned guest IRQ on release KVM: Switch assigned device IRQ forwarding to threaded handler KVM: Refactor IRQ names of assigned devices KVM: Save/restore state of assigned PCI device KVM: Clean up kvm_vm_ioctl_assigned_device KVM: Document device assigment API Documentation/kvm/api.txt | 178 + include/linux/kvm_host.h | 13 +--- virt/kvm/assigned-dev.c | 125 3 files changed, 227 insertions(+), 89 deletions(-) Acked-by: Michael S. Tsirkin m...@redhat.com -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5-light 0/6] KVM: Improve IRQ assignment for device passthrough
On Tue, Nov 16, 2010 at 10:30:01PM +0100, Jan Kiszka wrote: This is the rebased light version of the previous series, i.e. without PCI-2.3-based IRQ masking or any SRCU conversion. PCI-2.3 support is under rework to explore options for automatic mode switches. Jan Kiszka (6): KVM: Clear assigned guest IRQ on release KVM: Switch assigned device IRQ forwarding to threaded handler KVM: Refactor IRQ names of assigned devices KVM: Save/restore state of assigned PCI device KVM: Clean up kvm_vm_ioctl_assigned_device KVM: Document device assigment API Documentation/kvm/api.txt | 178 + include/linux/kvm_host.h | 13 +--- virt/kvm/assigned-dev.c | 125 3 files changed, 227 insertions(+), 89 deletions(-) Applied, thanks. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 0/9] KVM: Improve IRQ assignment for device passthrough
On Mon, Nov 08, 2010 at 12:21:44PM +0100, Jan Kiszka wrote: Nine patches (yeah, it's getting more and more) to improve classic device assigment /wrt IRQs. Highlight is the last one that resolves the host IRQ sharing issue for all PCI 2.3 devices. Quite essential when passing non-MSI-ready devices like many USB host controllers. As there were concerns regarding the overhead of IRQ masking via the PCI config space, I did some micro-benchmarks. Well, the concerns are valid: disable_irq_nosync: ~600 cycles pci_2_3_irq_check_and_mask: ~6000 cycles (EHCI) ~22000 cycles (AR9287, with peaks 10) Specifically the varying impact of the device like in the Atheros case is worrying (this device is actually known to cause horrible latencies to the host, but who knows what other devices do). So I decided to go with PCI-2.3 masking as default off in the to-be-sent qemu-kvm patch. Maybe something to consider vor VFIO as well. Looks fine to me. Michael, Alex? Also needs a rebase. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 0/9] KVM: Improve IRQ assignment for device passthrough
Am 16.11.2010 17:55, Marcelo Tosatti wrote: On Mon, Nov 08, 2010 at 12:21:44PM +0100, Jan Kiszka wrote: Nine patches (yeah, it's getting more and more) to improve classic device assigment /wrt IRQs. Highlight is the last one that resolves the host IRQ sharing issue for all PCI 2.3 devices. Quite essential when passing non-MSI-ready devices like many USB host controllers. As there were concerns regarding the overhead of IRQ masking via the PCI config space, I did some micro-benchmarks. Well, the concerns are valid: disable_irq_nosync: ~600 cycles pci_2_3_irq_check_and_mask: ~6000 cycles (EHCI) ~22000 cycles (AR9287, with peaks 10) Specifically the varying impact of the device like in the Atheros case is worrying (this device is actually known to cause horrible latencies to the host, but who knows what other devices do). So I decided to go with PCI-2.3 masking as default off in the to-be-sent qemu-kvm patch. Maybe something to consider vor VFIO as well. Looks fine to me. Michael, Alex? Also needs a rebase. I think Avi wanted to explore possibilities to avoid host_pci_2_3=on|off by switching between both modes on demand. I've some ideas, but still need to look into design details. Given that this would have impact on the ABI, I guess we better wait with merging 9/9. Moreover, Avi preferred to skip the srcu conversion and run with a simply lock across kvm_set_irq (just like we do now). But I could offer to rebase and repost the independent rest of the series tonight. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v5-light 0/6] KVM: Improve IRQ assignment for device passthrough
This is the rebased light version of the previous series, i.e. without PCI-2.3-based IRQ masking or any SRCU conversion. PCI-2.3 support is under rework to explore options for automatic mode switches. Jan Kiszka (6): KVM: Clear assigned guest IRQ on release KVM: Switch assigned device IRQ forwarding to threaded handler KVM: Refactor IRQ names of assigned devices KVM: Save/restore state of assigned PCI device KVM: Clean up kvm_vm_ioctl_assigned_device KVM: Document device assigment API Documentation/kvm/api.txt | 178 + include/linux/kvm_host.h | 13 +--- virt/kvm/assigned-dev.c | 125 3 files changed, 227 insertions(+), 89 deletions(-) -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5-light 0/6] KVM: Improve IRQ assignment for device passthrough
On Tue, 2010-11-16 at 22:30 +0100, Jan Kiszka wrote: This is the rebased light version of the previous series, i.e. without PCI-2.3-based IRQ masking or any SRCU conversion. PCI-2.3 support is under rework to explore options for automatic mode switches. Jan Kiszka (6): KVM: Clear assigned guest IRQ on release KVM: Switch assigned device IRQ forwarding to threaded handler KVM: Refactor IRQ names of assigned devices KVM: Save/restore state of assigned PCI device KVM: Clean up kvm_vm_ioctl_assigned_device KVM: Document device assigment API Documentation/kvm/api.txt | 178 + include/linux/kvm_host.h | 13 +--- virt/kvm/assigned-dev.c | 125 3 files changed, 227 insertions(+), 89 deletions(-) Series looks good to me. Acked-by: Alex Williamson alex.william...@redhat.com -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v4 0/9] KVM: Improve IRQ assignment for device passthrough
Nine patches (yeah, it's getting more and more) to improve classic device assigment /wrt IRQs. Highlight is the last one that resolves the host IRQ sharing issue for all PCI 2.3 devices. Quite essential when passing non-MSI-ready devices like many USB host controllers. As there were concerns regarding the overhead of IRQ masking via the PCI config space, I did some micro-benchmarks. Well, the concerns are valid: disable_irq_nosync: ~600 cycles pci_2_3_irq_check_and_mask: ~6000 cycles (EHCI) ~22000 cycles (AR9287, with peaks 10) Specifically the varying impact of the device like in the Atheros case is worrying (this device is actually known to cause horrible latencies to the host, but who knows what other devices do). So I decided to go with PCI-2.3 masking as default off in the to-be-sent qemu-kvm patch. Maybe something to consider vor VFIO as well. Changes in v4: - Allow user space to push its INTx mask state to the kernel and respect this on IRQ masking and delivery - Specify IRQF_ONESHOT for threaded IRQ of assigned device - Switch IRQ subsystem locking from RCU to SRCU - Fix for SRCU struct leakage - Small cleanup for kvm_vm_ioctl_assigned_device - Documentation for device assignment API (please check carefully if I got it right) Changes in v3: - Save/restore PCI device state across guest usage - Make PCI-2.3-based IRQ sharing configurable by user space - Fix potential nesting of pci_block_user_cfg_access - Do not track PCI-level IRQ masking in software, user space may change it concurrently - Optimize kvm_assigned_dev_ack_irq for the case another IRQ is already pending - Cleanups according to review comments Changes in v2: - Reworked IRQ forwarding path to use threaded IRQs (direct signalling from IRQ context does not work out of the box and may be too lengthy) - Refactored host IRQ naming of assigned devices (cosmetic change) - Avoid unmask on ack when the next IRQ is pending, rather reassert the guest line (PCI-2.3 patch) - Refactored PCI-2.3 patch (but still no control knob for shared mode - is that a must?) Jan Kiszka (9): KVM: Fix srcu struct leakage KVM: Switch IRQ subsystem to SRCU KVM: Clear assigned guest IRQ on release KVM: Switch assigned device IRQ forwarding to threaded handler KVM: Refactor IRQ names of assigned devices KVM: Save/restore state of assigned PCI device KVM: Clean up kvm_vm_ioctl_assigned_device KVM: Document device assigment API KVM: Allow host IRQ sharing for passed-through PCI 2.3 devices Documentation/kvm/api.txt | 203 arch/x86/kvm/x86.c|1 + include/linux/kvm.h |6 + include/linux/kvm_host.h | 16 +-- virt/kvm/assigned-dev.c | 383 ++-- virt/kvm/irq_comm.c | 31 ++-- virt/kvm/kvm_main.c | 20 ++- 7 files changed, 541 insertions(+), 119 deletions(-) -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2 0/4] KVM: Improve IRQ assignment for device passthrough
Four patches to improve classic device assigment /wrt IRQs. Highlight is the last one that resolves the host IRQ sharing issue for all PCI 2.3 devices. Quite essential when passing non-MSI-ready devices like many USB host controllers. Changes in v2: - Reworked IRQ forwarding path to use threaded IRQs (direct signalling from IRQ context does not work out of the box and may be too lengthy) - Refactored host IRQ naming of assigned devices (cosmetic change) - Avoid unmask on ack when the next IRQ is pending, rather reassert the guest line (PCI-2.3 patch) - Refactored PCI-2.3 patch (but still no control knob for shared mode - is that a must?) Jan Kiszka (4): KVM: Clear assigned guest IRQ on release KVM: Switch assigned device IRQ forwarding to threaded handler KVM: Refactor IRQ names of assigned devices KVM: Allow host IRQ sharing for passed-through PCI 2.3 devices include/linux/kvm_host.h | 14 +-- virt/kvm/assigned-dev.c | 279 + 2 files changed, 208 insertions(+), 85 deletions(-) -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/4] KVM: Improve IRQ assignment for device passthrough
- Refactored PCI-2.3 patch (but still no control knob for shared mode - is that a must?) I think it's the prudent thing to do. -- MST -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/4] KVM: Improve IRQ assignment for device passthrough
Am 02.11.2010 18:11, Michael S. Tsirkin wrote: - Refactored PCI-2.3 patch (but still no control knob for shared mode - is that a must?) I think it's the prudent thing to do. What a pity... Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/3] KVM: Improve IRQ assignment for device passthrough
Three patches to improve classic device assigment /wrt IRQs. Highlight is the last one that resolves the host IRQ sharing issue for all PCI 2.3 devices. Quite essential when passing non-MSI-ready devices like many USB host controllers. Jan Kiszka (3): KVM: Fold assigned interrupt work into IRQ handler KVM: Clear assigned guest IRQ on release KVM: Allow host IRQ sharing for passed-through PCI 2.3 devices include/linux/kvm_host.h |2 +- virt/kvm/assigned-dev.c | 215 ++ 2 files changed, 160 insertions(+), 57 deletions(-) -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
device passthrough
Hi, All: Is there any method to directly assign a device to Guest OS without VT-d? Thanks Mu-- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: device passthrough
* Mu Lin (m...@juniper.net) wrote: Is there any method to directly assign a device to Guest OS without VT-d? Assuming you mean a PCI device, no, there isn't. Without an IOMMU[1] you can't directly assign a PCI device to a guest (nor is it safe). There have been patches floating around to allow this, but they don't maintain secure isolation. thanks, -chris [1] VT-d is an Intel chipset feature, so you could certainly do it on an AMD platform that has an AMD IOMMU. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: device passthrough
Thanks, Chris, Do you know where is the patch, I just need something quick and dirty for now, my shining new board does have VT-d but the BIOS is not ready yet, I want to have something working now. Mu -Original Message- From: Chris Wright [mailto:chr...@sous-sol.org] Sent: Friday, May 28, 2010 12:33 PM To: Mu Lin Cc: kvm@vger.kernel.org Subject: Re: device passthrough * Mu Lin (m...@juniper.net) wrote: Is there any method to directly assign a device to Guest OS without VT-d? Assuming you mean a PCI device, no, there isn't. Without an IOMMU[1] you can't directly assign a PCI device to a guest (nor is it safe). There have been patches floating around to allow this, but they don't maintain secure isolation. thanks, -chris [1] VT-d is an Intel chipset feature, so you could certainly do it on an AMD platform that has an AMD IOMMU. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: device passthrough
* Mu Lin (m...@juniper.net) wrote: Do you know where is the patch, I just need something quick and dirty for now, my shining new board does have VT-d but the BIOS is not ready yet, I want to have something working now. Sorry, I don't have a handy pointer. You can search for either pv dma changes (paravirtualizing the guest's request for dma addrs so that it gets host physical addr to program card for dma) or reserved-ram for pci-passthrough (1:1 mapping of guest to host physical memory). I don't recall a recent attempt to bring them forward, so expect anything you find to be quite stale. thanks, -chris -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Kvm device passthrough
Hi, Folks: Could you provide pointer to the kvm device passthrough howto/FAQ? I have two questions: 1. my host os, the Linux doesn't have the native device driver for some home grown pci devices, the driver is in the guest os, does device passthrough work in this case? Assuming I have VT-d. 2. How about i2c devices? Thanks Mu-- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: pci device passthrough: Failed to assign irq
I'm trying to pass a pcie-device into a kvm guest. qemu-kvm is started by libvirt with this command: [...] Failed to assign irq for hostdev0: Operation not permitted Perhaps you are assigning a device that shares an IRQ with another device? Found out what was causing the issue: Current libvirt seems to drop CAP_SYS_RAWIO before forking qemu-kvm. When I patch libvirt to not drop the capabilities, everything works as expected. So no problem in kvm, sorry for the noise. Kind regards, Gerd -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: MSI-X not enabled for ixgbe device-passthrough
Chris Wright wrote: * Hannes Reinecke (h...@suse.de) wrote: Chris Wright wrote: * Hannes Reinecke (h...@suse.de) wrote: Hi all, I'm trying to setup a system with device-passthrough for an ixgbe NIC. The device itself seems to work, but it isn't using MSI-X. So some more advanced features like DCB offloading etc won't work. Please send the relevant dmesg from the guest when it initializes the device. BTW, more typical case for that NIC is to assign the VF to the guest, not the whole PF. Yes, I know. But my kernel I'm testing with doesn't have one for ixgbe. Ah, you mean it's an older (heh, ok, older actually means not brand new upstream) kernel, w/out the recent sr-iov additions, fair enough. So I tested that one. And KVM really should enable MSI-X here, VFs notwithstanding. Yeah, although it's not just KVM involved, it's very much driven by the guest too. The guest will see (or at least should see) the MSI-X capability, and decide based on the number of queues whether to enable MSI-X (completely driver dependent here). Did you have a chance to boot the guest again, and send the lspci -vvv from the guest POV? You should see two PCI capabilities (MSI and 0x40 and MSI-X at 0x50). Actually, I have one of these devices, let me give it a quick test... working fine here. Here's some relevant information: Host: $ sudo lspci -v -s 04:00.0 04:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at f808 (64-bit, prefetchable) [size=512K] I/O ports at d020 [size=32] Memory at f8104000 (64-bit, prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=64 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-3f-a1-b0 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) $ grep kvm /proc/interrupts 95:289 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm_assigned_msix_device 96: 32 0 0 0292 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm_assigned_msix_device 97: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm_assigned_msix_device Guest side: $ sudo lspci -vvv -s 00:04.0 00:04.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast TAbort- TAbort- MAbort- SERR- PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 11 Region 0: Memory at f208 (32-bit, non-prefetchable) Region 2: I/O ports at c200 Region 4: Memory at f210 (32-bit, non-prefetchable) Capabilities: [40] MSI: Enable- Count=1/1 Maskable- 64bit- Address: Data: Capabilities: [50] MSI-X: Enable+ Count=64 Masked- Vector table: BAR=4 offset= PBA: BAR=4 offset=2000 $ grep eth1 /proc/interrupts 177:387 PCI-MSI-X eth1-rx-0 185:421 PCI-MSI-X eth1-tx-0 193: 2 PCI-MSI-X eth1:lsc $ dmesg | grep -e 00:04.0 -e ixgbe ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 2.0.8-k2 ixgbe: Copyright (c) 1999-2009 Intel Corporation. ACPI: PCI Interrupt :00:04.0[A] - Link [LNKD] - GSI 10 (level, high) - IRQ 10 PCI: Setting latency timer of device :00:04.0 to 64 ixgbe: :00:04.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 ixgbe :00:04.0: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:3f:a1:b0 ixgbe :00:04.0: MAC: 2, PHY: 11, SFP+: 5, PBA No: e66562-003 ixgbe :00:04.0: Intel(R) 10 Gigabit Network Connection ixgbe: eth1 NIC Link is Up 10 Gbps, Flow Control: None Ah. So I'll have to shout at Alex Graf. No problems there :-) Thanks for your help. Cheers, Hannes -- Dr. Hannes Reinecke zSeries Storage h...@suse.de +49 911 74053 688 SUSE LINUX
Re: MSI-X not enabled for ixgbe device-passthrough
* Hannes Reinecke (h...@suse.de) wrote: Ah. So I'll have to shout at Alex Graf. No problems there :-) I like that, when in doubt, shout at Alex ;-) thanks, -chris -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: MSI-X not enabled for ixgbe device-passthrough
Am 29.03.2010 um 18:46 schrieb Chris Wright chr...@sous-sol.org: * Hannes Reinecke (h...@suse.de) wrote: Ah. So I'll have to shout at Alex Graf. No problems there :-) I like that, when in doubt, shout at Alex ;-) Yep, whenever in doubt, just shout at me :). Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: MSI-X not enabled for ixgbe device-passthrough
Chris Wright wrote: * Hannes Reinecke (h...@suse.de) wrote: Hi all, I'm trying to setup a system with device-passthrough for an ixgbe NIC. The device itself seems to work, but it isn't using MSI-X. So some more advanced features like DCB offloading etc won't work. Please send the relevant dmesg from the guest when it initializes the device. BTW, more typical case for that NIC is to assign the VF to the guest, not the whole PF. Yes, I know. But my kernel I'm testing with doesn't have one for ixgbe. So I tested that one. And KVM really should enable MSI-X here, VFs notwithstanding. Cheers, Hannes -- Dr. Hannes Reinecke zSeries Storage h...@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: MSI-X not enabled for ixgbe device-passthrough
* Hannes Reinecke (h...@suse.de) wrote: Chris Wright wrote: * Hannes Reinecke (h...@suse.de) wrote: Hi all, I'm trying to setup a system with device-passthrough for an ixgbe NIC. The device itself seems to work, but it isn't using MSI-X. So some more advanced features like DCB offloading etc won't work. Please send the relevant dmesg from the guest when it initializes the device. BTW, more typical case for that NIC is to assign the VF to the guest, not the whole PF. Yes, I know. But my kernel I'm testing with doesn't have one for ixgbe. Ah, you mean it's an older (heh, ok, older actually means not brand new upstream) kernel, w/out the recent sr-iov additions, fair enough. So I tested that one. And KVM really should enable MSI-X here, VFs notwithstanding. Yeah, although it's not just KVM involved, it's very much driven by the guest too. The guest will see (or at least should see) the MSI-X capability, and decide based on the number of queues whether to enable MSI-X (completely driver dependent here). Did you have a chance to boot the guest again, and send the lspci -vvv from the guest POV? You should see two PCI capabilities (MSI and 0x40 and MSI-X at 0x50). Actually, I have one of these devices, let me give it a quick test... working fine here. Here's some relevant information: Host: $ sudo lspci -v -s 04:00.0 04:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at f808 (64-bit, prefetchable) [size=512K] I/O ports at d020 [size=32] Memory at f8104000 (64-bit, prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=64 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-3f-a1-b0 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) $ grep kvm /proc/interrupts 95:289 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm_assigned_msix_device 96: 32 0 0 0292 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm_assigned_msix_device 97: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm_assigned_msix_device Guest side: $ sudo lspci -vvv -s 00:04.0 00:04.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast TAbort- TAbort- MAbort- SERR- PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 11 Region 0: Memory at f208 (32-bit, non-prefetchable) Region 2: I/O ports at c200 Region 4: Memory at f210 (32-bit, non-prefetchable) Capabilities: [40] MSI: Enable- Count=1/1 Maskable- 64bit- Address: Data: Capabilities: [50] MSI-X: Enable+ Count=64 Masked- Vector table: BAR=4 offset= PBA: BAR=4 offset=2000 $ grep eth1 /proc/interrupts 177:387 PCI-MSI-X eth1-rx-0 185:421 PCI-MSI-X eth1-tx-0 193: 2 PCI-MSI-X eth1:lsc $ dmesg | grep -e 00:04.0 -e ixgbe ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 2.0.8-k2 ixgbe: Copyright (c) 1999-2009 Intel Corporation. ACPI: PCI Interrupt :00:04.0[A] - Link [LNKD] - GSI 10 (level, high) - IRQ 10 PCI: Setting latency timer of device :00:04.0 to 64 ixgbe: :00:04.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 ixgbe :00:04.0: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:3f:a1:b0 ixgbe :00:04.0: MAC: 2, PHY: 11, SFP+: 5, PBA No: e66562-003 ixgbe :00:04.0: Intel(R) 10 Gigabit Network Connection ixgbe: eth1 NIC Link is Up 10 Gbps, Flow Control: None -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: MSI-X not enabled for ixgbe device-passthrough
Sheng Yang wrote: On Wednesday 24 March 2010 23:54:15 Hannes Reinecke wrote: Hi all, I'm trying to setup a system with device-passthrough for an ixgbe NIC. The device itself seems to work, but it isn't using MSI-X. So some more advanced features like DCB offloading etc won't work. How about lspci result in the guest? I have to boot a different system; will provide it shortly. And some guest dmesg would also help. Ok: device: 07:00.0: driver=pci-assign host=07:00.0 device: 07:00.1: driver=pci-assign host=07:00.1 [0.00] Initializing cgroup subsys cpuset [0.00] Initializing cgroup subsys cpu [0.00] Linux version 2.6.32.9-0.5-default (ge...@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP 2010-03-15 12:22:00 +0100 [0.00] Command line: console=ttyS0 [0.00] KERNEL supported cpus: [0.00] Intel GenuineIntel [0.00] AMD AuthenticAMD [0.00] Centaur CentaurHauls [0.00] BIOS-provided physical RAM map: [0.00] BIOS-e820: - 0009f400 (usable) [0.00] BIOS-e820: 0009f400 - 000a (reserved) [0.00] BIOS-e820: 000f - 0010 (reserved) [0.00] BIOS-e820: 0010 - 07ffd000 (usable) [0.00] BIOS-e820: 07ffd000 - 0800 (reserved) [0.00] BIOS-e820: fffbc000 - 0001 (reserved) [0.00] DMI 2.4 present. [0.00] last_pfn = 0x7ffd max_arch_pfn = 0x4 [0.00] PAT not supported by CPU. [0.00] Scanning 1 areas for low memory corruption [0.00] modified physical RAM map: [0.00] modified: - 1000 (usable) [0.00] modified: 1000 - 6000 (reserved) [0.00] modified: 6000 - 0009f400 (usable) [0.00] modified: 0009f400 - 000a (reserved) [0.00] modified: 000f - 0010 (reserved) [0.00] modified: 0010 - 07ffd000 (usable) [0.00] modified: 07ffd000 - 0800 (reserved) [0.00] modified: fffbc000 - 0001 (reserved) [0.00] init_memory_mapping: -07ffd000 [0.00] RAMDISK: 07849000 - 07feff7e [0.00] ACPI: RSDP 000f8950 00014 (v00 BOCHS ) [0.00] ACPI: RSDT 07ffde30 00034 (v01 BOCHS BXPCRSDT 0001 BXPC 0001) [0.00] ACPI: FACP 07fffe70 00074 (v01 BOCHS BXPCFACP 0001 BXPC 0001) [0.00] ACPI: DSDT 07ffdfd0 01E22 (v01 BXPC BXDSDT 0001 INTL 20090123) [0.00] ACPI: FACS 07fffe00 00040 [0.00] ACPI: SSDT 07ffdf90 00037 (v01 BOCHS BXPCSSDT 0001 BXPC 0001) [0.00] ACPI: APIC 07ffdeb0 00072 (v01 BOCHS BXPCAPIC 0001 BXPC 0001) [0.00] ACPI: HPET 07ffde70 00038 (v01 BOCHS BXPCHPET 0001 BXPC 0001) [0.00] No NUMA configuration found [0.00] Faking a node at -07ffd000 [0.00] Bootmem setup node 0 -07ffd000 [0.00] NODE_DATA [9000 - 0003cfff] [0.00] bootmap [0003d000 - 0003dfff] pages 1 [0.00] (7 early reservations) == bootmem [00 - 0007ffd000] [0.00] #0 [00 - 001000] BIOS data page == [00 - 001000] [0.00] #1 [006000 - 008000] TRAMPOLINE == [006000 - 008000] [0.00] #2 [000100 - 0001cd6778]TEXT DATA BSS == [000100 - 0001cd6778] [0.00] #3 [0007849000 - 0007feff7e] RAMDISK == [0007849000 - 0007feff7e] [0.00] #4 [09f400 - 10]BIOS reserved == [09f400 - 10] [0.00] #5 [0001cd7000 - 0001cd7049] BRK == [0001cd7000 - 0001cd7049] [0.00] #6 [008000 - 009000] PGTABLE == [008000 - 009000] [0.00] found SMP MP-table at [880f89a0] f89a0 [0.00] kvm-clock: cpu 0, msr 0:1940501, boot clock [0.00] Zone PFN ranges: [0.00] DMA 0x - 0x1000 [0.00] DMA320x1000 - 0x0010 [0.00] Normal 0x0010 - 0x0010 [0.00] Movable zone start PFN for each node [0.00] early_node_map[3] active PFN ranges [0.00] 0: 0x - 0x0001 [0.00] 0: 0x0006 - 0x009f [0.00] 0: 0x0100 - 0x7ffd [0.00] ACPI: PM-Timer IO Port: 0xb008 [0.00] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [0.00] ACPI: IOAPIC (id[0x01] address[0xfec0] gsi_base[0]) [0.00
Re: MSI-X not enabled for ixgbe device-passthrough
* Hannes Reinecke (h...@suse.de) wrote: Hi all, I'm trying to setup a system with device-passthrough for an ixgbe NIC. The device itself seems to work, but it isn't using MSI-X. So some more advanced features like DCB offloading etc won't work. Please send the relevant dmesg from the guest when it initializes the device. BTW, more typical case for that NIC is to assign the VF to the guest, not the whole PF. thanks, -chris -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI device passthrough / memory mapping issue
On 03/19/2010 12:46 AM, Fede wrote: I'm currently working to enable vga passthrough in kvm. assigned_dev_iomem_map: e_phys=1000 r_virt=0x7fa64af0e000 type=0 len=0200 region_num=3 kvm_register_phys_mem:580 memory: gpa: 1000, size: 200, uaddr: 7fa64af0e000, slot: 7,flags: 0 create_userspace_phys_mem: File exists assigned_dev_iomem_map: Error: create new mapping failed This worked a month ago. But after some git updates there's a problem. When the real device regions are mapped from real virtual memory to guest physical addresses in kvm, it overlaps region 3 with the guest physical memory assigned to kernel space (0x1000 to 0x1020) I've been trying to look for the answer to this question: Why is gpa 0x1000 chosen and not any other free memory space? The BIOS generally assigns addresses, then the OS updates them if it wants to. Is the crash before or after the OS has started loading? If before, suggest you add some printfs to seabios to explain its decisions. It seems that this addresses are being chosen here (for example, for region 3): assigned_dev_pci_read_config: (4.0): address=001c val=0x len=4 assigned_dev_pci_write_config: (4.0): address=001c val=0x len=4 assigned_dev_pci_read_config: (4.0): address=001c val=0xfe00 len=4 The above sequence is how the bios determines the BAR size. (0x200 in this case). assigned_dev_pci_write_config: (4.0): address=001c val=0x len=4 Now it clears the mess from the previous step. assigned_dev_pci_read_config: (4.0): address=001c val=0x len=4 assigned_dev_pci_write_config: (4.0): address=001c val=0x1000 len=4 Here it assigns a new address. It's clearly wrong. A log in seabios will explain this. What does this mean? Why PCI BARs are being written and read? Why the values that are written differs from the ones that are read after? BARs are not RAM cells. the least significant address bits are hardwired to zero, that's how the BIOS detects the BAR size (and required alignment). The last write is the gpa that is used. Is this a bug? I can't find the source of this gpa addresses. I need to change them. Looks like a bug in seabios. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
MSI-X not enabled for ixgbe device-passthrough
Hi all, I'm trying to setup a system with device-passthrough for an ixgbe NIC. The device itself seems to work, but it isn't using MSI-X. So some more advanced features like DCB offloading etc won't work. lspci output of the device: 07:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 Flags: bus master, fast devsel, latency 0, IRQ 24 Memory at f5c8 (64-bit, prefetchable) [size=512K] I/O ports at 5000 [size=32] Memory at f5c7 (64-bit, prefetchable) [size=16K] [virtual] Expansion ROM at e710 [disabled] [size=512K] Capabilities: [40] Power Management version 3 Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ Count=1/1 Enable- Capabilities: [70] MSI-X: Enable+ Mask- TabSize=64 Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSVoil- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSVoil- UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSVoil- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140] Device Serial Number 40-9e-3c-ff-ff-21-1b-00 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 1 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [160] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ IOVSta: Migration- Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00 VF offset: 128, stride: 2, Device ID: 10ed Supported Page Size: 0553, System Page Size: 0001 VF Migration: offset: , BIR: 1 Kernel driver in use: ixgbe Kernel modules: ixgbe please let me know if you need more information. Cheers, Hannes -- Dr. Hannes Reinecke zSeries Storage h...@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: MSI-X not enabled for ixgbe device-passthrough
On Wednesday 24 March 2010 23:54:15 Hannes Reinecke wrote: Hi all, I'm trying to setup a system with device-passthrough for an ixgbe NIC. The device itself seems to work, but it isn't using MSI-X. So some more advanced features like DCB offloading etc won't work. How about lspci result in the guest? And some guest dmesg would also help. -- regards Yang, Sheng lspci output of the device: 07:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 Flags: bus master, fast devsel, latency 0, IRQ 24 Memory at f5c8 (64-bit, prefetchable) [size=512K] I/O ports at 5000 [size=32] Memory at f5c7 (64-bit, prefetchable) [size=16K] [virtual] Expansion ROM at e710 [disabled] [size=512K] Capabilities: [40] Power Management version 3 Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ Count=1/1 Enable- Capabilities: [70] MSI-X: Enable+ Mask- TabSize=64 Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSVoil- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSVoil- UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSVoil- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140] Device Serial Number 40-9e-3c-ff-ff-21-1b-00 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 1 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [160] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ IOVSta: Migration- Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00 VF offset: 128, stride: 2, Device ID: 10ed Supported Page Size: 0553, System Page Size: 0001 VF Migration: offset: , BIR: 1 Kernel driver in use: ixgbe Kernel modules: ixgbe please let me know if you need more information. Cheers, Hannes -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
PCI device passthrough / memory mapping issue
I'm currently working to enable vga passthrough in kvm. This is a current git qemu-kvm run with some debug printings: slash...@fidel ~/kvm/vms $ qemu-system-x86_64 -hda i386ubuntu904.img -boot c -m 1024 -net nic -net user,hostfwd=tcp::-:22 -pcidevice host=01:00.0 vm_register_phys_mem:580 memory: gpa: 0, size: a, uaddr: 7fa65f077000, slot: 0, flags: 0 kvm_register_phys_mem:580 memory: gpa: 10, size: 3ff0, uaddr: 7fa65f177000, slot: 1, flags: 0 kvm_register_phys_mem:580 memory: gpa: e, size: 2, uaddr: 7fa6a2fc, slot: 2, flags: 0 kvm_register_phys_mem:580 memory: gpa: c, size: 2, uaddr: 7fa6a2f7d000, slot: 3, flags: 0 kvm_register_phys_mem:580 memory: gpa: fffe, size: 2, uaddr: 7fa6a2fc, slot: 4, flags: 0 device: 01:00.0: driver=pci-assign host=01:00.0 get_real_device: region 0 size 16777216 start 0xe200 type 512 resource_fd 13 get_real_device: region 1 size 268435456 start 0xd000 type 4608 resource_fd 14 get_real_device: region 3 size 33554432 start 0xe000 type 512 resource_fd 15 get_real_device: region 5 size 128 start 0xe000 type 256 resource_fd 0 get_real_device: region 6 size 524288 start 0xe300 type 4608 resource_fd 0 assigned_dev_register_regions: MAP PHYSICAL MEMORY e_physbase=0xe200 assigned_dev_register_regions: NON_PCI_ROM_SLOT virt_base=0x7fa65cf0e000 assigned_dev_register_regions: MAP PHYSICAL MEMORY e_physbase=0xd000 assigned_dev_register_regions: NON_PCI_ROM_SLOT virt_base=0x7fa64cf0e000 assigned_dev_register_regions: MAP PHYSICAL MEMORY e_physbase=0xe000 assigned_dev_register_regions: NON_PCI_ROM_SLOT virt_base=0x7fa64af0e000 assigned_dev_register_regions: MAP PHYSICAL MEMORY e_physbase=0xe300 assigned_dev_register_regions: PCI_ROM_SLOT virt_base=0x7fa64ae8e000 assigned_dev_pci_read_config: (4.0): address= val=0x10de len=2 assigned_dev_pci_read_config: (4.0): address=000e val=0x len=1 assigned_dev_pci_read_config: (4.0): address= val=0x10de len=2 assigned_dev_pci_read_config: (4.0): address=0002 val=0x0622 len=2 assigned_dev_pci_read_config: (4.0): address=000e val=0x len=1 kvm_register_phys_mem:580 memory: gpa: f000, size: 100, uaddr: 7fa65e075000, slot: 5, flags: 0 kvm_dirty_pages_log_enable_slot:293 start f000 len 100 kvm_dirty_pages_log_change:266 slot 5 start f000 len 100 flags 1 assigned_dev_pci_read_config: (4.0): address= val=0x10de len=2 assigned_dev_pci_read_config: (4.0): address=000e val=0x len=1 assigned_dev_pci_read_config: (4.0): address=000a val=0x0300 len=2 assigned_dev_pci_read_config: (4.0): address= val=0x10de len=2 assigned_dev_pci_read_config: (4.0): address=0002 val=0x0622 len=2 assigned_dev_pci_read_config: (4.0): address=0010 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0010 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0010 val=0xff00 len=4 assigned_dev_pci_write_config: (4.0): address=0010 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0010 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0010 val=0xf300 len=4 assigned_dev_pci_read_config: (4.0): address=0014 val=0x0008 len=4 assigned_dev_pci_write_config: (4.0): address=0014 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0014 val=0xf008 len=4 assigned_dev_pci_write_config: (4.0): address=0014 val=0x0008 len=4 assigned_dev_pci_read_config: (4.0): address=0014 val=0x0008 len=4 assigned_dev_pci_write_config: (4.0): address=0014 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0018 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0018 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0018 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0018 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=001c val=0x len=4 assigned_dev_pci_write_config: (4.0): address=001c val=0x len=4 assigned_dev_pci_read_config: (4.0): address=001c val=0xfe00 len=4 assigned_dev_pci_write_config: (4.0): address=001c val=0x len=4 assigned_dev_pci_read_config: (4.0): address=001c val=0x len=4 assigned_dev_pci_write_config: (4.0): address=001c val=0x1000 len=4 assigned_dev_pci_read_config: (4.0): address=0020 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0020 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0020 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0020 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0024 val=0x0001 len=4 assigned_dev_pci_write_config: (4.0): address=0024 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0024 val=0xff81 len=4 assigned_dev_pci_write_config: (4.0): address=0024 val=0x0001 len=4 assigned_dev_pci_read_config: (4.0): address=0024 val=0x0001 len=4 assigned_dev_pci_write_config: (4.0):
PCI device passthrough for FreeDOS guests
Hi, I want to trace the PCI device accesses (config space, I/O ports, memory BARs) of a DOS application (EEPROM flasher) and it seems KVM can help with this. The current plan is to install KVM, use FreeDOS as a guest, activate passthrough for the PCI device I'm interested in and log the accesses somehow. Possible problems I'd have to solve: - According to http://www.linux-kvm.org/wiki/images/d/d0/KvmForum2008$kdf2008_14.pdf PCI device passthrough needs a CPU with VT-d/IOMMU support. - http://www.linux-kvm.org/page/TODO says passthrough won't work for conventional PCI devices (I assume a PCI graphics card is such a device). - http://www.linux-kvm.org/page/TODO says passthrough won't work for devices without Function Level Reset (FLR). - http://www.linux-kvm.org/page/TODO says VT support for real mode is not good, so FreeDOS might be a problem. - The KVM wiki does not mention how to get PCI passthrough working. - No idea if logging of PCI accesses is possible with KVM or if I have to use helper tools for that. Any insights/hints are appreciated. If KVM is the wrong tool for my task, please point me in the right direction. Please CC me as I'm not subscribed to the list. Thanks. Regards, Carl-Daniel P.S. The linux-kvm.org wiki returns zero results if you search the wiki text for PCI, but a Google search finds quite a few pages mentioning PCI. Is there a bug in the seach function? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: PCI device passthrough for FreeDOS guests
Carl-Daniel Hailfinger wrote: Hi, I want to trace the PCI device accesses (config space, I/O ports, memory BARs) of a DOS application (EEPROM flasher) and it seems KVM can help with this. The current plan is to install KVM, use FreeDOS as a guest, activate passthrough for the PCI device I'm interested in and log the accesses somehow. Possible problems I'd have to solve: - According to http://www.linux-kvm.org/wiki/images/d/d0/KvmForum2008$kdf2008_14.pdf PCI device passthrough needs a CPU with VT-d/IOMMU support. - http://www.linux-kvm.org/page/TODO says passthrough won't work for conventional PCI devices (I assume a PCI graphics card is such a It supports conventional PCI devices passgthrough now. device). - http://www.linux-kvm.org/page/TODO says passthrough won't work for devices without Function Level Reset (FLR). FLR is not must for passthrough. If the device status is correct, passthrough can work. FLR is used to keep device in correct status before and after passthrough. Now more device reset methods are supported, such as secondary bus reset and D0hot-D3 transition. So even the device won't support PCIe FLR, it still can be reset. It's not a problem now. The VT-d TODO is obsolete. I will update it. - http://www.linux-kvm.org/page/TODO says VT support for real mode is not good, so FreeDOS might be a problem. - The KVM wiki does not mention how to get PCI passthrough working. There is VT-d Howto: http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM. pls refer to it to passthrough pci. - No idea if logging of PCI accesses is possible with KVM or if I have to use helper tools for that. You can log PCI accesses. Pls refer to qemu-kvm/hw/device-assignment.c. Regards, Weidong Any insights/hints are appreciated. If KVM is the wrong tool for my task, please point me in the right direction. Please CC me as I'm not subscribed to the list. Thanks. Regards, Carl-Daniel P.S. The linux-kvm.org wiki returns zero results if you search the wiki text for PCI, but a Google search finds quite a few pages mentioning PCI. Is there a bug in the seach function? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Wed, 2009-05-06 at 11:51 +0800, Sheng Yang wrote: On Wednesday 06 May 2009 01:45:47 Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 04:28 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. Greetings Yu, Sheng and Co, So I have been making progress this morning.. So far, I have hooked up a LSI mpt-function PCIe SAS adapter into the KVM guest with a Sandisk SATA SSD 32 GB drive. It is using MSI interrupts (not MSI-X) and I am able to push ~70 MB/sec from a 2nd Linux/iSCSI Initiator machine (running Open-iSCSI) with the 1500 byte MTUs on e1000e ports from within the KVM guest. Is MSI-X can't be enabled or the device only have MSI capability? Just curious... AFAICT MSI-X is not supported on the mpt-fusion SAS device when running on the KVM host.. The interesting thing is that I am having to use IBLOCK export (using using submit_bio(), and complete emulation of SCSI control path) for SATA SSD in order to get I/O running stable Using the pSCSI export I am getting immediate exceptions from scsi_execute_async() in the v2.6.29.2 KVM guest.. Didn't see exception in the log below... (And buried with iscsi log I can't understand. Looking forward for the help from others...) Any thing notable show in the host side? I think the target to get pSCSI work well now? Yeah, the log attached was using the SATA SSD with the IBLOCK export (the one that worked..). The exception that I was seeing from pSCSI export from the same SATA SSD from within KVM Guest running v2.6.29.2 involved a non-zero return from drivers/scsi/scsi_lib.c:scsi_execute_async() for basically all bulk struct scatterlist I/O (the control path seemed OK however). Unfortuately, scsi_execute_async() was (it has been removed for v2.6.30) really bad at returning any useful value and hardcodes a return of 'DRIVER_ERROR 24' in the error path, so I was unable to determine anything useful for what was actually causing it to fail during my initial testing.. I will keep poking at it and see if I can find anything interesting.. BTW: Maybe you can try the patch from Marcelo titled [patch 0/4] use smp_send_reschedule in vcpu_kick / assigned dev host intx race fix. Thanks for the pointer, I will have a look. Many thanks for your most valuable of time, --nab -- regards Yang, Sheng Using a 2nd SAS disk I am able to use target_core_mod/pSCSI export and push badblocks and LTP disktest traffic however.. Here is a bit about the the setup looks, *) Linux/iSCSI Initiator node accessing KVM Guest LIO-Target v3.0 storage: subjekt:~# lsscsi [6:0:0:0]diskATA ST3250820AS 3.AA /dev/sda [10:0:0:0] cd/dvd PIONEER DVD-ROM DVD-305 1.06 /dev/scd1 [18:0:0:0] cd/dvd TOSHIBA DVD/HD X807616
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. --nab One issue I did notice while using the pci-stub method of device-assignment with same e1000 port (02:00.0) was while using an iSCSI Initiator (Open-iSCSI) on the KVM Host machine and doing sustained traffic into the LIO-Target KVM Guest on the same local KVM host to max out traffic between the other onboard e1000e port (03.00.0), I see the following: pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 assign device: host bdf = 2:0:0 pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X scsi4 : iSCSI Initiator over TCP/IP scsi 4:0:0:0: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:0: Attached scsi generic sg1 type 0 scsi 4:0:0:1: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:1: Attached scsi generic sg2 type 0 sd 4:0:0:0: [sdb] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:1: [sdc] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:0: [sdb] Write Protect is off sd 4:0:0:0: [sdb] Mode Sense: 2f 00 00 00 sd 4:0:0:1: [sdc] Write Protect is off sd 4:0:0:1: [sdc] Mode Sense: 2f 00 00 00 sd 4:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sd 4:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sdb:6 sdc: unknown partition table sd 4:0:0:0: [sdb] Attached SCSI disk unknown partition table sd 4:0:0:1: [sdc] Attached SCSI disk [ cut here ] WARNING: at kernel/irq/manage.c:260 enable_irq+0x36/0x50() Hardware name: empty Unbalanced enable for IRQ 59 Modules linked in: ipt_REJECT xt_tcpudp bridge stp sunrpc iptable_filter ip_tables xt_state nf_conntrack ip6table_filter ip6_tables x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table ext3 jbd loop dm_multipath scsi_dh kvm_intel kvm uinput i2c_i801 firewire_ohci joydev firewire_core sg i2c_core 8250_pnp crc_itu_t e1000e 8250 serial_core rtc_cmos pcspkr serio_raw rtc_core rtc_lib button sd_mod dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod uhci_hcd ohci_hcd ehci_hcd ata_piix libata scsi_mod [last unloaded: microcode] Pid: 51, comm: events/0 Tainted: GW 2.6.30-rc3 #11 Call Trace: [80235fee] ? warn_slowpath+0xcb/0xe8 [80253a7c] ? generic_exec_single+0x6a/0x88 [8022acec] ? update_curr+0x67/0xeb [a0198748] ? vcpu_kick_intr+0x0/0x1 [kvm] [8020a5d8] ? __switch_to+0xb6/0x274 [8022b70a] ? __dequeue_entity+0x1b/0x2f [a01ac7e4] ?
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Tue, 2009-05-05 at 04:28 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. Greetings Yu, Sheng and Co, So I have been making progress this morning.. So far, I have hooked up a LSI mpt-function PCIe SAS adapter into the KVM guest with a Sandisk SATA SSD 32 GB drive. It is using MSI interrupts (not MSI-X) and I am able to push ~70 MB/sec from a 2nd Linux/iSCSI Initiator machine (running Open-iSCSI) with the 1500 byte MTUs on e1000e ports from within the KVM guest. The interesting thing is that I am having to use IBLOCK export (using using submit_bio(), and complete emulation of SCSI control path) for SATA SSD in order to get I/O running stable Using the pSCSI export I am getting immediate exceptions from scsi_execute_async() in the v2.6.29.2 KVM guest.. Using a 2nd SAS disk I am able to use target_core_mod/pSCSI export and push badblocks and LTP disktest traffic however.. Here is a bit about the the setup looks, *) Linux/iSCSI Initiator node accessing KVM Guest LIO-Target v3.0 storage: subjekt:~# lsscsi [6:0:0:0]diskATA ST3250820AS 3.AA /dev/sda [10:0:0:0] cd/dvd PIONEER DVD-ROM DVD-305 1.06 /dev/scd1 [18:0:0:0] cd/dvd TOSHIBA DVD/HD X807616 MC08 /dev/scd2 [32:0:0:0] diskLIO-ORG RAMDISK-DR 3.0 /dev/sdb [32:0:0:1] diskLIO-ORG RAMDISK-DR 3.0 /dev/sdc [32:0:0:2] diskLIO-ORG FILEIO 3.0 /dev/sdd [32:0:0:3] diskLIO-ORG IBLOCK 3.0 /dev/sde subjekt:~# sg_inq -i /dev/sde VPD INQUIRY: Device Identification page Designation descriptor number 1, descriptor length: 20 id_type: NAA, code_set: Binary associated with the addressed logical unit NAA 6, IEEE Company_id: 0x1405 Vendor Specific Identifier: 0xa97e4ce21 Vendor Specific Identifier Extension: 0xc0711de829b000c2 [0x6001405a97e4ce21c0711de829b000c2] Designation descriptor number 2, descriptor length: 52 id_type: T10 vendor identification, code_set: ASCII associated with the addressed logical unit vendor id: LIO-ORG vendor specific: IBLOCK:a97e4ce21c0711de829b000c2943d57b Designation descriptor number 3, descriptor length: 8 transport: Internet SCSI (iSCSI) id_type: Relative target port, code_set: Binary associated with the target port Relative target port: 0x1 Designation descriptor number 4, descriptor length: 8 transport: Internet SCSI (iSCSI) id_type: Target port group, code_set: Binary associated with the target port Target port group: 0x0 Designation descriptor number 5, descriptor length: 8 id_type: Logical unit group, code_set: Binary associated with the addressed logical unit Logical unit group: 0x0 Designation descriptor number 6, descriptor length: 80 transport: Internet SCSI (iSCSI) id_type: SCSI name string, code_set: UTF-8
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Tuesday 05 May 2009 18:43:46 Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. One issue I did notice while using the pci-stub method of device-assignment with same e1000 port (02:00.0) was while using an iSCSI Initiator (Open-iSCSI) on the KVM Host machine and doing sustained traffic into the LIO-Target KVM Guest on the same local KVM host to max out traffic between the other onboard e1000e port (03.00.0), I see the following: pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 assign device: host bdf = 2:0:0 pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X scsi4 : iSCSI Initiator over TCP/IP scsi 4:0:0:0: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:0: Attached scsi generic sg1 type 0 scsi 4:0:0:1: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:1: Attached scsi generic sg2 type 0 sd 4:0:0:0: [sdb] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:1: [sdc] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:0: [sdb] Write Protect is off sd 4:0:0:0: [sdb] Mode Sense: 2f 00 00 00 sd 4:0:0:1: [sdc] Write Protect is off sd 4:0:0:1: [sdc] Mode Sense: 2f 00 00 00 sd 4:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sd 4:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sdb:6 sdc: unknown partition table sd 4:0:0:0: [sdb] Attached SCSI disk unknown partition table sd 4:0:0:1: [sdc] Attached SCSI disk [ cut here ] WARNING: at kernel/irq/manage.c:260 enable_irq+0x36/0x50() Hardware name: empty Unbalanced enable for IRQ 59 Modules linked in: ipt_REJECT xt_tcpudp bridge stp sunrpc iptable_filter ip_tables xt_state nf_conntrack ip6table_filter ip6_tables x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table ext3 jbd loop dm_multipath scsi_dh kvm_intel kvm uinput i2c_i801 firewire_ohci joydev firewire_core sg i2c_core 8250_pnp crc_itu_t e1000e 8250 serial_core rtc_cmos pcspkr serio_raw rtc_core rtc_lib button sd_mod dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod uhci_hcd ohci_hcd ehci_hcd ata_piix libata scsi_mod [last unloaded: microcode] Pid: 51, comm: events/0 Tainted: GW 2.6.30-rc3 #11 Call Trace: [80235fee] ? warn_slowpath+0xcb/0xe8 [80253a7c] ? generic_exec_single+0x6a/0x88 [8022acec] ? update_curr+0x67/0xeb [a0198748] ? vcpu_kick_intr+0x0/0x1 [kvm] [8020a5d8] ? __switch_to+0xb6/0x274 [8022b70a] ? __dequeue_entity+0x1b/0x2f [a01ac7e4] ? kvm_irq_delivery_to_apic+0xb3/0xf7 [kvm] [a01aa4d4] ? __apic_accept_irq+0x15a/0x173 [kvm] [a01ac883] ? kvm_set_msi+0x5b/0x60 [kvm] [80266d97] ? enable_irq+0x36/0x50 [a0195ab5] ? kvm_assigned_dev_interrupt_work_handler+0x6d/0xbc [kvm] [802449fa] ? worker_thread+0x182/0x223 [8024820b] ? autoremove_wake_function+0x0/0x2a [80244878] ? worker_thread+0x0/0x223 [80244878] ? worker_thread+0x0/0x223 [80247e72] ? kthread+0x54/0x7e [8020cb0a] ? child_rip+0xa/0x20 [804d0af5] ? _spin_lock+0x5/0x8 [80247e1e] ? kthread+0x0/0x7e [8020cb00] ? child_rip+0x0/0x20 ---[ end trace 3fbc2dd20bf89ef1 ]--- connection1:0: ping timeout of 5 secs expired, last rx 4295286327, last ping 4295285518, now 4295286768 connection1:0: detected conn error (1011) Attached are the v2.6.30-rc3 KVM host and v2.6.29.2 KVM guest dmesg output. When the 'Unbalanced enable for IRQ
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Tuesday 05 May 2009 19:28:15 Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW This means the fault address is too big It's got 51 bits width which is far beyond the physical address limit of current IA32e(48 bits). Don't know how you can get this... -- regards Yang, Sheng pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. --nab One issue I did notice while using the pci-stub method of device-assignment with same e1000 port (02:00.0) was while using an iSCSI Initiator (Open-iSCSI) on the KVM Host machine and doing sustained traffic into the LIO-Target KVM Guest on the same local KVM host to max out traffic between the other onboard e1000e port (03.00.0), I see the following: pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 assign device: host bdf = 2:0:0 pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X scsi4 : iSCSI Initiator over TCP/IP scsi 4:0:0:0: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:0: Attached scsi generic sg1 type 0 scsi 4:0:0:1: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:1: Attached scsi generic sg2 type 0 sd 4:0:0:0: [sdb] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:1: [sdc] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:0: [sdb] Write Protect is off sd 4:0:0:0: [sdb] Mode Sense: 2f 00 00 00 sd 4:0:0:1: [sdc] Write Protect is off sd 4:0:0:1: [sdc] Mode Sense: 2f 00 00 00 sd 4:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sd 4:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sdb:6 sdc: unknown partition table sd 4:0:0:0: [sdb] Attached SCSI disk unknown partition table sd 4:0:0:1: [sdc] Attached SCSI disk [ cut here ] WARNING: at kernel/irq/manage.c:260 enable_irq+0x36/0x50() Hardware name: empty Unbalanced enable for IRQ 59 Modules linked in: ipt_REJECT xt_tcpudp bridge stp sunrpc iptable_filter ip_tables xt_state nf_conntrack ip6table_filter ip6_tables x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table ext3 jbd loop dm_multipath scsi_dh kvm_intel kvm uinput i2c_i801 firewire_ohci joydev firewire_core sg i2c_core 8250_pnp crc_itu_t e1000e 8250 serial_core rtc_cmos pcspkr serio_raw rtc_core rtc_lib button sd_mod dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod uhci_hcd ohci_hcd ehci_hcd ata_piix libata scsi_mod [last unloaded: microcode] Pid: 51, comm: events/0 Tainted: GW 2.6.30-rc3 #11
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Wednesday 06 May 2009 01:45:47 Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 04:28 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. Greetings Yu, Sheng and Co, So I have been making progress this morning.. So far, I have hooked up a LSI mpt-function PCIe SAS adapter into the KVM guest with a Sandisk SATA SSD 32 GB drive. It is using MSI interrupts (not MSI-X) and I am able to push ~70 MB/sec from a 2nd Linux/iSCSI Initiator machine (running Open-iSCSI) with the 1500 byte MTUs on e1000e ports from within the KVM guest. Is MSI-X can't be enabled or the device only have MSI capability? Just curious... The interesting thing is that I am having to use IBLOCK export (using using submit_bio(), and complete emulation of SCSI control path) for SATA SSD in order to get I/O running stable Using the pSCSI export I am getting immediate exceptions from scsi_execute_async() in the v2.6.29.2 KVM guest.. Didn't see exception in the log below... (And buried with iscsi log I can't understand. Looking forward for the help from others...) Any thing notable show in the host side? I think the target to get pSCSI work well now? BTW: Maybe you can try the patch from Marcelo titled [patch 0/4] use smp_send_reschedule in vcpu_kick / assigned dev host intx race fix. -- regards Yang, Sheng Using a 2nd SAS disk I am able to use target_core_mod/pSCSI export and push badblocks and LTP disktest traffic however.. Here is a bit about the the setup looks, *) Linux/iSCSI Initiator node accessing KVM Guest LIO-Target v3.0 storage: subjekt:~# lsscsi [6:0:0:0]diskATA ST3250820AS 3.AA /dev/sda [10:0:0:0] cd/dvd PIONEER DVD-ROM DVD-305 1.06 /dev/scd1 [18:0:0:0] cd/dvd TOSHIBA DVD/HD X807616 MC08 /dev/scd2 [32:0:0:0] diskLIO-ORG RAMDISK-DR 3.0 /dev/sdb [32:0:0:1] diskLIO-ORG RAMDISK-DR 3.0 /dev/sdc [32:0:0:2] diskLIO-ORG FILEIO 3.0 /dev/sdd [32:0:0:3] diskLIO-ORG IBLOCK 3.0 /dev/sde subjekt:~# sg_inq -i /dev/sde VPD INQUIRY: Device Identification page Designation descriptor number 1, descriptor length: 20 id_type: NAA, code_set: Binary associated with the addressed logical unit NAA 6, IEEE Company_id: 0x1405 Vendor Specific Identifier: 0xa97e4ce21 Vendor Specific Identifier Extension: 0xc0711de829b000c2 [0x6001405a97e4ce21c0711de829b000c2] Designation descriptor number 2, descriptor length: 52 id_type: T10 vendor identification, code_set: ASCII associated with the addressed logical unit vendor id: LIO-ORG vendor specific: IBLOCK:a97e4ce21c0711de829b000c2943d57b Designation descriptor number 3, descriptor length: 8 transport: Internet SCSI (iSCSI) id_type: Relative target port, code_set:
Re: [PATCHSETS #2] KVM device passthrough support with AMD IOMMU
Joerg Roedel wrote: Hi, the two patchsets posted as reply to this email implement KVM device passthrough support for AMD IOMMU hardware. The changes to the previous posts are descibed below The first patchset is version 4 of the generic iommu api patchset which generalizes the VT-d functions exported to KVM into a common api where the AMD IOMMU code can plug into. In this version the patchset was rebased to the latest post of Han Weidong's patches. The second patchset finally implements the KVM device passthrough support in the AMD IOMMU code. Together with KVM-79 I successfully passed an 10GBit network card into an KVM guest. In this version the patchset was changed to remove any device before a protection domain is freed instead of printing a BUG. Also the patchset was rebased to the updated IOMMU-API patches. These two patchsets apply in order in top of the latest post of Han Weidong's Multiple device assignement support patches. Anybody who wants to try this out can pull the whole stuff from git://git.kernel.org/pub/scm/linux/kernel/git/joro/linux-2.6-iommu.git kvm-amd-iommu Please give these patches a good review. Ack for the kvm bits. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHSETS #2] KVM device passthrough support with AMD IOMMU
On Wed, Dec 10, 2008 at 02:22:08PM +, David Woodhouse wrote: On Wed, 2008-12-10 at 15:11 +0100, Joerg Roedel wrote: So now its open how this will be merged alltogether. We can merge it in three steps (first from Dave's tree, then Avi and at last my IOMMU updates which has to happen in that order). The other option is to merge this all with one pull-request I send to Linus. It should not conflict with your updates Avi because all these changes apply also cleanly to current linus/master. Is this acceptable for everyone? David, its important to hear your opinion on that because this single pull-request would include all VT-d updates you have queued in your tree? Is this ok for you? Yes, that's fine by me. Should I apply patches 1-14 of Weidong's patch set from Monday first? Or just let you do that? I already did it on-top of your tree because Han Weidong's patches 1-17 were rebased to your tree and my IOMMU-API patches apply on-top of his patches. Joerg -- | AMD Saxony Limited Liability Company Co. KG Operating | Wilschdorfer Landstr. 101, 01109 Dresden, Germany System| Register Court Dresden: HRA 4896 Research | General Partner authorized to represent: Center| AMD Saxony LLC (Wilmington, Delaware, US) | General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHSETS #2] KVM device passthrough support with AMD IOMMU
On Wed, 2008-12-10 at 19:42 +0100, Joerg Roedel wrote: Ok, I add it, thanks. Who is the author, Mike or you? Might as well attribute it to Mike; he spotted it. -- dwmw2 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING with device passthrough and MSI interrupts
Hi, passing a device through to a guest I sometimes get the following WARN message on the host: Dec 8 12:28:52 astat kernel: [ 248.472109] WARNING: at kernel/irq/manage.c:225 enable_irq+0x6b/0x90() Dec 8 12:28:52 astat kernel: [ 248.472114] Unbalanced enable for IRQ 488 Dec 8 12:28:52 astat kernel: [ 248.472118] Modules linked in: kvm_amd kvm af_packet radeon drm rfcomm sco bridge stp bnep l2cap bluetooth ppdev ipv6 powernow_k8 cpufreq_stats cpufreq_powersave cpufreq_conservative cpufreq_ondemand freq_table cpufreq_userspace video output rfkill input_polldev sbs sbshc battery iptable_filter ip_tables x_tables ac parport_pc lp parport psmouse serio_raw pcspkr i2c_piix4 i2c_core shpchp pci_hotplug container button evdev ext3 jbd mbcache usbhid hid sg sd_mod atiixp pata_acpi ata_generic ehci_hcd pata_atiixp ahci ohci_hcd bnx2 libata scsi_mod usbcore thermal processor fan thermal_sys fuse Dec 8 12:28:52 astat kernel: [ 248.472205] Pid: 5520, comm: qemu-system-x86 Tainted: GW 2.6.28-rc7 #3 Dec 8 12:28:52 astat kernel: [ 248.472210] Call Trace: Dec 8 12:28:52 astat kernel: [ 248.472224] [8024124d] warn_slowpath+0xcd/0x110 Dec 8 12:28:52 astat kernel: [ 248.472235] [8035cfd1] elv_insert+0x141/0x300 Dec 8 12:28:52 astat kernel: [ 248.472243] [8035d9b3] drive_stat_acct+0xc3/0xf0 Dec 8 12:28:52 astat kernel: [ 248.472250] [8035fe19] __make_request+0xf9/0x4d0 Dec 8 12:28:52 astat kernel: [ 248.472256] [802355e8] enqueue_task_fair+0x188/0x1e0 Dec 8 12:28:52 astat kernel: [ 248.472290] [a0351319] gfn_to_hva+0x9/0x70 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472318] [a0351621] kvm_read_guest_page+0x61/0x70 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472343] [a0351319] gfn_to_hva+0x9/0x70 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472367] [a0351621] kvm_read_guest_page+0x61/0x70 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472390] [a0351678] kvm_read_guest+0x48/0x80 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472415] [a0361a1c] paging64_walk_addr+0x17c/0x330 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472422] [8028983b] enable_irq+0x6b/0x90 Dec 8 12:28:52 astat kernel: [ 248.472447] [a0354359] kvm_notify_acked_irq+0x39/0x60 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472471] [a0353eb2] kvm_ioapic_update_eoi+0x42/0x90 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472495] [a036a4eb] apic_mmio_write+0x50b/0x6c0 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472505] [8021ef25] do_suspend_lowlevel+0x65/0x130 Dec 8 12:28:52 astat kernel: [ 248.472529] [a035b30b] emulator_write_emulated_onepage+0x9b/0x120 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472556] [a0362f03] x86_emulate_insn+0x423/0x4a40 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472583] [a0367872] x86_decode_insn+0x352/0xed0 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472607] [a03550d9] kvm_get_cs_db_l_bits+0x29/0x50 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472633] [a03575a7] emulate_instruction+0x127/0x330 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472657] [a03608cc] kvm_mmu_page_fault+0x5c/0xa0 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472681] [a0359929] kvm_arch_vcpu_ioctl_run+0x299/0x850 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472705] [a03507cb] kvm_vcpu_ioctl+0x2fb/0x5b0 [kvm] Dec 8 12:28:52 astat kernel: [ 248.472714] [802d4b2f] vfs_ioctl+0x2f/0xb0 Dec 8 12:28:52 astat kernel: [ 248.472721] [802d4c2c] do_vfs_ioctl+0x7c/0x480 Dec 8 12:28:52 astat kernel: [ 248.472728] [80266754] sys_futex+0xc4/0x170 Dec 8 12:28:52 astat kernel: [ 248.472735] [802d50d1] sys_ioctl+0xa1/0xb0 Dec 8 12:28:52 astat kernel: [ 248.472741] [8020c30b] system_call_fastpath+0x16/0x1b Dec 8 12:28:52 astat kernel: [ 248.472746] ---[ end trace 0f717136b7f1724f ]--- IRQ 488 is assigned to the device passed through to the guest. Joerg -- | AMD Saxony Limited Liability Company Co. KG Operating | Wilschdorfer Landstr. 101, 01109 Dresden, Germany System| Register Court Dresden: HRA 4896 Research | General Partner authorized to represent: Center| AMD Saxony LLC (Wilmington, Delaware, US) | General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHSETS] KVM device passthrough support with AMD IOMMU
On Thu, Dec 04, 2008 at 06:21:42PM +0100, Joerg Roedel wrote: git://git.kernel.org/pub/scm/linux/kernel/git/joro/linux-2.6-iommu.git kvm-amd-iommu FYI, I just updated the branch above. I added a cleanup function which removes the devices from a protection domain before it is released. Joerg -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCHSETS] KVM device passthrough support with AMD IOMMU
Hi, the two patchsets posted as reply to this email implement KVM device passthrough support for AMD IOMMU hardware. The first patchset is version 3 of the generic iommu api patchset which generalizes the VT-d functions exported to KVM into a common api where the AMD IOMMU code can plug into. The second patchset finally implements the KVM device passthrough support in the AMD IOMMU code. Together with KVM-79 I successfully passed an 10GBit network card into an KVM guest. These two patchsets apply in order in top of the latest post of Han Weidong's Multiple device assignement support patches. Anybody who wants to try this out can pull the whole stuff from git://git.kernel.org/pub/scm/linux/kernel/git/joro/linux-2.6-iommu.git kvm-amd-iommu Please give these patches a good review. Thanks, Joerg -- | AMD Saxony Limited Liability Company Co. KG Operating | Wilschdorfer Landstr. 101, 01109 Dresden, Germany System| Register Court Dresden: HRA 4896 Research | General Partner authorized to represent: Center| AMD Saxony LLC (Wilmington, Delaware, US) | General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
