Re: PCI passthrough of 40G ethernet interface (Openstack/KVM)
I also see the following in dmesg in the VM. [0.095758] ACPI: PCI Root Bridge [PCI0] (domain [bus 00-ff]) [0.096006] acpi PNP0A03:00: ACPI _OSC support notification failed, disabling PCIe ASPM [0.096915] acpi PNP0A03:00: Unable to request _OSC control (_OSC support mask: 0x08) [0.097072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. Does this indicate any issue related to PCI passthrough? Would really appreciate any input on how to bebug this further. On Fri, Mar 13, 2015 at 10:08 AM, jacob jacob opstk...@gmail.com wrote: So, it could be the i40e driver then ? Because IIUC, VFs use a separate driver. Just to rule out the possibility that there might be some driver fixes that could help with this, it might be a good idea to try a 3.19 or later upstream kernel. I tried with the latest DPDK release too (dpdk-1.8.0) and see the same issue. As mentioned earlier, i do not see any issues at all when running tests using either i40e or dpdk on the host itself. This is the reason why i am suspecting if it is anything to do with KVM/libvirt. Both with regular PCI passthrough and VF passthrough i see issues. It is always pointing to some issue with packet transmission. Receive seems to work ok. On Thu, Mar 12, 2015 at 8:02 PM, Bandan Das b...@redhat.com wrote: jacob jacob opstk...@gmail.com writes: On Thu, Mar 12, 2015 at 3:07 PM, Bandan Das b...@redhat.com wrote: jacob jacob opstk...@gmail.com writes: Hi, Seeing failures when trying to do PCI passthrough of Intel XL710 40G interface to KVM vm. 0a:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01) You are assigning the PF right ? Does assigning VFs work or it's the same behavior ? Yes.Assigning VFs worked ok.But this had other issues while bringing down VMs. Interested in finding out if PCI passthrough of 40G intel XL710 interface is qualified in some specific kernel/kvm release. So, it could be the i40e driver then ? Because IIUC, VFs use a separate driver. Just to rule out the possibility that there might be some driver fixes that could help with this, it might be a good idea to try a 3.19 or later upstream kernel. From dmesg on host: [80326.559674] kvm: zapping shadow pages for mmio generation wraparound [80327.271191] kvm [175994]: vcpu0 unhandled rdmsr: 0x1c9 [80327.271689] kvm [175994]: vcpu0 unhandled rdmsr: 0x1a6 [80327.272201] kvm [175994]: vcpu0 unhandled rdmsr: 0x1a7 [80327.272681] kvm [175994]: vcpu0 unhandled rdmsr: 0x3f6 [80327.376186] kvm [175994]: vcpu0 unhandled rdmsr: 0x606 These are harmless and are related to unimplemented PMU msrs, not VFIO. Bandan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough of 40G ethernet interface (Openstack/KVM)
So, it could be the i40e driver then ? Because IIUC, VFs use a separate driver. Just to rule out the possibility that there might be some driver fixes that could help with this, it might be a good idea to try a 3.19 or later upstream kernel. I tried with the latest DPDK release too (dpdk-1.8.0) and see the same issue. As mentioned earlier, i do not see any issues at all when running tests using either i40e or dpdk on the host itself. This is the reason why i am suspecting if it is anything to do with KVM/libvirt. Both with regular PCI passthrough and VF passthrough i see issues. It is always pointing to some issue with packet transmission. Receive seems to work ok. On Thu, Mar 12, 2015 at 8:02 PM, Bandan Das b...@redhat.com wrote: jacob jacob opstk...@gmail.com writes: On Thu, Mar 12, 2015 at 3:07 PM, Bandan Das b...@redhat.com wrote: jacob jacob opstk...@gmail.com writes: Hi, Seeing failures when trying to do PCI passthrough of Intel XL710 40G interface to KVM vm. 0a:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01) You are assigning the PF right ? Does assigning VFs work or it's the same behavior ? Yes.Assigning VFs worked ok.But this had other issues while bringing down VMs. Interested in finding out if PCI passthrough of 40G intel XL710 interface is qualified in some specific kernel/kvm release. So, it could be the i40e driver then ? Because IIUC, VFs use a separate driver. Just to rule out the possibility that there might be some driver fixes that could help with this, it might be a good idea to try a 3.19 or later upstream kernel. From dmesg on host: [80326.559674] kvm: zapping shadow pages for mmio generation wraparound [80327.271191] kvm [175994]: vcpu0 unhandled rdmsr: 0x1c9 [80327.271689] kvm [175994]: vcpu0 unhandled rdmsr: 0x1a6 [80327.272201] kvm [175994]: vcpu0 unhandled rdmsr: 0x1a7 [80327.272681] kvm [175994]: vcpu0 unhandled rdmsr: 0x3f6 [80327.376186] kvm [175994]: vcpu0 unhandled rdmsr: 0x606 These are harmless and are related to unimplemented PMU msrs, not VFIO. Bandan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough of 40G ethernet interface (Openstack/KVM)
On Thu, 2015-03-12 at 12:17 -0400, jacob jacob wrote: Hi, Seeing failures when trying to do PCI passthrough of Intel XL710 40G interface to KVM vm. How is the device being assigned, pci-assign or vfio-pci? What QEMU version? What host kernel version? Thanks, Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough of 40G ethernet interface (Openstack/KVM)
Hi Alex, Thanks for the response. I tried both pci-assign and vfio-pci. The issue is seen in both cases. i40e driver complains about data tx timeout. # libvirtd --version libvirtd (libvirt) 1.2.9.2 Name: qemu-system-x86 Arch: x86_64 Epoch : 2 Version : 2.1.3 Release : 2.fc21 Kernel 3.18.7-200.fc21.x86_64 Rgds Jacob On Thu, Mar 12, 2015 at 12:26 PM, Alex Williamson alex.william...@redhat.com wrote: On Thu, 2015-03-12 at 12:17 -0400, jacob jacob wrote: Hi, Seeing failures when trying to do PCI passthrough of Intel XL710 40G interface to KVM vm. How is the device being assigned, pci-assign or vfio-pci? What QEMU version? What host kernel version? Thanks, Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough of 40G ethernet interface (Openstack/KVM)
jacob jacob opstk...@gmail.com writes: Hi, Seeing failures when trying to do PCI passthrough of Intel XL710 40G interface to KVM vm. 0a:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01) You are assigning the PF right ? Does assigning VFs work or it's the same behavior ? From dmesg on host: [80326.559674] kvm: zapping shadow pages for mmio generation wraparound [80327.271191] kvm [175994]: vcpu0 unhandled rdmsr: 0x1c9 [80327.271689] kvm [175994]: vcpu0 unhandled rdmsr: 0x1a6 [80327.272201] kvm [175994]: vcpu0 unhandled rdmsr: 0x1a7 [80327.272681] kvm [175994]: vcpu0 unhandled rdmsr: 0x3f6 [80327.376186] kvm [175994]: vcpu0 unhandled rdmsr: 0x606 These are harmless and are related to unimplemented PMU msrs, not VFIO. Bandan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough of 40G ethernet interface (Openstack/KVM)
jacob jacob opstk...@gmail.com writes: On Thu, Mar 12, 2015 at 3:07 PM, Bandan Das b...@redhat.com wrote: jacob jacob opstk...@gmail.com writes: Hi, Seeing failures when trying to do PCI passthrough of Intel XL710 40G interface to KVM vm. 0a:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01) You are assigning the PF right ? Does assigning VFs work or it's the same behavior ? Yes.Assigning VFs worked ok.But this had other issues while bringing down VMs. Interested in finding out if PCI passthrough of 40G intel XL710 interface is qualified in some specific kernel/kvm release. So, it could be the i40e driver then ? Because IIUC, VFs use a separate driver. Just to rule out the possibility that there might be some driver fixes that could help with this, it might be a good idea to try a 3.19 or later upstream kernel. From dmesg on host: [80326.559674] kvm: zapping shadow pages for mmio generation wraparound [80327.271191] kvm [175994]: vcpu0 unhandled rdmsr: 0x1c9 [80327.271689] kvm [175994]: vcpu0 unhandled rdmsr: 0x1a6 [80327.272201] kvm [175994]: vcpu0 unhandled rdmsr: 0x1a7 [80327.272681] kvm [175994]: vcpu0 unhandled rdmsr: 0x3f6 [80327.376186] kvm [175994]: vcpu0 unhandled rdmsr: 0x606 These are harmless and are related to unimplemented PMU msrs, not VFIO. Bandan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough of 40G ethernet interface (Openstack/KVM)
On Thu, Mar 12, 2015 at 3:07 PM, Bandan Das b...@redhat.com wrote: jacob jacob opstk...@gmail.com writes: Hi, Seeing failures when trying to do PCI passthrough of Intel XL710 40G interface to KVM vm. 0a:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01) You are assigning the PF right ? Does assigning VFs work or it's the same behavior ? Yes.Assigning VFs worked ok.But this had other issues while bringing down VMs. Interested in finding out if PCI passthrough of 40G intel XL710 interface is qualified in some specific kernel/kvm release. From dmesg on host: [80326.559674] kvm: zapping shadow pages for mmio generation wraparound [80327.271191] kvm [175994]: vcpu0 unhandled rdmsr: 0x1c9 [80327.271689] kvm [175994]: vcpu0 unhandled rdmsr: 0x1a6 [80327.272201] kvm [175994]: vcpu0 unhandled rdmsr: 0x1a7 [80327.272681] kvm [175994]: vcpu0 unhandled rdmsr: 0x3f6 [80327.376186] kvm [175994]: vcpu0 unhandled rdmsr: 0x606 These are harmless and are related to unimplemented PMU msrs, not VFIO. Bandan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
