Getting less than 100 packets per second throughput between VF's under my Fedora FC20 VM running under ESXi 6.0 with DPDK l2fwd (Used as ./l2fwd -c 0xf -n 4 -- -p 0x3 -T 1)
Questions: --------------- Q1) Is DPDK + SRIOV under ESXi supposed to use the igb_uio driver or the vfio-pci driver inside Linux guest os ? Q2) What is the expected l2fwd performance when running DPDK under the Linux guest OS under ESXI with SRIOV ? Q3) Any idea what may be preventing vfio-pci driver from binding to the VF's inside the guest instance ? Q4) Why is igb_uio performing so poorly ? My hardware setup: --------------------------- HP DL380-G8 host with dual Xeon-2670 CPU 64 GB RAM Intel 82599 NIC connected to 10G switch HP onboard bnx2 NIC's. VMware Hypervisor ESXi 6.0: --------------------------------------- On above hardware running ESXi 6.0 hypervisor and Intel ixgbe VMware (driver version: 3.21.4.3iov). ESXi side SRIOV is setup as follows: "ixgbe_enabled = 1 options = max_vfs=0,8,8,0" (Total 16 VF's configured) VF's are pass-through to the two guest instances. First 4 VF's to one vm instance and next 2 VF's to second vm instance, remaining VF's are unused, have also tried with single guest instances having 4 VF's and remaining unused, but it did not make any difference My VMware guest instances: ---------------------------------------- Have tried two different guests: Linux guest setup each with 8 cores and 8GB memory (memory is setup as locked down to vm to help with iommu). 1) Fedora FC20 OS (Linux kernel 3.19.8-100.fc20) 2) Ubuntu 15.04 Server distro (Linux kernel 3.19.0-15), On both these guests systems my Linux kernel boot line is set with "intel_iommu=on and iommu=pt" My kernel command info for Fedora guest is as follows: $cat /proc/cmdline --------------------------- BOOT_IMAGE=/vmlinuz-3.19.8-100.fc20.x86_64 root=UUID=f0cde7fc-d835-4d90-b086-82bf88f58f88 ro console=tty0 console=ttyS0,9600 rhgb quiet net.ifnames=0 biosdevname=0 intel_iommu=on iommu=pt My dmesg has message: "Intel-IOMMU: enabled" Kernel IOMMU config: ------------------------------ cat /boot/config-3.19.8-100.fc20.x86_64 |grep -i iommu # CONFIG_GART_IOMMU is not set # CONFIG_CALGARY_IOMMU is not set CONFIG_IOMMU_HELPER=y CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_IOMMU_API=y CONFIG_IOMMU_SUPPORT=y CONFIG_AMD_IOMMU=y CONFIG_AMD_IOMMU_STATS=y CONFIG_AMD_IOMMU_V2=m CONFIG_INTEL_IOMMU=y # CONFIG_INTEL_IOMMU_DEFAULT_ON is not set CONFIG_INTEL_IOMMU_FLOPPY_WA=y # CONFIG_IOMMU_STRESS is not set cat /boot/config-3.19.8-100.fc20.x86_64 |grep -i VIRTUAL CONFIG_FB_VIRTUAL=m # CONFIG_DEBUG_VIRTUAL is not set CONFIG_VIRTUALIZATION=y ./dpdk_nic_bind.py --status ---------------------------------------- Network devices using DPDK-compatible driver ============================================ 0000:03:00.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=ixgbevf,vfio-pci 0000:0b:00.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=ixgbevf,vfio-pci 0000:13:00.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=ixgbevf,vfio-pci 0000:1b:00.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=ixgbevf,vfio-pci Network devices using kernel driver =================================== 0000:02:00.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth4 drv=e1000 unused=igb_uio,vfio-pci *Active* Other network devices ===================== <none> The l2fw program is run with: ./l2fwd -c 0xf -n 4 -- -p 0x3 -T 1 [Using the default igb_uio driver shipped with Fedora Fc20] Getting around 70 packets per second with these settings: When I tried using the vfio-pci driver (again using the default vfio-pci driver from Fedora FC20), get the following errors: ./dpdk_nic_bind.py --bind vfio-pci 0000:03:00.0 Error: bind failed for 0000:03:00.0 - Cannot bind to driver vfio-pci dmesg shows: [ 882.685134] vfio-pci: probe of 0000:03:00.0 failed with error -22 My guest side lspci shows: ----------------------------------- 03:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 0b:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 13:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 1b:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) My ESXi side VMware lscpi shows: ---------------------------------------------- 0000:09:00.0 Network controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection [vmnic4] 0000:09:00.1 Network controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection [vmnic5] 0000:09:10.1 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.9.1_VF_0] 0000:09:10.3 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.9.1_VF_1] 0000:09:10.5 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.9.1_VF_2] 0000:09:10.7 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.9.1_VF_3] 0000:09:11.1 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.9.1_VF_4] 0000:09:11.3 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.9.1_VF_5] 0000:09:11.5 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.9.1_VF_6] 0000:09:11.7 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.9.1_VF_7] 0000:0c:00.0 Network controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection [vmnic6] 0000:0c:00.1 Network controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection [vmnic7] 0000:0c:10.0 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.12.0_VF_0] 0000:0c:10.2 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.12.0_VF_1] 0000:0c:10.4 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.12.0_VF_2] 0000:0c:10.6 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.12.0_VF_3] 0000:0c:11.0 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.12.0_VF_4] 0000:0c:11.2 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.12.0_VF_5] 0000:0c:11.4 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.12.0_VF_6] 0000:0c:11.6 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.12.0_VF_7] --- Regards Ale