[dpdk-dev] Splitting fast-path between two processes
Hi, I have been using DPDK (under 2.1) with a single DPDK fastpath process, however now I want to interface with another legacy user-space application that is able to accept ethernet packets, what is the recommended/best performance approach to relay/share packets back-forth between my DPDK application and this legacy application ? I want to avoid turning my legacy application into a complete DPDK application if possible. Thanks in advance for your help. Regards,-Mansoor
[dpdk-dev] Recommended method of using DPDK inside a Vmware ESXi guest ?
Hi, I am trying to use dpdk inside a Vmware 6.0 ESXi guest and see two possible approaches to achieving this: Use vmxnet3-usermap with PMD Use igb_uio with SRIOV passthrough from ESXi host to guest My guest systems are either Fedora Linux FC20 or Ubuntu 15.04 instances with 8 vCpu's and 8 Gigs of memory. When using l2fwd example using igb_uio, the performance numbers I got were very low (<100 PPS) and when I tried using vmxnet3-usermap from dpdk.org, it does not even seem to compile under the Linux 3.x Kernel. My system is a HP DL380 G8 platform with dual Xeon 2670 CPU's and 64 Gb physical memory and Intel 82599 NIC card running ESXi 6.0 Do I need to enable any additional Kernel build or boot options, based on some posts here, appears ESXi does not have emulated IOMMU. Thanks in advance for your help. -Ale Mansoor
[dpdk-dev] vmxnet2-usermap kmod compile errors with ubuntu 15.04
Downloaded the latest vmxnet3-usermap package (ver 1.2) from dpdk.org, tried
compiling it under an Ubuntu VM but it fails to compile, is there a newer
version of this driver available from somewhere that will compile correctly
under Ubuntu 15.04 ?
The kernel (Ubuntu 15.04) "uname -a" ===> Linux ubuntu-vm-mansoor
3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC 2015 x86_64 x86_64
x86_64 GNU/Linux
First I got an error about undefined VM_RESERVED, which I fixed by setting to
(VM_DONTEXPAND | VM_DONTDUMP) to get past the error, now I get following
compile errors, have followed the instructions inside the
"vmxnet3-usermap-1.2/kmod/README" file.
Also noticed the message "Using 2.6.x kernel build system", have setup the RTE
environment variables as below:
# env | grep
RTERTE_INCLUDE=/home/mansoor/dpdk_download/dpdk_2.1/dpdk-2.1.0/build/includeRTE_SDK=/home/mansoor/dpdk_download/dpdk_2.1/dpdk-2.1.0RTE_TARGET=x86_64-native-linuxapp-gcc
Thanks in advance for your help.
# makeUsing 2.6.x kernel build system.make -C
/lib/modules/3.19.0-15-generic/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \
MODULEBUILDDIR= modulesmake[1]: Entering directory
'/usr/src/linux-headers-3.19.0-15-generic' CC [M]
/home/mansoor/dpdk_download/dpdk_2.1/dpdk-2.1.0/vmxnet3-usermap-1.2/kmod/vmxnet3_ethtool.o/home/mansoor/dpdk_download/dpdk_2.1/dpdk-2.1.0/vmxnet3-usermap-1.2/kmod/vmxnet3_ethtool.c:
In function
???vmxnet3_set_features???:/home/mansoor/dpdk_download/dpdk_2.1/dpdk-2.1.0/vmxnet3-usermap-1.2/kmod/vmxnet3_ethtool.c:361:48:
error: ???NETIF_F_HW_VLAN_RX??? undeclared (first use in this function) if
(changed & (NETIF_F_RXCSUM | NETIF_F_LRO | NETIF_F_HW_VLAN_RX)) {
^/home/mansoor/dpdk_download/dpdk_2.1/dpdk-2.1.0/vmxnet3-usermap-1.2/kmod/vmxnet3_ethtool.c:361:48:
note: each undeclared identifier is reported only once for each function it
appears
in/home/mansoor/dpdk_download/dpdk_2.1/dpdk-2.1.0/vmxnet3-usermap-1.2/kmod/vmxnet3_ethtool.c:
In function
???vmxnet3_set_ethtool_ops???:/home/mansoor/dpdk_download/dpdk_2.1/dpdk-2.1.0/vmxnet3-usermap-1.2/kmod/vmxnet3_ethtool.c:677:2:
error: implicit declaration of function ???SET_ETHTOOL_OPS???
[-Werror=implicit-function-declaration] SET_ETHTOOL_OPS(netdev,
_ethtool_ops); ^cc1: some warnings being treated as
errorsscripts/Makefile.build:257: recipe for target
'/home/mansoor/dpdk_download/dpdk_2.1/dpdk-2.1.0/vmxnet3-usermap-1.2/kmod/vmxnet3_ethtool.o'
failedmake[2]: ***
[/home/mansoor/dpdk_download/dpdk_2.1/dpdk-2.1.0/vmxnet3-usermap-1.2/kmod/vmxnet3_ethtool.o]
Error 1Makefile:1394: recipe for target
'_module_/home/mansoor/dpdk_download/dpdk_2.1/dpdk-2.1.0/vmxnet3-usermap-1.2/kmod'
failedmake[1]: ***
[_module_/home/mansoor/dpdk_download/dpdk_2.1/dpdk-2.1.0/vmxnet3-usermap-1.2/kmod]
Error 2make[1]: Leaving directory
'/usr/src/linux-headers-3.19.0-15-generic'Makefile:123: recipe for target
'vmxnet3-usermap.ko' failedmake: *** [vmxnet3-usermap.ko] Error 2
[dpdk-dev] Poor SRIOV performance with ESXi Linux guest
Thank you for your input, earlier on this same ESXi server another similar guest was able to run the Kernel mode ixgbevf driver and push several hundred kilo packets per second via the kernel mode driver. I am trying to get my hands on a second similar physical system for comparison, but likely the hardware and ESXi SRIOV VF mapping into the guest is not the issue. Are there any Kernel options or DPDK options that may influence performance so significantly under ESXi ? I used the tools/setup.sh script under DPDK version 2.1 for building and setting up my DPDK environment. My Kernel UIO settings are below: -- cat /boot/config-3.19.8-100.fc20.x86_64 |grep -i uio CONFIG_HID_HUION=m CONFIG_UIO=m CONFIG_UIO_CIF=m # CONFIG_UIO_PDRV_GENIRQ is not set # CONFIG_UIO_DMEM_GENIRQ is not set CONFIG_UIO_AEC=m CONFIG_UIO_SERCOS3=m CONFIG_UIO_PCI_GENERIC=m # CONFIG_UIO_NETX is not set # CONFIG_UIO_MF624 is not set -- Regards Ale > Date: Wed, 2 Sep 2015 15:31:04 -0700 > From: stephen at networkplumber.org > To: mansooraa at hotmail.com > CC: dev at dpdk.org > Subject: Re: [dpdk-dev] Poor SRIOV performance with ESXi Linux guest > > On Wed, 2 Sep 2015 22:18:27 + > Ale Mansoor wrote: > > > Getting less than 100 packets per second throughput between VF's under my > > Fedora FC20 VM running under ESXi 6.0 with DPDK l2fwd (Used as ./l2fwd -c > > 0xf -n 4 -- -p 0x3 -T 1) > > That is many orders of magnitude less than expected. > > > > Questions: > > --- > > > > Q1) Is DPDK + SRIOV under ESXi supposed to use the igb_uio driver or the > > vfio-pci driver inside Linux guest os ? > > You have to use igb_uio, there is no emulated IOMMU in ESX > > > Q2) What is the expected l2fwd performance when running DPDK under the > > Linux guest OS under ESXI with SRIOV ? > > Depends on many things. With SRIOV you should reach 10Mpps or more. > Did you try running Linux on baremetal on same hardware first? > > > Q3) Any idea what may be preventing vfio-pci driver from binding to the > > VF's inside the guest instance ? > > vfio-pci needs IOMMU which is not available in guest. > > > Q4) Why is igb_uio performing so poorly ? > > Don't blame igb_uio. It is probably something in system or vmware. >
[dpdk-dev] Poor SRIOV performance with ESXi Linux guest
Getting less than 100 packets per second throughput between VF's under my Fedora FC20 VM running under ESXi 6.0 with DPDK l2fwd (Used as ./l2fwd -c 0xf -n 4 -- -p 0x3 -T 1) Questions: --- Q1) Is DPDK + SRIOV under ESXi supposed to use the igb_uio driver or the vfio-pci driver inside Linux guest os ? Q2) What is the expected l2fwd performance when running DPDK under the Linux guest OS under ESXI with SRIOV ? Q3) Any idea what may be preventing vfio-pci driver from binding to the VF's inside the guest instance ? Q4) Why is igb_uio performing so poorly ? My hardware setup: --- HP DL380-G8 host with dual Xeon-2670 CPU 64 GB RAM Intel 82599 NIC connected to 10G switch HP onboard bnx2 NIC's. VMware Hypervisor ESXi 6.0: --- On above hardware running ESXi 6.0 hypervisor and Intel ixgbe VMware (driver version: 3.21.4.3iov). ESXi side SRIOV is setup as follows: "ixgbe_enabled = 1 options = max_vfs=0,8,8,0" (Total 16 VF's configured) VF's are pass-through to the two guest instances. First 4 VF's to one vm instance and next 2 VF's to second vm instance, remaining VF's are unused, have also tried with single guest instances having 4 VF's and remaining unused, but it did not make any difference My VMware guest instances: Have tried two different guests: Linux guest setup each with 8 cores and 8GB memory (memory is setup as locked down to vm to help with iommu). 1) Fedora FC20 OS (Linux kernel 3.19.8-100.fc20) 2) Ubuntu 15.04 Server distro (Linux kernel 3.19.0-15), On both these guests systems my Linux kernel boot line is set with "intel_iommu=on and iommu=pt" My kernel command info for Fedora guest is as follows: $cat /proc/cmdline --- BOOT_IMAGE=/vmlinuz-3.19.8-100.fc20.x86_64 root=UUID=f0cde7fc-d835-4d90-b086-82bf88f58f88 ro console=tty0 console=ttyS0,9600 rhgb quiet net.ifnames=0 biosdevname=0 intel_iommu=on iommu=pt My dmesg has message: "Intel-IOMMU: enabled" Kernel IOMMU config: -- cat /boot/config-3.19.8-100.fc20.x86_64 |grep -i iommu # CONFIG_GART_IOMMU is not set # CONFIG_CALGARY_IOMMU is not set CONFIG_IOMMU_HELPER=y CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_IOMMU_API=y CONFIG_IOMMU_SUPPORT=y CONFIG_AMD_IOMMU=y CONFIG_AMD_IOMMU_STATS=y CONFIG_AMD_IOMMU_V2=m CONFIG_INTEL_IOMMU=y # CONFIG_INTEL_IOMMU_DEFAULT_ON is not set CONFIG_INTEL_IOMMU_FLOPPY_WA=y # CONFIG_IOMMU_STRESS is not set cat /boot/config-3.19.8-100.fc20.x86_64 |grep -i VIRTUAL CONFIG_FB_VIRTUAL=m # CONFIG_DEBUG_VIRTUAL is not set CONFIG_VIRTUALIZATION=y ./dpdk_nic_bind.py --status Network devices using DPDK-compatible driver :03:00.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=ixgbevf,vfio-pci :0b:00.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=ixgbevf,vfio-pci :13:00.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=ixgbevf,vfio-pci :1b:00.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=ixgbevf,vfio-pci Network devices using kernel driver === :02:00.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth4 drv=e1000 unused=igb_uio,vfio-pci *Active* Other network devices = The l2fw program is run with: ./l2fwd -c 0xf -n 4 -- -p 0x3 -T 1 [Using the default igb_uio driver shipped with Fedora Fc20] Getting around 70 packets per second with these settings: When I tried using the vfio-pci driver (again using the default vfio-pci driver from Fedora FC20), get the following errors: ./dpdk_nic_bind.py --bind vfio-pci :03:00.0 Error: bind failed for :03:00.0 - Cannot bind to driver vfio-pci dmesg shows: [ 882.685134] vfio-pci: probe of :03:00.0 failed with error -22 My guest side lspci shows: --- 03:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 0b:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 13:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 1b:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) My ESXi side VMware lscpi shows: -- :09:00.0 Network controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection [vmnic4] :09:00.1 Network controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection [vmnic5] :09:10.1 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.9.1_VF_0] :09:10.3 Network controller: Intel Corporation 82599 Ethernet Controller Virtual Function [PF_0.9.1_VF_1] :09:10.5 Network controller: Intel Corporation 82599 Ethernet
