[dpdk-dev] SR-IOV/DPDK/VPP with vfio-pci
[root at localhost ~]# modinfo i40e filename: /lib/modules/3.10.0-327.18.2.el7.x86_64/updates/drivers/net/ethernet/intel/i40e/i40e.ko version:1.4.25 [root at localhost ~]# modinfo i40evf filename: /lib/modules/3.10.0-327.18.2.el7.x86_64/updates/drivers/net/ethernet/intel/i40evf/i40evf.ko version:1.4.15 dpdk ver : 16.04 nic type : 83:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02) [root at localhost ~]# ethtool -i ens11 driver: i40e version: 1.4.25 firmware-version: 5.04 0x8000253b 0.0.0 bus-info: :83:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes [root at localhost ~]# Testcase: - Enable SRIOV # echo 4 > /sys/class/net/ens11/device/sriov_numvfs 83:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02) 83:02.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 02) 83:02.1 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 02) 83:02.2 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 02) 83:02.3 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 02) 85:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02) - modprobe vfio-pci [root at localhost ~]# /root/ckannan/vpp/build-root/build-vpp-native/dpdk/dpdk-16.04/tools/dpdk_nic_bind.py --bind=vfio-pci :83:02.0 [root at localhost ~]# /root/ckannan/vpp/build-root/build-vpp-native/dpdk/dpdk-16.04/tools/dpdk_nic_bind.py --bind=vfio-pci :83:02.1 [root at localhost ~]# /root/ckannan/vpp/build-root/build-vpp-native/dpdk/dpdk-16.04/tools/dpdk_nic_bind.py --bind=vfio-pci :83:02.2 [root at localhost ~]# /root/ckannan/vpp/build-root/build-vpp-native/dpdk/dpdk-16.04/tools/dpdk_nic_bind.py --bind=vfio-pci :83:02.3 - launch testpmd ./testpmd -l 13,14,15,16 -n 4 -- -i -a --nb-cores=3 --nb-ports=1 I start seeing errors like these... Jul 01 17:23:29 localhost.localdomain kernel: dmar: DRHD: handling fault status reg 2 Jul 01 17:23:29 localhost.localdomain kernel: dmar: DMAR:[DMA Write] Request device [83:02.0] fault addr 0 DMAR:[fault reason 05] PTE Write access is not set Jul 01 17:23:29 localhost.localdomain kernel: i40e :83:00.0: TX driver issue detected, PF reset issued Jul 01 17:23:29 localhost.localdomain kernel: i40e :83:00.0: TX driver issue detected on VF 0 On Sun, Jun 5, 2016 at 8:47 AM, Zhang, Helin wrote: > Hi Kannan > > Could you help to provide more details of your test cases? How to > reproduce that issue? It would be better to try with DPDK example > applications. > I think the kernel driver version, DPDK version, XL710 firmware version > are the basic information we want to know. > If possible, I'd suggest you to try with latest version of above and try > if the issue is still there. > Thank you very much! > > Regards, > Helin > > > -Original Message- > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Chandrasekar > > Kannan > > Sent: Thursday, June 2, 2016 2:22 AM > > To: dev at dpdk.org > > Subject: [dpdk-dev] SR-IOV/DPDK/VPP with vfio-pci > > > > I have been attempting to VPP (dpdk) application with SR-IOV enabled. > > Using the vfio-pci driver/i40e/XL710nics/. > > I'm encountering a SEGV on the i40e code path. > > > > Has anyone else seen this behaviour ?. > > > > full mail thread discussion with vpp-dev is here - > > https://lists.fd.io/pipermail/vpp-dev/2016-June/001348.html > > > > > > > > here's a backtrace during vpp crash... > > > > Program received signal SIGSEGV, Segmentation fault. > > [Switching to Thread 0x7f2b811fd700 (LWP 8725)] > > 0x00455539 in rx_recv_pkts () > > (gdb) backtrace > > #0 0x00455539 in rx_recv_pkts () > > #1 0x00455e76 in i40e_recv_pkts_bulk_alloc () > > #2 0x7f2be43d8bf7 in rte_eth_rx_burst (nb_pkts=256, > > rx_pkts=0x7f2bb1e66a80, queue_id=1, port_id=) > > at > > /w/workspace/vpp-merge-master-centos7/build-root/install-vpp- > > native/dpdk/include/rte_ethdev.h:2641 > > #3 dpdk_rx_burst (dm=0x7f2be4673980 , queue_id=1, > > xd=0x7f2bb1898540) > > at > > /w/workspace/vpp-merge-master-centos7/build- > > data/../vnet/vnet/devices/dpdk/dpdk_priv.h:64 > > #4 dpdk_device_input (dm=0x7f2be4673980 , queue_id=1, > > cpu_index=, node=0x7f2bb1ef59fc, xd=) > > at > > /w/workspace/vpp-merge-master-centos7/build- > > data/../vnet/vnet/devices/dpdk/node.c:508 > > #5 dpdk_input_rss (vm=, node=0x7f2bb1ef59fc, > > f= > out>) > >
[dpdk-dev] [vpp-dev] VLAN packets dropped... ?
Awesome!. Thanks for the quick response John!. -Chandra On Wed, Jun 1, 2016 at 1:02 PM, John Lo (loj) wrote: > We will be adding a patch to the i40e driver that would check the packet > for presence of VLAN tag set the required PKT_RX_VLAN_PKT flag. I hope to > get it done by the end of this week. -John > > > > *From:* Chandrasekar Kannan [mailto:ckannan at console.to] > *Sent:* Wednesday, June 01, 2016 2:45 PM > *To:* John Daley (johndale) > *Cc:* John Lo (loj); Ananyev, Konstantin; Wiles, Keith; vpp-dev; Zhang, > Helin; dev at dpdk.org > *Subject:* Re: [vpp-dev] VLAN packets dropped... ? > > > > Just checking back on this thread to see where things are. Are we saying > this is a bug in dpdk or vpp ?. That part wasn't quite clear to me. > > > > -Chandra > > > > > > > > On Thu, May 26, 2016 at 3:56 PM, John Daley (johndale) > wrote: > > John, > As just discussed, what you suggest was considered but there are other > apps depending a different behavior of the flag, so we thought the best > thing to do is deprecate it. That is part of what Olivier's patch discussed > in http://www.dpdk.org/ml/archives/dev/2016-May/038786.html does. Adding > a new ptype for VLAN tagged packets after the patch is accepted would allow > VPP to check if the ptype is supported by the PMD and if so, use it to > determine if the delivered packet actually has a VLAN tag in it. No need to > know if stripping is enabled/disabled. If the ptype is not supported, the > app would have check the packet in SW. > > > -Original Message- > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of John Lo (loj) > > Sent: Thursday, May 26, 2016 11:11 AM > > To: Ananyev, Konstantin ; Wiles, Keith > > ; Chandrasekar Kannan > > Cc: vpp-dev ; Zhang, Helin > intel.com>; > > dev at dpdk.org > > Subject: Re: [dpdk-dev] [vpp-dev] VLAN packets dropped... ? > > > > Hi Konstantin, > > > > Thanks for the link to DPDK discussion wrt this VLAN offload flag > > PKT_RX_VLAN_PKT. It seems the proposal was to deprecate > > PKT_RX_VLAN_PKT and introduce two new flags PKT_RX_VLAN_STRIPPED > > and PKT_RX_QINQ_STRIPPED. > > > > It would be a really good idea to keep PKT_RX_VLAN_PKT to indicate at > least > > one VLAN tag is present on the packet. For the case of i40e where its HW > > cannot detect VLAN tag if strip is not enabled, it should be reasonable > for the > > i40e DPDK driver software to make a check and set this flag. I would > think the > > overhead to check the 1st ethertype field in the MAC header against dot1q > > or dot1ad values in software be pretty minimal. > > > > Regards, > > John > > > > > > -Original Message- > > From: Ananyev, Konstantin [mailto:konstantin.ananyev at intel.com] > > Sent: Wednesday, May 25, 2016 3:23 PM > > To: John Lo (loj); Wiles, Keith; Chandrasekar Kannan > > Cc: vpp-dev; Zhang, Helin; dev at dpdk.org > > Subject: RE: [vpp-dev] VLAN packets dropped... ? > > > > > > > I suppose this has to do with what is expected usage of the > > > PKT_RX_VLAN_PKT offload flag. Is it set only for VLAN packets with the > > > VLAN stripped or should always be set if VLAN is/was present in the > > > received packet. It seems that different DPDK drivers are behaving > > differently which will make it really hard for VPP to take advantage of > NIC > > and driver offload capability to provide better performance. > > > > Yes, that's true ixgbe/igb from one side and i40e do raise > PKT_RX_VLAN_PKT > > in a different manner. > > There is an attempt to make it unified across all supported devices: > > http://dpdk.org/dev/patchwork/patch/12938/ > > > > Though, I am not sure it will help you with your issue. > > As I understand, for you the desired behaviour is: > > If it is a vlan packet, keep the packet intact (don't strip the vlan) > and raise > > PKT_RX_VLAN_PK inside mbuf->ol_flags. > > That's what ixgbe is doing with rte_eth_conf.rxmode.hw_vlan_strip==0. > > Correct? > > As far as I know, i40e HW doesn't provide such ability. > > i40e Receive HW Descriptor can only flag was VLAN tag stripped from the > > packet or not, but if stripping is disabled it wouldn't indicate in any > way is > > VLAN tag is present inside the packet or not. > > I am CC-ing it to dpdk.org in case I am missing something here. > > Probably someone knows a way to make it work in that way. > > Konstantin > > > > > > > > If VPP cannot rely on offload flags for VLAN
[dpdk-dev] [vpp-dev] VLAN packets dropped... ?
Just checking back on this thread to see where things are. Are we saying this is a bug in dpdk or vpp ?. That part wasn't quite clear to me. -Chandra On Thu, May 26, 2016 at 3:56 PM, John Daley (johndale) wrote: > John, > As just discussed, what you suggest was considered but there are other > apps depending a different behavior of the flag, so we thought the best > thing to do is deprecate it. That is part of what Olivier's patch discussed > in http://www.dpdk.org/ml/archives/dev/2016-May/038786.html does. Adding > a new ptype for VLAN tagged packets after the patch is accepted would allow > VPP to check if the ptype is supported by the PMD and if so, use it to > determine if the delivered packet actually has a VLAN tag in it. No need to > know if stripping is enabled/disabled. If the ptype is not supported, the > app would have check the packet in SW. > > > -Original Message- > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of John Lo (loj) > > Sent: Thursday, May 26, 2016 11:11 AM > > To: Ananyev, Konstantin ; Wiles, Keith > > ; Chandrasekar Kannan > > Cc: vpp-dev ; Zhang, Helin > intel.com>; > > dev at dpdk.org > > Subject: Re: [dpdk-dev] [vpp-dev] VLAN packets dropped... ? > > > > Hi Konstantin, > > > > Thanks for the link to DPDK discussion wrt this VLAN offload flag > > PKT_RX_VLAN_PKT. It seems the proposal was to deprecate > > PKT_RX_VLAN_PKT and introduce two new flags PKT_RX_VLAN_STRIPPED > > and PKT_RX_QINQ_STRIPPED. > > > > It would be a really good idea to keep PKT_RX_VLAN_PKT to indicate at > least > > one VLAN tag is present on the packet. For the case of i40e where its HW > > cannot detect VLAN tag if strip is not enabled, it should be reasonable > for the > > i40e DPDK driver software to make a check and set this flag. I would > think the > > overhead to check the 1st ethertype field in the MAC header against dot1q > > or dot1ad values in software be pretty minimal. > > > > Regards, > > John > > > > > > -Original Message- > > From: Ananyev, Konstantin [mailto:konstantin.ananyev at intel.com] > > Sent: Wednesday, May 25, 2016 3:23 PM > > To: John Lo (loj); Wiles, Keith; Chandrasekar Kannan > > Cc: vpp-dev; Zhang, Helin; dev at dpdk.org > > Subject: RE: [vpp-dev] VLAN packets dropped... ? > > > > > > > I suppose this has to do with what is expected usage of the > > > PKT_RX_VLAN_PKT offload flag. Is it set only for VLAN packets with the > > > VLAN stripped or should always be set if VLAN is/was present in the > > > received packet. It seems that different DPDK drivers are behaving > > differently which will make it really hard for VPP to take advantage of > NIC > > and driver offload capability to provide better performance. > > > > Yes, that's true ixgbe/igb from one side and i40e do raise > PKT_RX_VLAN_PKT > > in a different manner. > > There is an attempt to make it unified across all supported devices: > > http://dpdk.org/dev/patchwork/patch/12938/ > > > > Though, I am not sure it will help you with your issue. > > As I understand, for you the desired behaviour is: > > If it is a vlan packet, keep the packet intact (don't strip the vlan) > and raise > > PKT_RX_VLAN_PK inside mbuf->ol_flags. > > That's what ixgbe is doing with rte_eth_conf.rxmode.hw_vlan_strip==0. > > Correct? > > As far as I know, i40e HW doesn't provide such ability. > > i40e Receive HW Descriptor can only flag was VLAN tag stripped from the > > packet or not, but if stripping is disabled it wouldn't indicate in any > way is > > VLAN tag is present inside the packet or not. > > I am CC-ing it to dpdk.org in case I am missing something here. > > Probably someone knows a way to make it work in that way. > > Konstantin > > > > > > > > If VPP cannot rely on offload flags for VLAN so packets always have to > go > > through ethernet-input node, there is a performance cost. > > > For the 10GE case, before the inverse patch of the ixgbe driver, 10GE > > > Rx-vector path removed support of offload flag with DPDK 16.04 and so > > > ethernet-input node is always used. The 10GE IPv4 throughput rate > > > dropped from 6.17MPPSx2 to 4.92MPPSx2 for bidirectional traffic (2 > > > streams each with a single IP address as destination) on a single core > > > on my server. Konstantin suggested at that time to use scalar mode > which > > does support offload flags properly. The scalar mode did by-pass > ethernet- > > input and p
[dpdk-dev] SR-IOV/DPDK/VPP with vfio-pci
I have been attempting to VPP (dpdk) application with SR-IOV enabled. Using the vfio-pci driver/i40e/XL710nics/. I'm encountering a SEGV on the i40e code path. Has anyone else seen this behaviour ?. full mail thread discussion with vpp-dev is here - https://lists.fd.io/pipermail/vpp-dev/2016-June/001348.html here's a backtrace during vpp crash... Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f2b811fd700 (LWP 8725)] 0x00455539 in rx_recv_pkts () (gdb) backtrace #0 0x00455539 in rx_recv_pkts () #1 0x00455e76 in i40e_recv_pkts_bulk_alloc () #2 0x7f2be43d8bf7 in rte_eth_rx_burst (nb_pkts=256, rx_pkts=0x7f2bb1e66a80, queue_id=1, port_id=) at /w/workspace/vpp-merge-master-centos7/build-root/install-vpp-native/dpdk/include/rte_ethdev.h:2641 #3 dpdk_rx_burst (dm=0x7f2be4673980 , queue_id=1, xd=0x7f2bb1898540) at /w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/dpdk_priv.h:64 #4 dpdk_device_input (dm=0x7f2be4673980 , queue_id=1, cpu_index=, node=0x7f2bb1ef59fc, xd=) at /w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/node.c:508 #5 dpdk_input_rss (vm=, node=0x7f2bb1ef59fc, f=) at /w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/node.c:821 #6 0x7f2be47eb77a in dispatch_node (vm=vm at entry=0x7f2bb1f02cd4, node=node at entry=0x7f2bb1ef59fc, type=type at entry=VLIB_NODE_TYPE_INPUT, dispatch_state=dispatch_state at entry=VLIB_NODE_STATE_POLLING, frame=frame at entry=0x0, last_time_stamp=1482097661494035) at /w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/main.c:996 #7 0x7f2be43db6d8 in dpdk_worker_thread_internal (have_io_threads=0, callback=0x0, vm=0x7f2bb1f02cd4) at /w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:209 #8 dpdk_worker_thread (w=, io_name=, callback=0x0) at /w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:265 #9 0x7f2be3791530 in clib_calljmp () at /w/workspace/vpp-merge-master-centos7/build-data/../vppinfra/vppinfra/longjmp.S:110 #10 0x7f2b811fcc40 in ?? () #11 0x004eb9c7 in eal_thread_loop () ---Type to continue, or q to quit--- #12 0x in ?? () (gdb) (gdb) thread apply all bt Thread 10 (Thread 0x7f2b8c685700 (LWP 8722)): #0 0x7f2be32584ad in accept () at ../sysdeps/unix/syscall-template.S:81 #1 0x004ef178 in pci_vfio_mp_sync_thread () #2 0x7f2be3251dc5 in start_thread (arg=0x7f2b8c685700) at pthread_create.c:308 #3 0x7f2be2992ced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 9 (Thread 0x7f2b821ff700 (LWP 8723)): #0 0x7f2be29932c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x004efe04 in eal_intr_thread_main () #2 0x7f2be3251dc5 in start_thread (arg=0x7f2b821ff700) at pthread_create.c:308 #3 0x7f2be2992ced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 8 (Thread 0x7f2b819fe700 (LWP 8724)): #0 0x0045526b in rx_recv_pkts () #1 0x00455e76 in i40e_recv_pkts_bulk_alloc () #2 0x7f2be43d8bf7 in rte_eth_rx_burst (nb_pkts=256, rx_pkts=0x7f2bb1e75580, queue_id=0, port_id=) at /w/workspace/vpp-merge-master-centos7/build-root/install-vpp-native/dpdk/include/rte_ethdev.h:2641 #3 dpdk_rx_burst (dm=0x7f2be4673980 , queue_id=0, xd=0x7f2bb1a1fd40) at /w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/dpdk_priv.h:64 ---Type to continue, or q to quit--- #4 dpdk_device_input (dm=0x7f2be4673980 , queue_id=0, cpu_index=, node=0x7f2bb1ea18cc, xd=) at /w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/node.c:508 #5 dpdk_input_rss (vm=, node=0x7f2bb1ea18cc, f=) at /w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/node.c:821 #6 0x7f2be47eb77a in dispatch_node (vm=vm at entry=0x7f2bb1f237d4, node=node at entry=0x7f2bb1ea18cc, type=type at entry=VLIB_NODE_TYPE_INPUT, dispatch_state=dispatch_state at entry=VLIB_NODE_STATE_POLLING, frame=frame at entry=0x0, last_time_stamp=148209754800) at /w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/main.c:996 #7 0x7f2be43db6d8 in dpdk_worker_thread_internal (have_io_threads=0, callback=0x0, vm=0x7f2bb1f237d4) at /w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:209 #8 dpdk_worker_thread (w=, io_name=, callback=0x0) at /w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:265 #9 0x7f2be3791530 in clib_calljmp () at /w/workspace/vpp-merge-master-centos7/build-data/../vppinfra/vppinfra/longjmp.S:110 #10 0x7f2b819fdc40 in ?? () #11 0x004eb9c7 in eal_thread_loop () #12 0x in ?? () Thread 7 (Thread 0x7f2b811fd700 (LWP 8725)): #0 0x00455539 in rx_recv_pkts () #1 0x00455e76 in i40
