[vpp-dev] Segmentation fault when dpdk number-rx-queues > 1 in startup.conf
Hi, all
I meet one issue that when I set dpdk rx queue number larger than 1 (which also
enables RSS) in startup.conf, for example:
dev default {
# Number of receive queues, enables RSS
# Default is 1
num-rx-queues 2
}
When start VPP, it will meet segmentation fault, the error log is:
..
dpdk [debug ]: [0] interface dpdk_eth0 created
interface/rx-queue [debug ]: set_input_node: node dpdk-input for interface
dpdk_eth0
interface/rx-queue [debug ]: register: interface dpdk_eth0 queue-id 0 thread 1
interface/rx-queue [debug ]: register: interface dpdk_eth0 queue-id 1 thread 2
dpdk [debug ]: [0] configuring device name: :d8:00.0, numa: 1, driver:
net_ice, bus: pci
dpdk [debug ]: [0] Supported RX offloads: vlan-strip ipv4-cksum udp-cksum
tcp-cksum qinq-strip
outer-ipv4-cksum vlan-filter vlan-extend scatter
timestamp keep-crc rss-hash
dpdk [debug ]: [0] Configured RX offloads: ipv4-cksum scatter
dpdk [debug ]: [0] Supported TX offloads: vlan-insert ipv4-cksum udp-cksum
tcp-cksum sctp-cksum
tcp-tso outer-ipv4-cksum qinq-insert multi-segs
mbuf-fast-free outer-udp-cksum
dpdk [debug ]: [0] Configured TX offloads: ipv4-cksum udp-cksum tcp-cksum
multi-segs
Segmentation fault (core dumped)
I think I find the bad commit: ce4083ce48958d9d3956e8317445a5552780af1a ("dpdk:
offloads cleanup")
Does anyone also meet issue? Is there any solution to it? Thanks!
Best Regards,
Xu Ting
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20946): https://lists.fd.io/g/vpp-dev/message/20946
Mute This Topic: https://lists.fd.io/mt/89520993/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] #vpp-hoststack - Issue with UDP receiver application with epoll_wait()
Hi Florin, Just converted the app to edge triggered. And it's working great. Thanks! -Kyle On 3/2/22, 6:23 PM, "Florin Coras" wrote: Hi Kyle, Is the app by chance trying to do epoll in level trigger mode? If yes, support for it was pushed in sometime around 21.06 so the code you are running might be missing some features. It’s also less tested when compared with edge trigger. Conversely, if the app's running epoll in edge trigger mode, make sure it drains the buffers, i.e., reads until it gets an EAGAIN. UDP with LD_PRELOAD is part of our CI (see here [1]) but compared to your use case iperf “connects” the udp sessions. Regards, Florin [1] https://git.fd.io/vpp/tree/test/test_vcl.py#n783 > On Mar 2, 2022, at 2:46 PM, Kyle Upton (kyupton) via lists.fd.io wrote: > > Hi, > > I have a simple UDP recvfrom() application (using LD_PRELOAD). When I just use recvfrom(), I can receive packets played from a small PCAP file. > However, when I wrap the recvfrom() in an epoll_wait(), I only receive one packet. The second packet appears to be in the Rx fifo (tail is greater than head). Subsequent PCAP replays show the tail increasing by the size of the packets. > > VPP release is 21.06 > > DBGvpp# show app server > Connection App Wrk > [0:0][CT:U] 0.0.0.0:2055->0.0.0.0:0 ldp-83368-app 0 > [0:1][U] 0.0.0.0:2055->0.0.0.0:0ldp-83368-app 0 > > root@3355a029760b:/opt/project/vpp/vcl# VCL_DEBUG=2 LDP_DEBUG=2 LD_PRELOAD=/opt/project/vpp/vpp-repo/build-root/install-vpp_debug-native/vpp/lib/libvcl_ldpreload.so ./udp_rx > VCL<83368>: configured VCL debug level (2) from VCL_DEBUG! > ldp<83368>: fd 3: calling libc_close > VCL<83368>: allocated VCL heap = 0x7f2eb0cf2000, size 268435456 (0x1000) > VCL<83368>: configured app_scope_global (1) > VCL<83368>: configured app_scope_local (1) > VCL<83368>: configured app-socket-api (/var/run/vpp/app_ns_sockets/default) > VCL<83368>: completed parsing vppcom config! > ldp<83368>: fd 3: calling libc_close > ldp<83368>: calling libc_socket > ldp<83368>: fd 3: calling libc_connect(): addr 0x7ffcb7750368, len 110 > vppcom_app_create:1377: vcl<83368:0>: app_name 'ldp-83368-app', my_client_index 1 (0x1) > ldp<83368>: configured LDP debug level (2) from env var LDP_DEBUG! > ldp<83368>: LDP initialization: done! > ldp_constructor:2690: LDP<83368>: LDP constructor: done! > ldp<83368>: calling vls_create: proto 1 (UDP), is_nonblocking 0 > vppcom_session_create:1436: vcl<83368:0>: created session 0 > Created a socket with fd: 32 > ldp<83368>: fd 32 vlsh 0, cmd 3 > socket flags are: 2 > ldp<83368>: fd 32 vlsh 0, cmd 4 > ldp<83368>: fd 32: calling vls_bind: vlsh 0, addr 0x7ffcb774f500, len 16 > vppcom_session_bind:1571: vcl<83368:0>: session 0 handle 0: binding to local IPv4 address 0.0.0.0 port 2055, proto UDP > vppcom_session_listen:1603: vcl<83368:0>: session 0: sending vpp listen request... > vcl_session_app_add_segment_handler:1033: vcl<83368:0>: mapped new segment '14609-27' size 134217728 > vcl_session_bound_handler:665: vcl<83368:0>: session 0 [0x1]: listen succeeded! > ldp<83368>: fd 32: returning 0 > vppcom_epoll_create:2668: vcl<83368:0>: Created vep_idx 1 > ldp<83368>: epoll_create epfd 33 vlsh 1 > ldp<83368>: epfd 33 ep_vlsh 1, fd 32 vlsh 0, op 1 > ldp<83368>: epfd 33: calling vls_epoll_ctl: ep_vlsh 1 op 1, vlsh 0, event 0x7ffcb774f4f4 > vppcom_epoll_ctl:2788: vcl<83368:0>: EPOLL_CTL_ADD: vep_sh 1, sh 0, events 0x1, data 0x20! > Let us wait for a remote client to send some data > About to epoll_wait() > > fd=32; events: EPOLLIN > Received data (len 132 bytes): > About to epoll_wait() > > > DBGvpp# show session verbose 2 > [0:0][CT:U] 0.0.0.0:2055->0.0.0.0:0 LISTEN > [0:1][U] 0.0.0.0:2055->0.0.0.0:0LISTEN > index 0 flags: OWNS_PORT, LISTEN > Rx fifo: cursize 1073 nitems 1048576 has_event 1 min_alloc 65536 > head 177 tail 1250 segment manager 3 > vpp session 1 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 4294967295 > Tx fifo: cursize 0 nitems 1048576 has_event 0 min_alloc 65536 > head 0 tail 0 segment manager 3 > vpp session 1 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 0 > Thread 0: active sessions 2 > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20945): https://lists.fd.io/g/vpp-dev/message/20945 Mute This Topic: https://lists.fd.io/mt/89513626/21656 Mute
Re: [vpp-dev] #vpp-hoststack - Issue with UDP receiver application with epoll_wait()
Hi Kyle, Is the app by chance trying to do epoll in level trigger mode? If yes, support for it was pushed in sometime around 21.06 so the code you are running might be missing some features. It’s also less tested when compared with edge trigger. Conversely, if the app's running epoll in edge trigger mode, make sure it drains the buffers, i.e., reads until it gets an EAGAIN. UDP with LD_PRELOAD is part of our CI (see here [1]) but compared to your use case iperf “connects” the udp sessions. Regards, Florin [1] https://git.fd.io/vpp/tree/test/test_vcl.py#n783 > On Mar 2, 2022, at 2:46 PM, Kyle Upton (kyupton) via lists.fd.io > wrote: > > Hi, > > I have a simple UDP recvfrom() application (using LD_PRELOAD). When I just > use recvfrom(), I can receive packets played from a small PCAP file. > However, when I wrap the recvfrom() in an epoll_wait(), I only receive one > packet. The second packet appears to be in the Rx fifo (tail is greater than > head). Subsequent PCAP replays show the tail increasing by the size of the > packets. > > VPP release is 21.06 > > DBGvpp# show app server > Connection App Wrk > [0:0][CT:U] 0.0.0.0:2055->0.0.0.0:0 ldp-83368-app 0 > [0:1][U] 0.0.0.0:2055->0.0.0.0:0ldp-83368-app 0 > > root@3355a029760b:/opt/project/vpp/vcl# VCL_DEBUG=2 LDP_DEBUG=2 > LD_PRELOAD=/opt/project/vpp/vpp-repo/build-root/install-vpp_debug-native/vpp/lib/libvcl_ldpreload.so > ./udp_rx > VCL<83368>: configured VCL debug level (2) from VCL_DEBUG! > ldp<83368>: fd 3: calling libc_close > VCL<83368>: allocated VCL heap = 0x7f2eb0cf2000, size 268435456 (0x1000) > VCL<83368>: configured app_scope_global (1) > VCL<83368>: configured app_scope_local (1) > VCL<83368>: configured app-socket-api (/var/run/vpp/app_ns_sockets/default) > VCL<83368>: completed parsing vppcom config! > ldp<83368>: fd 3: calling libc_close > ldp<83368>: calling libc_socket > ldp<83368>: fd 3: calling libc_connect(): addr 0x7ffcb7750368, len 110 > vppcom_app_create:1377: vcl<83368:0>: app_name 'ldp-83368-app', > my_client_index 1 (0x1) > ldp<83368>: configured LDP debug level (2) from env var LDP_DEBUG! > ldp<83368>: LDP initialization: done! > ldp_constructor:2690: LDP<83368>: LDP constructor: done! > ldp<83368>: calling vls_create: proto 1 (UDP), is_nonblocking 0 > vppcom_session_create:1436: vcl<83368:0>: created session 0 > Created a socket with fd: 32 > ldp<83368>: fd 32 vlsh 0, cmd 3 > socket flags are: 2 > ldp<83368>: fd 32 vlsh 0, cmd 4 > ldp<83368>: fd 32: calling vls_bind: vlsh 0, addr 0x7ffcb774f500, len 16 > vppcom_session_bind:1571: vcl<83368:0>: session 0 handle 0: binding to local > IPv4 address 0.0.0.0 port 2055, proto UDP > vppcom_session_listen:1603: vcl<83368:0>: session 0: sending vpp listen > request... > vcl_session_app_add_segment_handler:1033: vcl<83368:0>: mapped new segment > '14609-27' size 134217728 > vcl_session_bound_handler:665: vcl<83368:0>: session 0 [0x1]: listen > succeeded! > ldp<83368>: fd 32: returning 0 > vppcom_epoll_create:2668: vcl<83368:0>: Created vep_idx 1 > ldp<83368>: epoll_create epfd 33 vlsh 1 > ldp<83368>: epfd 33 ep_vlsh 1, fd 32 vlsh 0, op 1 > ldp<83368>: epfd 33: calling vls_epoll_ctl: ep_vlsh 1 op 1, vlsh 0, event > 0x7ffcb774f4f4 > vppcom_epoll_ctl:2788: vcl<83368:0>: EPOLL_CTL_ADD: vep_sh 1, sh 0, events > 0x1, data 0x20! > Let us wait for a remote client to send some data > About to epoll_wait() > > fd=32; events: EPOLLIN > Received data (len 132 bytes): > About to epoll_wait() > > > DBGvpp# show session verbose 2 > [0:0][CT:U] 0.0.0.0:2055->0.0.0.0:0 LISTEN > [0:1][U] 0.0.0.0:2055->0.0.0.0:0LISTEN > index 0 flags: OWNS_PORT, LISTEN > Rx fifo: cursize 1073 nitems 1048576 has_event 1 min_alloc 65536 > head 177 tail 1250 segment manager 3 > vpp session 1 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 4294967295 > Tx fifo: cursize 0 nitems 1048576 has_event 0 min_alloc 65536 > head 0 tail 0 segment manager 3 > vpp session 1 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 0 > Thread 0: active sessions 2 > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20944): https://lists.fd.io/g/vpp-dev/message/20944 Mute This Topic: https://lists.fd.io/mt/89513626/21656 Mute #vpp-hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-hoststack Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] #vpp-hoststack - Issue with UDP receiver application with epoll_wait()
Hi, I have a simple UDP recvfrom() application (using LD_PRELOAD). When I just use recvfrom(), I can receive packets played from a small PCAP file. However, when I wrap the recvfrom() in an epoll_wait(), I only receive one packet. The second packet appears to be in the Rx fifo (tail is greater than head). Subsequent PCAP replays show the tail increasing by the size of the packets. VPP release is 21.06 DBGvpp# show app server Connection App Wrk [0:0][CT:U] 0.0.0.0:2055->0.0.0.0:0 ldp-83368-app 0 [0:1][U] 0.0.0.0:2055->0.0.0.0:0ldp-83368-app 0 root@3355a029760b:/opt/project/vpp/vcl# VCL_DEBUG=2 LDP_DEBUG=2 LD_PRELOAD=/opt/project/vpp/vpp-repo/build-root/install-vpp_debug-native/vpp/lib/libvcl_ldpreload.so ./udp_rx VCL<83368>: configured VCL debug level (2) from VCL_DEBUG! ldp<83368>: fd 3: calling libc_close VCL<83368>: allocated VCL heap = 0x7f2eb0cf2000, size 268435456 (0x1000) VCL<83368>: configured app_scope_global (1) VCL<83368>: configured app_scope_local (1) VCL<83368>: configured app-socket-api (/var/run/vpp/app_ns_sockets/default) VCL<83368>: completed parsing vppcom config! ldp<83368>: fd 3: calling libc_close ldp<83368>: calling libc_socket ldp<83368>: fd 3: calling libc_connect(): addr 0x7ffcb7750368, len 110 vppcom_app_create:1377: vcl<83368:0>: app_name 'ldp-83368-app', my_client_index 1 (0x1) ldp<83368>: configured LDP debug level (2) from env var LDP_DEBUG! ldp<83368>: LDP initialization: done! ldp_constructor:2690: LDP<83368>: LDP constructor: done! ldp<83368>: calling vls_create: proto 1 (UDP), is_nonblocking 0 vppcom_session_create:1436: vcl<83368:0>: created session 0 Created a socket with fd: 32 ldp<83368>: fd 32 vlsh 0, cmd 3 socket flags are: 2 ldp<83368>: fd 32 vlsh 0, cmd 4 ldp<83368>: fd 32: calling vls_bind: vlsh 0, addr 0x7ffcb774f500, len 16 vppcom_session_bind:1571: vcl<83368:0>: session 0 handle 0: binding to local IPv4 address 0.0.0.0 port 2055, proto UDP vppcom_session_listen:1603: vcl<83368:0>: session 0: sending vpp listen request... vcl_session_app_add_segment_handler:1033: vcl<83368:0>: mapped new segment '14609-27' size 134217728 vcl_session_bound_handler:665: vcl<83368:0>: session 0 [0x1]: listen succeeded! ldp<83368>: fd 32: returning 0 vppcom_epoll_create:2668: vcl<83368:0>: Created vep_idx 1 ldp<83368>: epoll_create epfd 33 vlsh 1 ldp<83368>: epfd 33 ep_vlsh 1, fd 32 vlsh 0, op 1 ldp<83368>: epfd 33: calling vls_epoll_ctl: ep_vlsh 1 op 1, vlsh 0, event 0x7ffcb774f4f4 vppcom_epoll_ctl:2788: vcl<83368:0>: EPOLL_CTL_ADD: vep_sh 1, sh 0, events 0x1, data 0x20! Let us wait for a remote client to send some data About to epoll_wait() fd=32; events: EPOLLIN Received data (len 132 bytes): About to epoll_wait() DBGvpp# show session verbose 2 [0:0][CT:U] 0.0.0.0:2055->0.0.0.0:0 LISTEN [0:1][U] 0.0.0.0:2055->0.0.0.0:0LISTEN index 0 flags: OWNS_PORT, LISTEN Rx fifo: cursize 1073 nitems 1048576 has_event 1 min_alloc 65536 head 177 tail 1250 segment manager 3 vpp session 1 thread 0 app session 0 thread 0 ooo pool 0 active elts newest 4294967295 Tx fifo: cursize 0 nitems 1048576 has_event 0 min_alloc 65536 head 0 tail 0 segment manager 3 vpp session 1 thread 0 app session 0 thread 0 ooo pool 0 active elts newest 0 Thread 0: active sessions 2 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20943): https://lists.fd.io/g/vpp-dev/message/20943 Mute This Topic: https://lists.fd.io/mt/89513626/21656 Mute #vpp-hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-hoststack Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Strange - SRIOV VF VLAN ID is disappeared #dpdk #forwarding
On Wed, Mar 2, 2022 at 05:03 PM, Ray Kinsella wrote: > > I think your approach of configuring VLAN on the sub-interfaces is the > correct one. If I take the way, the VF1 on the host can't receive any Request Reply, VF1 in zone1 linux namespace,tcpdump -i can not catch expect packet, the physical nic is X722,the VF1 driver on the host is iavf,It doesn't support X722,It's the key point? > > This plugins provides native device support for intel Adaptive Virtual > Function (AVF). AVF is driver specification for current and future Intel > Virtual Function devices. AVF defines communication channel between > Physical Functions (PF) and VF. In essence, today this driver can be used > only with Intel XL710 / X710 / XXV710 adapters. > the content above is from this link: https://docs.fd.io/vpp/20.09/d1/def/avf_plugin_doc.html top of this page -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20942): https://lists.fd.io/g/vpp-dev/message/20942 Mute This Topic: https://lists.fd.io/mt/89468543/21656 Mute #dpdk:https://lists.fd.io/g/vpp-dev/mutehashtag/dpdk Mute #forwarding:https://lists.fd.io/g/vpp-dev/mutehashtag/forwarding Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Strange - SRIOV VF VLAN ID is disappeared #dpdk #forwarding
Eric writes: > Thanks in advance > > the trace packet is: > > --- > Packet 17 > > 00:18:16:753781: dpdk-input > dp0 rx queue 0 > buffer 0x9679b: current data 0, length 60, buffer-pool 0, ref-count 1, > totlen-nifb 0, trace handle 0x10 > ext-hdr-valid > l4-cksum-computed l4-cksum-correct > PKT MBUF: port 0, nb_segs 1, pkt_len 60 > buf_len 2176, data_len 60, ol_flags 0x182, data_off 128, phys_addr 0x259e740 > packet_type 0x3 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 > rss 0xcc301824 fdir.hi 0x0 fdir.lo 0xcc301824 > Packet Offload Flags > PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result > PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid > PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid > Packet Types > RTE_PTYPE_L2_ETHER_ARP (0x0003) ARP packet > ARP: 56:26:1a:f2:fe:e1 -> ff:ff:ff:ff:ff:ff 802.1q vlan 100 > request, type ethernet/IP4, address size 6/4 > 56:26:1a:f2:fe:e1/1.1.1.1 -> 00:00:00:00:00:00/1.1.1.3 > 00:18:16:753783: ethernet-input > frame: flags 0x3, hw-if-index 1, sw-if-index 1 > ARP: 56:26:1a:f2:fe:e1 -> ff:ff:ff:ff:ff:ff 802.1q vlan 100 > 00:18:16:753800: error-drop > rx:dp0 > 00:18:16:753800: drop > ethernet-input: unknown vlan > > the host vf configuration has been destoried,the vlan on the vf. > > the vlan id is disappeared > > So a few things:- When you configure VLAN stripping in Linux on a given VF.When you start VPP that interface is claimed by VPP, and I pretty sure that any prior configuration is discarded. I don't think there is a way to configure VLAN stripping in VPP. I took a quick look and I don't see RTE_ETH_RX_OFFLOAD_VLAN_STRIP being passed to DPDK anywhere. I think the VPP philosophy is that it always wants to see the VLAN. I think your approach of configuring VLAN on the sub-interfaces is the correct one. -- Regards, Ray K -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20941): https://lists.fd.io/g/vpp-dev/message/20941 Mute This Topic: https://lists.fd.io/mt/89468543/21656 Mute #dpdk:https://lists.fd.io/g/vpp-dev/mutehashtag/dpdk Mute #forwarding:https://lists.fd.io/g/vpp-dev/mutehashtag/forwarding Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
