Dear VPP Team, I’m currently experiencing an issue with my VPP device where DPDK TX-Failure occurs on CQSFP2.
Interestingly, when traffic reaches around 30 Gbps during peak hours, the device operates normally. However, when traffic decreases to around 17 Gbps at night, DPDK TX-Failure starts to appear, followed by increased latency of approximately 40–70 ms. Below are the specifications of the device: * CPU: AMD EPYC 7542 * RAM: 32 GB * NIC: Mellanox ConnectX-5 516A-CDAT * OS: Debian 12.15 * VPP Version: 25.10 * FRRouting: 9.1.3 Restarting VPP does not resolve the issue. The issue is only resolved after rebooting the entire device. Based on this behavior, I suspect there may be an issue with the NIC, particularly on the CQSFP2 interface. However, I would like to confirm this before proceeding with further investigation. Could you please advise if there are any known issues or recommended troubleshooting steps for this kind of DPDK TX-Failure? Have you encountered a similar issue with Mellanox ConnectX-5 NICs? Any suggestions regarding what I should check on the VPP, DPDK, or NIC side would be greatly appreciated. Thank you.
cqsfp2 2 up cqsfp2
Link speed: 40 Gbps
RX Queues:
queue thread mode
0 vpp_wk_8 (9) polling
1 vpp_wk_9 (10) polling
2 vpp_wk_10 (11) polling
3 vpp_wk_11 (12) polling
4 vpp_wk_12 (13) polling
5 vpp_wk_13 (14) polling
6 vpp_wk_14 (15) polling
7 vpp_wk_15 (16) polling
8 vpp_wk_16 (17) polling
9 vpp_wk_17 (18) polling
10 vpp_wk_18 (19) polling
11 vpp_wk_19 (20) polling
TX Queues:
TX Hash: [name: hash-eth-l34 priority: 50 description: Hash ethernet L34
headers]
queue shared thread(s)
0 yes 0, 14
1 yes 1, 15
2 yes 2, 16
3 yes 3, 17
4 yes 4, 18
5 yes 5, 19
6 yes 6, 20
7 no 7
8 no 8
9 no 9
10 no 10
11 no 11
12 no 12
13 no 13
Ethernet address ec:0d:9a:a6:22:a7
Mellanox ConnectX-4/5/6 Family
carrier up full duplex max-frame-size 2048
flags: admin-up subif tx-offload intel-phdr-cksum rx-ip4-cksum
Devargs:
rx: queues 12 (max 1024), desc 8192 (min 0 max 32768 align 1)
tx: queues 14 (max 1024), desc 8192 (min 0 max 32768 align 1)
pci: device 15b3:1019 subsystem 15b3:0008 address 0000:01:00.01 numa 0
switch info: name 0000:01:00.1 domain id 1 port id 65535
max rx packet len: 65536
promiscuous: unicast off all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum tcp-lro
vlan-filter scatter timestamp keep-crc rss-hash
buffer-split
rx offload active: ipv4-cksum
tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum tcp-tso
outer-ipv4-cksum vxlan-tnl-tso gre-tnl-tso geneve-tnl-tso
multi-segs mbuf-fast-free udp-tnl-tso ip-tnl-tso
tx offload active: ipv4-cksum udp-cksum tcp-cksum outer-ipv4-cksum
rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
ipv6-ex ipv6 esp l4-dst-only l4-src-only l3-dst-only
l3-src-only
rss active: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
ipv6-ex ipv6
tx burst mode: Enhanced MPW + SWP + CSUM + INLINE + METADATA
tx burst function: (not available)
rx burst mode: Vector SSE
rx burst function: (not available)
tx frames ok 883092078
tx bytes ok 1080705716118
rx frames ok 430447332
rx bytes ok 102307491015
extended stats:
rx_good_packets 430447332
tx_good_packets 883092078
rx_good_bytes 102307491015
tx_good_bytes 1080705716118
rx_q0_packets 35512101
rx_q0_bytes 8146324141
rx_q1_packets 37756267
rx_q1_bytes 8873730275
rx_q2_packets 35536108
rx_q2_bytes 8737349227
rx_q3_packets 34368034
rx_q3_bytes 8282796344
rx_q4_packets 34644129
rx_q4_bytes 8044850188
rx_q5_packets 36396030
rx_q5_bytes 8876893275
rx_q6_packets 36895539
rx_q6_bytes 8492903722
rx_q7_packets 36546185
rx_q7_bytes 8822469980
rx_q8_packets 36093640
rx_q8_bytes 7964789450
rx_q9_packets 35631014
rx_q9_bytes 8633477833
rx_q10_packets 35299747
rx_q10_bytes 8619311558
rx_q11_packets 35768538
rx_q11_bytes 8812595022
tx_q0_packets 119358
tx_q0_bytes 34819202
tx_q1_packets 113555416
tx_q1_bytes 138893498333
tx_q2_packets 111363533
tx_q2_bytes 136407543419
tx_q3_packets 108871842
tx_q3_bytes 133461698418
tx_q4_packets 110684364
tx_q4_bytes 135307789883
tx_q5_packets 109233494
tx_q5_bytes 133378560378
tx_q6_packets 107235439
tx_q6_bytes 131129263080
tx_q7_packets 109384487
tx_q7_bytes 134181263809
tx_q8_packets 112527806
tx_q8_bytes 137856026681
tx_q9_packets 26086
tx_q9_bytes 4760256
tx_q10_packets 5273
tx_q10_bytes 1833365
tx_q11_packets 16233
tx_q11_bytes 5564373
tx_q12_packets 27437
tx_q12_bytes 12829324
tx_q13_packets 41310
tx_q13_bytes 30265597
rx_unicast_bytes 102307787814
rx_multicast_bytes 153115
rx_broadcast_bytes 129205
rx_unicast_packets 430447334
rx_multicast_packets 1337
rx_broadcast_packets 1142
tx_unicast_bytes 1080705681536
tx_multicast_bytes 34778
tx_broadcast_bytes 540
tx_unicast_packets 883091744
tx_multicast_packets 325
tx_broadcast_packets 9
tx_phy_packets 883092081
rx_phy_packets 430449425
tx_phy_bytes 1084238085358
rx_phy_bytes 104029822763
startup.conf
Description: Binary data
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#27213): https://lists.fd.io/g/vpp-dev/message/27213 Mute This Topic: https://lists.fd.io/mt/121485004/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
