Hi Jerome,
Apologies for the delay. Answers inline, with what we were able to capture before moving the NIC off AF_PACKET. > Is there a deployment constraint preventing either approach? No. Management currently shares the same NIC, which makes experiments costly (each failed attempt leaves us on console only), but there is no requirement to keep the interface in Linux. We are willing to move to DPDK or SR-IOV VFs. > show hardware-interfaces You were right — it reports v3, not v2. We had assumed V2 from the "v2" keyword in the create command. Relevant portion: host-enp129s0 1 up host-enp129s0 Link speed: unknown RX Queues: queue thread mode 0 main (0) interrupt TX Queues: TX Hash: [name: hash-eth-l34 priority: 50 description: Hash ethernet L34 headers] queue shared thread(s) 0 no 0 Ethernet address 02:fe:cc:4c:3c:6b Linux PACKET socket interface v3 FEATURES: qdisc-bpass-enabled cksum-gso-enabled RX Queue 0: block size:65536 nr:160 frame size:2048 nr:5120 next block:47 TX Queue 0: block size:69206016 nr:1 frame size:67584 nr:1024 next frame:1015 available:1024 request:0 sending:0 wrong:0 total:1024 > show threads ID Name Type LWP Sched Policy (Priority) lcore 0 vpp_main 311679 other (0) 16 Only vpp_main — no workers were ever spawned. > show interface rx-placement Thread 0 (vpp_main): node af-packet-input: host-enp129s0 queue 0 (interrupt) > cpu section actually loaded This is where we think our mistake was. The cpu block was commented out in the loaded startup.conf, which explains the missing workers. When we did uncomment corelist-workers, VPP sometimes failed to start, so we reverted. We have not yet established a working worker configuration on this box (20 cores, single NUMA, Intel Core Ultra 7 265K). > ethtool -l enp129s0 Pre-set maximums: RX: 0 TX: 0 Other: 1 Combined: 32 Current hardware settings: RX: 0 TX: 0 Other: 1 Combined: 20 So the NIC had 20 combined queues configured while VPP's AF_PACKET interface was using a single queue. > traffic rates, packet sizes, CPU utilisation Measured from interface counters over a 30s window during the failure: inside sub-if (.244): rx 510,726 pkts / 188,565,178 bytes (~17k pps, ~50 Mbps) tx 604,093 pkts / 685,590,742 bytes (~20k pps, ~183 Mbps) avg packet size: ~369 B inbound, ~1135 B outbound show runtime at a later point: vector rate 2.38, vector rates in 1.0528e4, out 1.0462e4, drop 6.0342e1, punt 1.3354e1 af-packet-input: 280122 calls, 15668846 vectors, 2.16e5 clocks, 55.94 vectors/call We did not capture per-core CPU utilisation at the moment of failure — apologies. > errno for the "tx sendto fatal failure" events We no longer have those log entries; the journal has rotated and the NIC has since been unbound from the kernel, so we cannot reproduce them without reverting. In the most recent run the counter was "tx sendto temporary failure" (192,543) rather than fatal, alongside 3,329,159 "tx frame not ready". > SIGABRT backtrace We do not have a usable backtrace — debug symbols were not installed and no core was retained. The trigger was an invalid CLI line in our startup script: ip route add 103.213.236.136/30 via drop We were attempting a blackhole route and guessed at the syntax. VPP aborted while parsing it, with the backtrace ending in vlib_cli_input. Could you confirm the correct syntax for a blackhole route in VPP? And regardless of the syntax error, a malformed route expression causing SIGABRT rather than a CLI error seems worth reporting. VPP aborted while reading it, with the backtrace ending in vlib_cli_input. We assume "via drop" is simply invalid syntax — if so, could you confirm the correct syntax for a blackhole route? Either way, an invalid route expression causing SIGABRT rather than a CLI error seems worth noting. === Where we are now === Based on your advice we bound the XL710 to vfio-pci. VPP now shows only local0 and emits no DPDK log lines at all. We are running kernel 5.15.0-60 with no intel_iommu=on in /proc/cmdline, though /sys/kernel/iommu_groups/ is populated. Hugepages are at 4096 (2MB), 4075 free. I will send the DPDK details in a separate message so this one stays focused. Thanks for your patience. Kind Regards, Bayozid Bostami Senior System Administrator I NOC Inspire Broadband Limited | 312, South Paikpara, Mirpur, Dhaka-1216, Bangladesh. Mob: +8801678331118 From: Bayozid Bostami <[email protected]> To: "vpp-dev"<[email protected]> Date: Tue, 06 Oct 2026 13:49:34 +0600 Subject: Re: [vpp-dev] af_packet "tx frame not ready" and instability under CGNAT load Hi Jerome, Thanks for your response. I will share the current config with you. Kind Regards, Bayozid Bostami Senior System Administrator I NOC Inspire Broadband | 312, South Paikpara, Mirpur, Dhaka-1216, Bangladesh. Mob: +8801678331118 From: Jerome Tollet via lists.fd.io <jtollet= mailto:[email protected] > To: < mailto:[email protected] > Date: Tue, 06 Oct 2026 12:30:41 +0600 Subject: Re: [vpp-dev] af_packet "tx frame not ready" and instability under CGNAT load Hi Bayozid, For your target of 10,000 CGNAT subscribers, I would recommend evaluating DPDK or VPP's native AVF/IAVF driver. AF_PACKET introduces kernel processing and additional packet copies, which can limit throughput and scaling. Your XL710 is supported by DPDK's i40e PMD. The native AVF/IAVF driver is another option using an SR-IOV VF. Is there a deployment constraint preventing either approach, such as needing the interface to remain available to Linux or lacking PCI/VF access? To understand the current setup, could you provide: - The full "show hardware-interfaces" output. You mention TPACKET_V2, but the posted creation command defaults to V3 in VPP 26.06. - "show threads", "show interface rx-placement", and the "cpu" section of the startup configuration actually loaded. - "ethtool -l enp129s0", to distinguish the NIC's configured queues from VPP's AF_PACKET queues. - Average and peak traffic rates in both directions, in packets/s and bits/s, plus typical packet sizes and CPU utilization. - The log entries containing the exact errno for the 36 "tx sendto fatal failure" events. These details would help distinguish transient congestion from a persistent TX problem and guide queue and worker sizing. For the SIGABRT, could you also provide the complete backtrace ("thread apply all bt full" in GDB, preferably with debugging symbols), any assertion message, the preceding startup logs, and "show version verbose"? Please include the smallest startup script that reproduces the crash. Thanks, Jerome
startup.conf
Description: Binary data
vpp.conf
Description: Binary data
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#27260): https://lists.fd.io/g/vpp-dev/message/27260 Mute This Topic: https://lists.fd.io/mt/121605598/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
