Hello VPP community,
We run a small ISP in Dhaka, Bangladesh and use VPP as a CGNAT box. We are hitting stability issues under real subscriber load and would appreciate guidance. === Environment === VPP version : 26.06-release (also tested 24.10) OS : Ubuntu 24.04, kernel 6.8 CPU : Intel Core Ultra 7 265K, 20 cores, single NUMA RAM : 62 GB NIC : 40G Intel xl710 Interface : af_packet v2 (host-enp129s0) === Topology === VLAN 243 (outside) -> Juniper MX, 10.100.112.6/30 VLAN 244 (inside) -> MikroTik BNG, 10.100.112.9/30 NAT pool : 4 public IPs Subscribers : testing with 10-15 users, target 10,000 === Configuration === create host-interface name enp129s0 create sub-interfaces host-enp129s0 243 create sub-interfaces host-enp129s0 244 set interface mtu ip4 1496 host-enp129s0.243 set interface mtu ip4 1496 host-enp129s0.244 set interface ip address host-enp129s0.243 10.100.112.6/30 set interface ip address host-enp129s0.244 10.100.112.9/30 ip route add 0.0.0.0/0 via 10.100.112.5 host-enp129s0.243 ip route add 10.100.31.0/24 via 10.100.112.10 host-enp129s0.244 nat44 plugin enable sessions 1000000 nat44 forwarding enable nat44 add address 103.213.236.136 - 103.213.236.139 set interface nat44 in host-enp129s0.244 out host-enp129s0.243 === Problem 1: TX ring exhaustion === "show errors" after about an hour of modest traffic: 2013850 host-enp129s0-tx tx frame not ready 36 host-enp129s0-tx tx sendto fatal failure 103516 nat44-ed-in2out-output-slowpath non-SYN packet try to create session "show hardware-interfaces" reports: TX Queue 0: frame size 67584, nr 1024, available 1024 RX Queue 0: single queue, interrupt mode Is there a way to increase the af_packet TX ring size in 26.06? "create host-interface" in this build does not expose rx-frame-size / tx-frame-size. Is DPDK the only practical path at this traffic level? === Problem 2: no workers spawned === "show threads" shows only vpp_main even with "corelist-workers 2-9" in startup.conf, and "show interface rx-placement" shows a single queue in interrupt mode. What is the correct way to get multiple workers and queues with af_packet v2? === Problem 3: crash while reading startup-config === VPP aborted with SIGABRT, backtrace ending in vlib_cli_input, while reading our startup-config file. We traced it to: ip route add 103.213.236.136/30 via drop Is this expected? What is the correct syntax for a blackhole route? === Problem 4: scaling guidance === For 10,000 subscribers with a regulatory requirement to retain per-subscriber NAT mappings for one year, would the community recommend det44 over nat44-ed? We would prefer deterministic port-block allocation over per-session IPFIX logging if that is sufficient. Any pointers would be much appreciated. Happy to provide further output or test patches. Kind Regards, Bayozid Bostami Senior System Administrator I NOC Inspire Broadband | 312, South Paikpara, Mirpur, Dhaka-1216, Bangladesh. Mob: +8801678331118
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#27233): https://lists.fd.io/g/vpp-dev/message/27233 Mute This Topic: https://lists.fd.io/mt/121605598/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
