Hi Bayozid,
Thanks for the detailed outputs.
I tested `ip route add 103.213.236.136/30 via drop` on VPP 26.06 debug, both
through the CLI and during startup, with and without NAT and with zero or two
workers. I could not reproduce the crash. This is valid blackhole-route syntax.
I recommend moving packet processing to dedicated workers, separate from the
main thread. Two workers and two RX queues per interface would be a reasonable
starting point. The attached configuration uses `cpu { workers 2 }`, leaving
CPU affinity selection to VPP.
For your XL710, I suggest two SR-IOV VFs using VPP's native IAVF driver: one
for each of your existing VLANs, 243 (outside) and 244 (inside). The PF remains
under Linux's i40e driver for management; only the VFs are bound to vfio-pci.
I validated the native configuration's interface initialization, queue
assignment, MTUs, addresses, routes and NAT on a real XL710 using VPP 26.10,
with laboratory addresses. This was not a traffic/load test. Please replace the
PCI-address placeholders before use.
Since your PF is currently bound to vfio-pci, restoring it to i40e should be
done during a maintenance window with console access. The attached migration
notes describe the precautions.
To investigate the SIGABRT, could you provide:
- `show version verbose`, the startup configuration and the smallest CLI script
reproducing it;
- the log lines immediately before the abort, especially any assertion or
"Out-of-memory" message;
- a full backtrace with matching debug symbols: `thread apply all bt full`;
- `lspci -nnk`, so we can confirm the PF/VF devices and their drivers.
Best regards,
Jerome
# Template for VPP 26.06. VPP selects the main/worker CPU affinity
automatically.
unix {
nodaemon
log /var/log/vpp/vpp.log
full-coredump
cli-listen /run/vpp/cli.sock
gid vpp
exec /etc/vpp/vpp-native-vf.cli
}
api-trace { on }
api-segment { gid vpp }
socksvr { default }
cpu {
workers 2
}
memory { main-heap-size 2G }
buffers { buffers-per-numa 65536 }
plugins {
plugin default { disable }
plugin nat_plugin.so { enable }
plugin ping_plugin.so { enable }
}
# The native iavf_driver.so is loaded from the vpp_drivers directory.
# Do not add a dpdk block or enable dpdk_plugin.so for this option.
comment {VPP 26.06 native IAVF. Replace both VF PCI placeholders before use.}
comment {VF0 is assigned VLAN 243 by the Linux i40e PF; VF1 uses VLAN 244.}
comment {VPP sees untagged traffic on these two VFs: do not create
subinterfaces.}
device attach pci/<OUTSIDE_VF_PCI> driver iavf
device create-interface pci/<OUTSIDE_VF_PCI> if-name outside num-rx-queues 2
num-tx-queues 3 rx-queues-size 1024 tx-queues-size 1024
device attach pci/<INSIDE_VF_PCI> driver iavf
device create-interface pci/<INSIDE_VF_PCI> if-name inside num-rx-queues 2
num-tx-queues 3 rx-queues-size 1024 tx-queues-size 1024
comment {The native device framework distributes the two RX queues across
workers.}
comment {Polling is the default when no interrupt-mode flag is requested.}
set interface mtu packet 1500 outside
set interface mtu packet 1500 inside
set interface mtu ip4 1496 outside
set interface mtu ip4 1496 inside
set interface ip address outside 10.100.112.6/30
set interface ip address inside 10.100.112.9/30
create loopback interface instance 0
set interface ip address loop0 103.213.236.29/32
ip route add 0.0.0.0/0 via 10.100.112.5 outside
ip route add 10.100.30.0/24 via 10.100.112.10 inside
ip route add 10.100.31.0/24 via 10.100.112.10 inside
ip route add 10.100.0.0/21 via 10.100.112.10 inside
ip route add 10.100.8.0/21 via 10.100.112.10 inside
comment {Preserve the existing NAT output-feature policy and timeout values.}
nat44 plugin enable sessions 1000000
nat44 forwarding enable
nat44 add address 103.213.236.136 - 103.213.236.139
set interface nat44 in outside output-feature
set nat timeout udp 60
set nat timeout icmp 30
set nat timeout tcp-established 3600 tcp-transitory 120
ip route add 103.213.236.136/30 via drop
comment {Bring up the interfaces only after the routing and NAT policy is
installed.}
set interface state loop0 up
set interface state outside up
set interface state inside up
Proposal for Bayozid: XL710 CGNAT, VPP 26.06
These are templates, not a configuration to apply blindly. No changes have
been made to the remote system. Confirm the PCI addresses, installed binaries,
and the upstream MAC/VLAN requirements first. The CPU configuration creates
two workers and leaves CPU affinity selection to VPP.
Preferred option: two SR-IOV VFs with the native IAVF driver
--------------------------------------------------------
Keep the PF under Linux i40e for management. Assign one VF to each dataplane
VLAN: VF0 outside/243, VF1 inside/244. Only those VFs are assigned to vfio-pci
and VPP. The Linux PF inserts/strips the assigned port VLAN; the two VPP VF
interfaces are untagged. Do not add VLAN 243/244 subinterfaces in VPP again.
The latest report says the PF is already bound to vfio-pci. Returning it to
i40e and creating VFs is a separate, disruptive migration step. Plan a
maintenance window with console/OOB access. Do not attempt to restore the
only management NIC through an SSH session using that NIC. Do not disable
IOMMU protections or use unsafe no-IOMMU mode as a workaround.
Read-only checks to collect first:
lspci -nnk
ethtool -i enp129s0
cat /proc/cmdline
cat /sys/class/net/enp129s0/device/sriov_numvfs
readlink -f /sys/class/net/enp129s0/device
ls -l /sys/class/net/enp129s0/device/virtfn*
The enp129s0 paths become available again only after the PF is restored to
i40e. Verify the real PF netdev name after rebinding. If VFs already exist,
do not remove or resize them without checking who owns them.
Host configuration outline (ONLY after the maintenance preconditions above):
- Restore the exact PF PCI function to i40e; keep its management setup.
- If sriov_numvfs is zero and the device supports it, create two VFs:
echo 2 | sudo tee /sys/class/net/enp129s0/device/sriov_numvfs
- Assign a unique MAC and a port VLAN to each VF, for example:
sudo ip link set dev enp129s0 vf 0 mac 02:fa:24:30:00:01 vlan 243
sudo ip link set dev enp129s0 vf 1 mac 02:fa:24:40:00:01 vlan 244
- Ordinary L3 routing does not require disabling spoof checking or
enabling promiscuous mode. Keep VF isolation unless a demonstrated
requirement makes a trusted/trunk VF necessary.
- Resolve virtfn0/virtfn1 to their real VF PCI BDFs and verify their IOMMU
groups. Bind only those two VFs to vfio-pci using the available host
device-binding tool. Never substitute the PF BDF for a VF BDF.
- Verify installed iavf_driver.so and nat_plugin.so. In VPP 26.06, use
device attach / device create-interface, not the old create interface avf
CLI. The native driver is not selected by enabling dpdk_plugin.so.
- Use startup-native-vf.conf and vpp-native-vf.cli after replacing all
PCI placeholders. The cpu block only needs workers 2 for this starting
configuration; there are no hard-coded CPU IDs.
VF MAC addresses differ from the old AF_PACKET interface. Confirm that the
adjacent router/switch accepts them and refreshes its neighbor entries. Test
ARP, VLAN isolation, and connectivity in both directions before enabling
production CGNAT traffic. Hardware VF VLAN steering and queue allocation
still require validation on the actual XL710/PF-driver/NVM combination.
Fallback: DPDK on the PF
-----------------------
If VF support or native-driver packaging prevents the preferred option,
startup-dpdk-pf.conf and vpp-dpdk-pf.cli use the XL710 PF via the DPDK i40e
PMD, with VLAN subinterfaces 243/244. The entire PF becomes unavailable to
Linux. This needs a separate management interface or OOB console.
Resolve and explicitly list the actual PF BDF. Confirm dpdk_plugin.so exists
and loads: an enable directive cannot install a missing plugin. Some package
layouts provide it in vpp-plugin-dpdk. Check the package appropriate to the
installed distribution rather than assuming that binding to vfio-pci is enough.
Do not retain create host-interface name enp129s0 in the DPDK CLI script. The
interface is created as uplink by the dpdk startup block.
Sizing and validation
---------------------
Start with two workers and two RX queues per dataplane interface, not all
20 CPU cores. VPP should use polling for the hardware dataplane. The 3 TX
queues cover the main thread plus two workers. Confirm the VF/PF grants this
queue count. Main heap 2G and 65,536 buffers per NUMA are initial sizing
suggestions, not a guarantee of one million active sessions under load.
The existing sessions 1000000 limit is PER THREAD in NAT44-ED, not a shared
global limit. Monitor memory and actual sessions before increasing workers
or loading production traffic. Retain the existing output-feature NAT policy
and timeout values; NAT policy changes need separate traffic validation.
If workers 2 still fails to start, collect the startup error, lscpu -e and
systemctl show vpp -p ExecStart -p CPUAffinity -p AllowedCPUs. Explicit core
pinning is an optional later optimization. When using corelist-workers,
VPP 26.06 also requires main-core to be specified.
After startup collect:
show version verbose
show plugins
show device drivers (native option)
show dpdk version (DPDK option)
show threads
show hardware-interfaces
show device (native option: RX thread/mode/queue details)
show interface rx-placement (DPDK option)
show interface
show nat44 summary
show runtime
show errors
show memory main-heap
The data plane should have two RX queues assigned to workers, not a single
interrupt-mode queue on vpp_main. Keep the main thread for control work.
For native IAVF in 26.06, queue assignment is automatic at interface creation
and polling is the default. Do not use the legacy set interface rx-placement
or set interface rx-mode commands for these native device-framework queues.
Retain the console startup output and assertion/Out-of-memory messages if
VPP aborts. The suffix vlib_cli_input in a backtrace does not identify which
command or allocator failed.
Primary references:
https://docs.kernel.org/networking/device_drivers/ethernet/intel/i40e.html
https://github.com/FDio/vpp/blob/v26.06/src/vnet/dev/cli.c
https://github.com/FDio/vpp/blob/v26.06/src/drivers/iavf/iavf.c
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#27262): https://lists.fd.io/g/vpp-dev/message/27262
Mute This Topic: https://lists.fd.io/mt/121605598/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
