Hi Jan,
I captured the ixl(4) TX queue stall while it was happening
on OpenBSD 7.9/amd64 GENERIC.MP.
1. Hardware and configuration
=============================
Intel XXV710 SFP28 dual-port adapter:
- ixl0: port 0, FW 9.856.79020, API 1.15, 8 queues
- ixl1: port 1, FW 9.856.79020, API 1.15, 8 queues
Both ports were active at 25GbaseCR full-duplex, MTU 9000,
as members of aggr2 (LACP).
Production traffic uses VLANs, CARP and PF forwarding.
There is no bridge configuration.
This setup worked on OpenBSD 7.7. OpenBSD 7.8 was not tested.
2. Reproduction and chronology
==============================
Synthetic iperf3 tests using TCP, UDP, different transfer
sizes, parallel streams and bidirectional traffic did not
reproduce the problem.
The iperf3 tests were performed with the same underlying
network configuration, but not all production VLANs
were present at that point.
After stopping iperf3, I switched the affected router
to CARP BACKUP, restored all production VLANs, and
rebooted it.
I then switched the router to CARP MASTER. Within a
few minutes, the TX queue stall reappeared under normal
production traffic, without iperf3 running.
I captured diagnostics at approximately 09:43:43 and
09:43:53 CEST on October 9, 2026.
I then moved production traffic back to the other firewall.
The affected router has remained CARP BACKUP since then.
I did not attempt another production failover because
the problem was clearly affecting traffic.
3. Main finding: ixl1 TX queue 4
================================
>From the two full kstat ixl1::: captures:
t0 t+10s delta
packets 1702287 1702302 +15
bytes 1998230281 1998231331 +1050
qdrops 34112 35509 +1397
qlen 1024 1024 unchanged
maxqlen 1024 1024 unchanged
oactive true true
oactives 1 1
errors 0 0
The software TX queue remained full, oactive stayed true,
and qdrops increased while the packet counter barely moved.
Other TX queues continued transmitting.
4. Persistence of the stall after failback
==========================================
After the initial captures, production traffic was
moved back to the other firewall. The affected router
remained CARP BACKUP.
At 10:56:01 CEST, the affected TX queue was still full:
09:43:53 10:56:01 delta
ixl1 TX q4 packets 1702302 1703119 +817
ixl1 TX q4 qdrops 35517 48556 +13039
ixl1 TX q4 qlen 1024 1024 unchanged
ixl1 TX q4 oactive true true unchanged
All other TX queues on both interfaces had qlen=0,
qdrops=0 and oactive=false at 10:56:01.
These are cumulative counters; no continuous sampling
was performed between the observations.
5. dmesg - device identification and driver messages
====================================================
ixl0 at pci15 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW
9.856.79020 API 1.15, msix, 8 queues, address [REDACTED]
ixl1 at pci15 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW
9.856.79020 API 1.15, msix, 8 queues, address [REDACTED]
Driver error/timeout/watchdog/reset messages:
No matching messages found in the captured dmesg.
6. netstat -s - relevant network counters
========================================
t0 t+10s
Output drops (no buffers) 0 0
Output discards (no route) 0 0
Output drops (full output queue) 0 0
Input drops (no buffers) 0 0
These are general network stack counters, not the
per-queue ixl software qdrops counters.
7. kstat - all TX queues, both interfaces
========================================
The per-queue summaries below were extracted separately
from the full kstat snapshots used in section 3.
Some counters therefore differ slightly because the
sampling times were not exactly identical.
Columns: interface queue packets qdrops qlen oactive
--- t0 ---
ixl0 q0 packets=1611029 qdrops=0 qlen=0 oactive=false
ixl0 q1 packets=1310909 qdrops=0 qlen=0 oactive=false
ixl0 q2 packets=1904026 qdrops=0 qlen=0 oactive=false
ixl0 q3 packets=2247120 qdrops=0 qlen=0 oactive=false
ixl0 q4 packets=1857250 qdrops=0 qlen=0 oactive=false
ixl0 q5 packets=3877865 qdrops=0 qlen=0 oactive=false
ixl0 q6 packets=996624 qdrops=0 qlen=0 oactive=false
ixl0 q7 packets=1700511 qdrops=0 qlen=0 oactive=false
ixl1 q0 packets=1383968 qdrops=0 qlen=0 oactive=false
ixl1 q1 packets=2462384 qdrops=0 qlen=0 oactive=false
ixl1 q2 packets=1413506 qdrops=0 qlen=0 oactive=false
ixl1 q3 packets=1496771 qdrops=0 qlen=0 oactive=false
ixl1 q4 packets=1702289 qdrops=34189 qlen=1024 oactive=true
ixl1 q5 packets=2852569 qdrops=0 qlen=0 oactive=false
ixl1 q6 packets=1769569 qdrops=0 qlen=0 oactive=false
ixl1 q7 packets=3165406 qdrops=0 qlen=0 oactive=false
--- t10 ---
ixl0 q0 packets=1621469 qdrops=0 qlen=0 oactive=false
ixl0 q1 packets=1345917 qdrops=0 qlen=0 oactive=false
ixl0 q2 packets=1919066 qdrops=0 qlen=0 oactive=false
ixl0 q3 packets=2267470 qdrops=0 qlen=0 oactive=false
ixl0 q4 packets=1866928 qdrops=0 qlen=0 oactive=false
ixl0 q5 packets=3884631 qdrops=0 qlen=0 oactive=false
ixl0 q6 packets=1011239 qdrops=0 qlen=0 oactive=false
ixl0 q7 packets=1711710 qdrops=0 qlen=0 oactive=false
ixl1 q0 packets=1401927 qdrops=0 qlen=0 oactive=false
ixl1 q1 packets=2479432 qdrops=0 qlen=0 oactive=false
ixl1 q2 packets=1419343 qdrops=0 qlen=0 oactive=false
ixl1 q3 packets=1503891 qdrops=0 qlen=0 oactive=false
ixl1 q4 packets=1702302 qdrops=35517 qlen=1024 oactive=true
ixl1 q5 packets=2860654 qdrops=0 qlen=0 oactive=false
ixl1 q6 packets=1819686 qdrops=0 qlen=0 oactive=false
ixl1 q7 packets=3183726 qdrops=0 qlen=0 oactive=false
Hardware TX counters (selected lines):
--- t0 ---
ixl0:
tx link xon: 0 packets
tx link xoff: 0 packets
tx link down: 0 packets
tx errs: 0 packets
tx discards: 0 packets
ixl1:
tx link xon: 0 packets
tx link xoff: 0 packets
tx link down: 0 packets
tx errs: 0 packets
tx discards: 0 packets
--- t10 ---
ixl0:
tx link xon: 0 packets
tx link xoff: 0 packets
tx link down: 0 packets
tx errs: 0 packets
tx discards: 0 packets
ixl1:
tx link xon: 0 packets
tx link xoff: 0 packets
tx link down: 0 packets
tx errs: 0 packets
tx discards: 0 packets
8. ifconfig ixl - link state
============================
--- t0 ---
ixl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 9000
trunk: trunkdev aggr2
media: Ethernet autoselect (25GbaseCR full-duplex)
status: active
ixl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 9000
trunk: trunkdev aggr2
media: Ethernet autoselect (25GbaseCR full-duplex)
status: active
--- t10 ---
ixl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 9000
trunk: trunkdev aggr2
media: Ethernet autoselect (25GbaseCR full-duplex)
status: active
ixl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 9000
trunk: trunkdev aggr2
media: Ethernet autoselect (25GbaseCR full-duplex)
status: active
9. vmstat -i - affected queue interrupt
=======================================
--- t0 ---
irq116/ixl1:4 833294 686
--- t10 ---
irq116/ixl1:4 836707 684
The interrupt count increased from 833294 to 836707
(+3413), but this does not establish whether TX
completions were progressing correctly.
10. Operational plan
====================
Once diagnostic collection is complete, I will migrate
the production VLANs away from the ixl interfaces to
other network interfaces, then reboot the affected router
to restore a reliable hot-standby configuration.
The XXV710 interfaces will no longer carry production
traffic.
I have retained the complete original diagnostics
collected during the stall. The excerpts above focus
on the relevant driver and TX queue state.
Please let me know if any additional driver state would
be useful for investigating this.
Thanks,
