I've just had similar issues with a quad "Intel X710 SFP+ rev 0x02, FW 7.0.50775 API 1.8, msix, 4 queues" though all queues stalled not just some (with OACTIVE in flags). I was using baby jumbos / jumbos (1800/8000).
Have now updated if_ixl.c to r1.119 and removed jumbos from the config and immediate problem resolved, let's see how things go. On 2026/10/09 11:15, Network Operations wrote: > Hi Jan, > > I captured the ixl(4) TX queue stall while it was happening > on OpenBSD 7.9/amd64 GENERIC.MP. > > 1. Hardware and configuration > ============================= > > Intel XXV710 SFP28 dual-port adapter: > - ixl0: port 0, FW 9.856.79020, API 1.15, 8 queues > - ixl1: port 1, FW 9.856.79020, API 1.15, 8 queues > > Both ports were active at 25GbaseCR full-duplex, MTU 9000, > as members of aggr2 (LACP). > > Production traffic uses VLANs, CARP and PF forwarding. > There is no bridge configuration. > > This setup worked on OpenBSD 7.7. OpenBSD 7.8 was not tested. > > 2. Reproduction and chronology > ============================== > > Synthetic iperf3 tests using TCP, UDP, different transfer > sizes, parallel streams and bidirectional traffic did not > reproduce the problem. > > The iperf3 tests were performed with the same underlying > network configuration, but not all production VLANs > were present at that point. > > After stopping iperf3, I switched the affected router > to CARP BACKUP, restored all production VLANs, and > rebooted it. > > I then switched the router to CARP MASTER. Within a > few minutes, the TX queue stall reappeared under normal > production traffic, without iperf3 running. > > I captured diagnostics at approximately 09:43:43 and > 09:43:53 CEST on October 9, 2026. > > I then moved production traffic back to the other firewall. > The affected router has remained CARP BACKUP since then. > > I did not attempt another production failover because > the problem was clearly affecting traffic. > > 3. Main finding: ixl1 TX queue 4 > ================================ > > From the two full kstat ixl1::: captures: > > t0 t+10s delta > packets 1702287 1702302 +15 > bytes 1998230281 1998231331 +1050 > qdrops 34112 35509 +1397 > qlen 1024 1024 unchanged > maxqlen 1024 1024 unchanged > oactive true true > oactives 1 1 > errors 0 0 > > The software TX queue remained full, oactive stayed true, > and qdrops increased while the packet counter barely moved. > > Other TX queues continued transmitting. > > > 4. Persistence of the stall after failback > ========================================== > > After the initial captures, production traffic was > moved back to the other firewall. The affected router > remained CARP BACKUP. > > At 10:56:01 CEST, the affected TX queue was still full: > > 09:43:53 10:56:01 delta > ixl1 TX q4 packets 1702302 1703119 +817 > ixl1 TX q4 qdrops 35517 48556 +13039 > ixl1 TX q4 qlen 1024 1024 unchanged > ixl1 TX q4 oactive true true unchanged > > All other TX queues on both interfaces had qlen=0, > qdrops=0 and oactive=false at 10:56:01. > > These are cumulative counters; no continuous sampling > was performed between the observations. > > 5. dmesg - device identification and driver messages > ==================================================== > > ixl0 at pci15 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW > 9.856.79020 API 1.15, msix, 8 queues, address [REDACTED] > ixl1 at pci15 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW > 9.856.79020 API 1.15, msix, 8 queues, address [REDACTED] > > Driver error/timeout/watchdog/reset messages: > No matching messages found in the captured dmesg. > > 6. netstat -s - relevant network counters > ======================================== > > t0 t+10s > Output drops (no buffers) 0 0 > Output discards (no route) 0 0 > Output drops (full output queue) 0 0 > Input drops (no buffers) 0 0 > > These are general network stack counters, not the > per-queue ixl software qdrops counters. > > > 7. kstat - all TX queues, both interfaces > ======================================== > > The per-queue summaries below were extracted separately > from the full kstat snapshots used in section 3. > Some counters therefore differ slightly because the > sampling times were not exactly identical. > > > Columns: interface queue packets qdrops qlen oactive > > --- t0 --- > ixl0 q0 packets=1611029 qdrops=0 qlen=0 oactive=false > ixl0 q1 packets=1310909 qdrops=0 qlen=0 oactive=false > ixl0 q2 packets=1904026 qdrops=0 qlen=0 oactive=false > ixl0 q3 packets=2247120 qdrops=0 qlen=0 oactive=false > ixl0 q4 packets=1857250 qdrops=0 qlen=0 oactive=false > ixl0 q5 packets=3877865 qdrops=0 qlen=0 oactive=false > ixl0 q6 packets=996624 qdrops=0 qlen=0 oactive=false > ixl0 q7 packets=1700511 qdrops=0 qlen=0 oactive=false > ixl1 q0 packets=1383968 qdrops=0 qlen=0 oactive=false > ixl1 q1 packets=2462384 qdrops=0 qlen=0 oactive=false > ixl1 q2 packets=1413506 qdrops=0 qlen=0 oactive=false > ixl1 q3 packets=1496771 qdrops=0 qlen=0 oactive=false > ixl1 q4 packets=1702289 qdrops=34189 qlen=1024 oactive=true > ixl1 q5 packets=2852569 qdrops=0 qlen=0 oactive=false > ixl1 q6 packets=1769569 qdrops=0 qlen=0 oactive=false > ixl1 q7 packets=3165406 qdrops=0 qlen=0 oactive=false > > --- t10 --- > ixl0 q0 packets=1621469 qdrops=0 qlen=0 oactive=false > ixl0 q1 packets=1345917 qdrops=0 qlen=0 oactive=false > ixl0 q2 packets=1919066 qdrops=0 qlen=0 oactive=false > ixl0 q3 packets=2267470 qdrops=0 qlen=0 oactive=false > ixl0 q4 packets=1866928 qdrops=0 qlen=0 oactive=false > ixl0 q5 packets=3884631 qdrops=0 qlen=0 oactive=false > ixl0 q6 packets=1011239 qdrops=0 qlen=0 oactive=false > ixl0 q7 packets=1711710 qdrops=0 qlen=0 oactive=false > ixl1 q0 packets=1401927 qdrops=0 qlen=0 oactive=false > ixl1 q1 packets=2479432 qdrops=0 qlen=0 oactive=false > ixl1 q2 packets=1419343 qdrops=0 qlen=0 oactive=false > ixl1 q3 packets=1503891 qdrops=0 qlen=0 oactive=false > ixl1 q4 packets=1702302 qdrops=35517 qlen=1024 oactive=true > ixl1 q5 packets=2860654 qdrops=0 qlen=0 oactive=false > ixl1 q6 packets=1819686 qdrops=0 qlen=0 oactive=false > ixl1 q7 packets=3183726 qdrops=0 qlen=0 oactive=false > > Hardware TX counters (selected lines): > --- t0 --- > ixl0: > tx link xon: 0 packets > tx link xoff: 0 packets > tx link down: 0 packets > tx errs: 0 packets > tx discards: 0 packets > ixl1: > tx link xon: 0 packets > tx link xoff: 0 packets > tx link down: 0 packets > tx errs: 0 packets > tx discards: 0 packets > --- t10 --- > ixl0: > tx link xon: 0 packets > tx link xoff: 0 packets > tx link down: 0 packets > tx errs: 0 packets > tx discards: 0 packets > ixl1: > tx link xon: 0 packets > tx link xoff: 0 packets > tx link down: 0 packets > tx errs: 0 packets > tx discards: 0 packets > > 8. ifconfig ixl - link state > ============================ > > --- t0 --- > ixl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 9000 > trunk: trunkdev aggr2 > media: Ethernet autoselect (25GbaseCR full-duplex) > status: active > ixl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 9000 > trunk: trunkdev aggr2 > media: Ethernet autoselect (25GbaseCR full-duplex) > status: active > > --- t10 --- > ixl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 9000 > trunk: trunkdev aggr2 > media: Ethernet autoselect (25GbaseCR full-duplex) > status: active > ixl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 9000 > trunk: trunkdev aggr2 > media: Ethernet autoselect (25GbaseCR full-duplex) > status: active > > > 9. vmstat -i - affected queue interrupt > ======================================= > > --- t0 --- > irq116/ixl1:4 833294 686 > --- t10 --- > irq116/ixl1:4 836707 684 > > The interrupt count increased from 833294 to 836707 > (+3413), but this does not establish whether TX > completions were progressing correctly. > > 10. Operational plan > ==================== > > Once diagnostic collection is complete, I will migrate > the production VLANs away from the ixl interfaces to > other network interfaces, then reboot the affected router > to restore a reliable hot-standby configuration. > > The XXV710 interfaces will no longer carry production > traffic. > > I have retained the complete original diagnostics > collected during the stall. The excerpts above focus > on the relevant driver and TX queue state. > > Please let me know if any additional driver state would > be useful for investigating this. > > Thanks, >
