On Wed, Sep 30, 2026 at 02:40:32PM +0200, Network Operations wrote:
> Synopsis:
> ixl(4): XXV710 TX queue stalls; regression after OpenBSD 7.7
>
> Category:
> kernel
>
> Environment:
> OpenBSD 7.9 GENERIC.MP amd64, with 7.9 syspatches.
> AMD EPYC 7F32 8-Core Processor.
> Intel XXV710 dual-port SFP28 adapter, ixl(4), 8 queues per port.
> The two ixl interfaces were members of an LACP aggregate, MTU 9000.
Could you try it with an MTU of 1500? I noticed some jumbo problems in
the past of ixl(4) interfaces, which I could never finally reproduce and
fix.
What does the machine do?
Is it just a server for e.g. http?
It is a router/firewall?
Do do you use bridging?
I read the provided information for you below, but could you provide
some more information and full output of the following command while the
error happens?
dmesg
netstat -s
kstat ixl0:::
kstat ixl1:::
ifconfig ixl
Which interface do you use for lacp? trunk(4) or aggr(4) Could you show
their /etc/hostsname.if files and an ifconfig output of it?
Thanks for your bugreport.
jan
> Description:
>
> We are observing persistent TX queue stalls with Intel XXV710 adapters
> using ixl(4) on OpenBSD 7.9.
>
> This is a regression introduced after OpenBSD 7.7.
>
> The same adapters, on the same firewalls and under the same production
> workload, operated on OpenBSD 7.7 without this problem.
>
> The production workload did not change between the known-good OpenBSD
> 7.7 deployment and the affected deployment.
>
> The same failure was reproducible on a second, identically configured
> OpenBSD firewall acting as the PF peer, with the same XXV710 hardware.
>
> The systems were upgraded from OpenBSD 7.7 through 7.8 to 7.9.
> We did not run 7.8 long enough to determine whether the regression was
> introduced in 7.8 or in 7.9.
>
> Therefore:
>
> OpenBSD 7.7: not affected
> OpenBSD 7.8: unknown
> OpenBSD 7.9: affected
>
> During the observed failures, both LACP members reported activity,
> aggregation, sync, collecting and distributing.
>
> The failure affects one TX queue while the other TX queues continue
> operating normally.
>
> One captured failure affected TX queue 4:
>
> packets: 6711341
> bytes: 7917369244
> qdrops: 360808
> errors: 0
> qlen: 1024
> maxqlen: 1024
> oactive: true
> oactives: 1
>
> Ten seconds later:
>
> packets: 6711341
> bytes: 7917369244
> qdrops: 361175
> errors: 0
> qlen: 1024
> maxqlen: 1024
> oactive: true
> oactives: 1
>
> Delta over this 10 second interval:
>
> packets: +0
> bytes: +0
> qdrops: +367
> qlen: remained 1024/1024
>
> The TX packet and byte counters therefore made no progress during this
> interval while additional packets continued to be dropped.
>
> The other TX queues on the interface had qlen 0 and no significant
> drops at the time.
>
> Later, the same stalled queue was captured with oactive false:
>
> packets: 6711341
> bytes: 7917369244
> qdrops: 377543
> errors: 0
> qlen: 1024
> maxqlen: 1024
> oactive: false
> oactives: 1
>
> Ten seconds later:
>
> packets: 6711341
> bytes: 7917369244
> qdrops: 378131
> errors: 0
> qlen: 1024
> maxqlen: 1024
> oactive: false
> oactives: 1
>
> Delta over this second 10 second interval:
>
> packets: +0
> bytes: +0
> qdrops: +588
> qlen: remained 1024/1024
> oactive: remained false
>
> The MSI-X interrupt associated with the affected queue continued to
> increment while the queue was stalled.
>
> During a 10 second interval:
>
> queue 4 MSI-X interrupt: +1910
>
> The other ixl interrupt counters also continued to increment.
>
> No corresponding hardware TX errors, TX discards or link flow-control
> events were observed:
>
> VSI tx errs: 0
> VSI tx discards: 0
> port tx link xon: 0
> port tx link xoff: 0
> port tx link down: 0
>
> TCP TSO was tested enabled and disabled.
>
> With TSO enabled, over 10 seconds:
>
> output packets: +50826
> output drops: +490
>
> With TSO disabled, over 10 seconds:
>
> output packets: +53721
> output drops: +406
>
> Disabling TSO did not eliminate the problem.
>
> The failure is not tied to one physical port.
>
> When the physical link on the affected ixl interface was disconnected,
> traffic moved to the other LACP member and the problem subsequently
> occurred on the other ixl interface.
>
> The adapter firmware was updated during the investigation.
>
> The problem was observed with:
>
> FW 9.854.78653 API 1.15
>
> After the firmware update, the interfaces reported:
>
> FW 9.856.79020 API 1.15
>
> The problem was observed again with the newer firmware.
>
> Other NICs used in the same systems, including bnxt(4) and mcx(4),
> have not exhibited this TX queue stall.
>
> The affected XXV710 interfaces have now been removed from production
> service because of the packet loss caused by this problem.
>
> We therefore cannot perform additional reproduction tests on the
> production firewalls. The observations above were collected while the
> failure was active.
>
> How-To-Repeat:
>
> The failure occurred under normal production traffic on OpenBSD 7.9.
>
> There is no known deterministic reproduction procedure.
>
> The failure was reproduced on two identically configured PF peers.
>
> The affected adapters have now been removed from production service,
> so further reproduction testing is currently not possible.
>
> Summary:
>
> Known good: OpenBSD 7.7
> Unknown: OpenBSD 7.8
> Known affected: OpenBSD 7.9
>
> Hardware: Intel XXV710 SFP28, ixl(4), 8 queues per port
>
> Firmware versions with which the problem was observed:
> 9.854.78653 API 1.15
> 9.856.79020 API 1.15
>
> Observed failure:
> - one TX queue stops making TX progress
> - TX packet and byte counters remain unchanged
> - qlen remains 1024/1024
> - qdrops continuously increase
> - the associated MSI-X interrupt continues to increment
> - no TX errors or TX discards are reported
> - the stalled queue was also observed with oactive false
> - the problem can occur on either LACP member
> - the problem was reproduced on two identically configured PF peers
>
> Fix:
>
> Unknown.
>
