Synopsis: ixl(4): XXV710 TX queue stalls; regression after OpenBSD 7.7 Category: kernel
Environment: OpenBSD 7.9 GENERIC.MP amd64, with 7.9 syspatches. AMD EPYC 7F32 8-Core Processor. Intel XXV710 dual-port SFP28 adapter, ixl(4), 8 queues per port. The two ixl interfaces were members of an LACP aggregate, MTU 9000. Description: We are observing persistent TX queue stalls with Intel XXV710 adapters using ixl(4) on OpenBSD 7.9. This is a regression introduced after OpenBSD 7.7. The same adapters, on the same firewalls and under the same production workload, operated on OpenBSD 7.7 without this problem. The production workload did not change between the known-good OpenBSD 7.7 deployment and the affected deployment. The same failure was reproducible on a second, identically configured OpenBSD firewall acting as the PF peer, with the same XXV710 hardware. The systems were upgraded from OpenBSD 7.7 through 7.8 to 7.9. We did not run 7.8 long enough to determine whether the regression was introduced in 7.8 or in 7.9. Therefore: OpenBSD 7.7: not affected OpenBSD 7.8: unknown OpenBSD 7.9: affected During the observed failures, both LACP members reported activity, aggregation, sync, collecting and distributing. The failure affects one TX queue while the other TX queues continue operating normally. One captured failure affected TX queue 4: packets: 6711341 bytes: 7917369244 qdrops: 360808 errors: 0 qlen: 1024 maxqlen: 1024 oactive: true oactives: 1 Ten seconds later: packets: 6711341 bytes: 7917369244 qdrops: 361175 errors: 0 qlen: 1024 maxqlen: 1024 oactive: true oactives: 1 Delta over this 10 second interval: packets: +0 bytes: +0 qdrops: +367 qlen: remained 1024/1024 The TX packet and byte counters therefore made no progress during this interval while additional packets continued to be dropped. The other TX queues on the interface had qlen 0 and no significant drops at the time. Later, the same stalled queue was captured with oactive false: packets: 6711341 bytes: 7917369244 qdrops: 377543 errors: 0 qlen: 1024 maxqlen: 1024 oactive: false oactives: 1 Ten seconds later: packets: 6711341 bytes: 7917369244 qdrops: 378131 errors: 0 qlen: 1024 maxqlen: 1024 oactive: false oactives: 1 Delta over this second 10 second interval: packets: +0 bytes: +0 qdrops: +588 qlen: remained 1024/1024 oactive: remained false The MSI-X interrupt associated with the affected queue continued to increment while the queue was stalled. During a 10 second interval: queue 4 MSI-X interrupt: +1910 The other ixl interrupt counters also continued to increment. No corresponding hardware TX errors, TX discards or link flow-control events were observed: VSI tx errs: 0 VSI tx discards: 0 port tx link xon: 0 port tx link xoff: 0 port tx link down: 0 TCP TSO was tested enabled and disabled. With TSO enabled, over 10 seconds: output packets: +50826 output drops: +490 With TSO disabled, over 10 seconds: output packets: +53721 output drops: +406 Disabling TSO did not eliminate the problem. The failure is not tied to one physical port. When the physical link on the affected ixl interface was disconnected, traffic moved to the other LACP member and the problem subsequently occurred on the other ixl interface. The adapter firmware was updated during the investigation. The problem was observed with: FW 9.854.78653 API 1.15 After the firmware update, the interfaces reported: FW 9.856.79020 API 1.15 The problem was observed again with the newer firmware. Other NICs used in the same systems, including bnxt(4) and mcx(4), have not exhibited this TX queue stall. The affected XXV710 interfaces have now been removed from production service because of the packet loss caused by this problem. We therefore cannot perform additional reproduction tests on the production firewalls. The observations above were collected while the failure was active. How-To-Repeat: The failure occurred under normal production traffic on OpenBSD 7.9. There is no known deterministic reproduction procedure. The failure was reproduced on two identically configured PF peers. The affected adapters have now been removed from production service, so further reproduction testing is currently not possible. Summary: Known good: OpenBSD 7.7 Unknown: OpenBSD 7.8 Known affected: OpenBSD 7.9 Hardware: Intel XXV710 SFP28, ixl(4), 8 queues per port Firmware versions with which the problem was observed: 9.854.78653 API 1.15 9.856.79020 API 1.15 Observed failure: - one TX queue stops making TX progress - TX packet and byte counters remain unchanged - qlen remains 1024/1024 - qdrops continuously increase - the associated MSI-X interrupt continues to increment - no TX errors or TX discards are reported - the stalled queue was also observed with oactive false - the problem can occur on either LACP member - the problem was reproduced on two identically configured PF peers Fix: Unknown.
