Thanks for looking at this.
The machines are routers/firewalls running PF. There is no bridging
involved.
The ixl(4) interfaces are the two physical members of an aggr(4)
interface using LACP. The aggregation also provides physical path
redundancy towards two upstream switching planes.
aggr2 is the parent interface for the vlan(4) interfaces. Layer 3
addresses are configured on the VLAN/CARP interfaces, not directly on
the ixl interfaces or on aggr2.
Regarding MTU 1500, there is an important detail which I omitted from
the original report.
The problem was initially observed while the ixl interfaces were still
configured with MTU 1500. During the investigation we noticed that
these links were intended to use MTU 9000 and changed the configuration
accordingly. The TX queue stall continued to occur with MTU 9000.
This was not a controlled 1500-vs-9000 A/B test, so I cannot say whether
the MTU affects the frequency of the problem. I can only say that the
problem was observed both before and after the change to MTU 9000.
The interface configuration after that change was:
/etc/hostname.ixl0:
up mtu 9000
/etc/hostname.ixl1:
up mtu 9000
/etc/hostname.aggr2:
lladdr <REDACTED>
trunkproto lacp trunkport ixl0 trunkport ixl1 up mtu 9000
The captured aggr2 state was:
aggr2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 9000
lladdr <REDACTED>
index 13 priority 0 llprio 7
trunk: trunkproto lacp
trunk id: [(8000,<REDACTED>,000D,0000,0000),
(8000,<REDACTED>,001E,0000,0000)]
ixl0 lacp actor system pri 0x8000 mac <REDACTED>,
key 0xd, port pri 0x8000 number 0x1
ixl0 lacp actor state
activity,aggregation,sync,collecting,distributing
ixl0 lacp partner system pri 0x8000 mac <REDACTED>,
key 0x1e, port pri 0x8000 number 0x1e
ixl0 lacp partner state
activity,aggregation,sync,collecting,distributing
ixl0 port active,collecting,distributing
ixl1 lacp actor system pri 0x8000 mac <REDACTED>,
key 0xd, port pri 0x8000 number 0x2
ixl1 lacp actor state
activity,aggregation,sync,collecting,distributing
ixl1 lacp partner system pri 0x8000 mac <REDACTED>,
key 0x1e, port pri 0x8000 number 0x1e
ixl1 lacp partner state
activity,aggregation,sync,collecting,distributing
ixl1 port active,collecting,distributing
groups: aggr
media: Ethernet autoselect
status: active
Unfortunately, I cannot provide complete output from:
dmesg
netstat -s
kstat ixl0:::
kstat ixl1:::
ifconfig ixl
while the error is happening.
The XXV710 adapters were removed from the production firewalls because
of the packet loss, and I did not save complete output from all of
these commands while the failure was active.
The per-queue kstat data and interrupt/counter samples in the original
report are the data I still have from the systems while the failure was
actually occurring.
If further testing becomes necessary, we may be able to arrange a
dedicated test VLAN with one of the XXV710 adapters, but this is not
set up at the moment.
Thanks.
