hrm, i thought we'd fixed this.
On Tue, Oct 06, 2026 at 04:50:16PM +0200, Paul de Weerd wrote:
> Hi,
>
> On my machine (dmesg below) I have an aq that is part of a veb. There
> usually is no cable attached .. the idea is that I can sometimes plug
> in a cable and then use my machine to provide connectivity to some
> device I'm testing (usually tcpdump'ing).
>
> [weerd@pom] $ cat /etc/hostname.aq0
> up
> [weerd@pom] $ cat /etc/hostname.veb0
> add vlan1
> add aq0
> add vport0
> up
> [weerd@pom] $ cat /etc/hostname.vlan1
> vnetid 1 parent igc0
> up
>
> The vport0 has an IP address for local use. vlan1 is a sub
> interface of igc0 which provides the actual connectivity.
>
> Observe how, in this configuration, mbufs are continously growing:
>
> [weerd@pom] $ while echo `date +%T` `netstat -m | head -n1`; do sleep 10; done
> 16:09:20 739 mbufs in use:
> 16:09:30 756 mbufs in use:
> 16:09:40 779 mbufs in use:
> 16:09:50 795 mbufs in use:
> 16:10:00 810 mbufs in use:
> 16:10:10 843 mbufs in use:
> 16:10:20 858 mbufs in use:
> 16:10:30 878 mbufs in use:
> 16:10:40 895 mbufs in use:
> 16:10:50 915 mbufs in use:
> 16:11:00 934 mbufs in use:
> 16:11:10 956 mbufs in use:
> 16:11:20 971 mbufs in use:
> 16:11:30 985 mbufs in use:
> 16:11:40 1001 mbufs in use:
> ...
> 16:37:22 4210 mbufs in use:
>
> Yesterday (before the system slowed to a crawl and stopped
> responding), this was at "266504 mbufs in use". I was surprised it
> could use that many mbufs - is there no cap to that?
yes, there is, which is why the system feels like it slowed down.
the amount of memory that mbufs can use (rather than the number of
mbufs) is limited by the kern.maxclusters sysctl. as the leak approaches
that limit, you're effectively restricting the network stack to a small
number of mbufs. instead of being able to run the network stack with
dozens or hundreds of mbufs, it ends up only being able to push a
handful through at a time, which is why it feels slow.
> I think those mbufs correspond to these packets in aq's transmit
> queues:
it's hard to argue with these numbers :)
>
> [weerd@pom] $ kstat -w 10 | awk '/.*:.*:.*/ {R=0} /^aq0:0:txq/ {R=$1}
> /packets:/ {if (R != 0) {T=strftime("%T"); print T, R, $0}}'
> 16:15:58 aq0:0:txq:0 packets: 3929 packets
> 16:15:58 aq0:0:txq:1 packets: 206 packets
> 16:15:58 aq0:0:txq:2 packets: 1987 packets
> 16:15:58 aq0:0:txq:3 packets: 100 packets
> 16:15:58 aq0:0:txq:4 packets: 102 packets
> 16:15:58 aq0:0:txq:5 packets: 227 packets
> 16:15:58 aq0:0:txq:6 packets: 408 packets
> 16:15:58 aq0:0:txq:7 packets: 98 packets
> 16:16:08 aq0:0:txq:0 packets: 3940 packets
> 16:16:08 aq0:0:txq:1 packets: 211 packets
> 16:16:08 aq0:0:txq:2 packets: 1992 packets
> 16:16:08 aq0:0:txq:3 packets: 100 packets
> 16:16:08 aq0:0:txq:4 packets: 102 packets
> 16:16:08 aq0:0:txq:5 packets: 228 packets
> 16:16:08 aq0:0:txq:6 packets: 422 packets
> 16:16:08 aq0:0:txq:7 packets: 98 packets
> 16:16:18 aq0:0:txq:0 packets: 3954 packets
> 16:16:18 aq0:0:txq:1 packets: 212 packets
> 16:16:18 aq0:0:txq:2 packets: 1998 packets
> 16:16:18 aq0:0:txq:3 packets: 100 packets
> 16:16:18 aq0:0:txq:4 packets: 102 packets
> 16:16:18 aq0:0:txq:5 packets: 228 packets
> 16:16:18 aq0:0:txq:6 packets: 427 packets
> 16:16:18 aq0:0:txq:7 packets: 98 packets
> ...
> 16:37:58 aq0:0:txq:0 packets: 5529 packets
> 16:37:58 aq0:0:txq:1 packets: 297 packets
> 16:37:58 aq0:0:txq:2 packets: 2739 packets
> 16:37:58 aq0:0:txq:3 packets: 115 packets
> 16:37:58 aq0:0:txq:4 packets: 128 packets
> 16:37:58 aq0:0:txq:5 packets: 303 packets
> 16:37:58 aq0:0:txq:6 packets: 583 packets
> 16:37:58 aq0:0:txq:7 packets: 133 packets
>
>
> So while the interface is up, there's no cable connected, ifconfig
> says
>
> aq0: flags=4008b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
> mtu 1500
> lladdr __:__:__:__:__:__
> index 2 priority 0 llprio 3
> media: Ethernet autoselect
> status: no carrier
>
> it is getting packets from veb0 and storing them indefenitely.
> Apparently, something thinks the link-state is up. I found this in
> net/if.h
>
> #define LINK_STATE_IS_UP(_s) \
> ((_s) >= LINK_STATE_UP || (_s) == LINK_STATE_UNKNOWN)
>
> So if the link state is UNKNOWN, the system considers it UP, but then
> aq_update_link_status doesn't seem to be called while bringing up the
> interface in aq_up in if_aq_pci.c, resulting in the UNKNOWN link state
> (resulting in these packets getting buffered).
>
> So, four questions here...
> 1. Should those mbufs keep growing without bound?
well, no. see above about kern.maxclusters.
> 2. Should an interface in UNKNOWN link state still receive packets?
> 3. Should an interface in UNKNOWN link state still queue packets?
this depends on the driver. what does link state mean for something like
gif(4)?
technically the interface link state is informational, ie, it's
interesting for the operator of the system, but it's not used by the
network stack. the stack only really cares if the interface is up or not
(ie, ifconfig aq0 up), but it's up to the driver to do the right thing
with a packet for the underlying transport, which is the hardware and
ethernet link in aq(4)s.
in my mind it's reasonable for a driver to queue packets when the link
is down, the hardware should just complete tx processing with errors.
aq hardware seems to be waiting for the link to appear to run the
ring and make progress though.
> 4. Should aq try to set its link state while bringing it up?
yes. aq(4) should initialise the link state to DOWN since it's "load
bearing" with LINK_STATE_IS_UP being used at the top of aq_start().
my aq(4) is busy at the moment, so i can't test the diff below. i'm
specifically interested in what the aq(4) link state looks like
when a machine is booted from cold both with and without link.
`systat if` is probably the easiest way to see it. with the current
driver i'm guessing that you see "aq0 up" if the machine is
booted without a working ethernet cable plugged in. after this diff
it should be "up:D". if you boot with a working ethernet cable
plugged in it should be "up:U".
>
> Thanks,
thanks for the bug report, it was very thorough.
Index: if_aq_pci.c
===================================================================
RCS file: /cvs/src/sys/dev/pci/if_aq_pci.c,v
diff -u -p -r1.38 if_aq_pci.c
--- if_aq_pci.c 25 Aug 2026 01:18:03 -0000 1.38
+++ if_aq_pci.c 7 Oct 2026 00:54:40 -0000
@@ -1384,6 +1384,7 @@ aq_attach(struct device *parent, struct
ifp->if_qstart = aq_start;
ifp->if_watchdog = aq_watchdog;
ifp->if_hardmtu = 9000;
+ ifp->if_link_state = LINK_STATE_DOWN;
ifp->if_capabilities = IFCAP_VLAN_MTU | IFCAP_CSUM_IPv4 |
IFCAP_CSUM_UDPv4 | IFCAP_CSUM_UDPv6 | IFCAP_CSUM_TCPv4 |
IFCAP_CSUM_TCPv6;
