Reforwarding this from an alternate address as the original never made it to the list. Sorry if the formatting got messed up.
Kyle Merchant ________________________________ From: [email protected] <[email protected]> Sent: Wednesday, February 12, 2025 23:04 To: [email protected] <[email protected]> Cc: [email protected] <[email protected]> Subject: mvpp on CN9130 Clearfog Base stuck on OACTIVE >Synopsis: mvpp lan side stuck on OACTIVE during periods of traffic >Category: kernel aarch64 >Environment: System : OpenBSD 7.6 Details: OpenBSD 7.6-current (GENERIC.MP) #303: Fri Feb 7 13:52:28 MST 2025 [email protected]:/usr/src/sys/arch/arm64/compile/GENERIC.MP Architecture: OpenBSD.arm64 Machine : Solidrun CN9130 Clearfog Base >Description: Using the clearfog as a router and I keep having the lan side port intermittently get stuck OACTIVE on the Tx queue. Queue isn't full at the time it gets stuck but quickly fills after. I have tried both the ethernet ports and sfp port in varying combination. Always lan side gets stuck. I have included below netstat -m, kstat, and pfctl -s info. This is a dual stack network. I have queues setup in my pf.conf as it seems to at least minimize the occurences. >How-To-Repeat: Fast way to trigger it is to run a speed test like speed.cloudflare.com or similar. Typically gets stuck half way through the test during the initiation of 10 MB upload. Packet loss is 0% up to that point. Otherwise I have not narrowed down what triggers it. Longest I have gone is 3 days without it happening. This also occurs using 7.6-stable. >Fix: The bandaid right now is pf queues, which extended it from a daily occurence to every 3 days. ifconfig down / up clears the OACTIVE when it gets stuck on. dmesg: OpenBSD 7.6-current (GENERIC.MP) #303: Fri Feb 7 13:52:28 MST 2025 [email protected]:/usr/src/sys/arch/arm64/compile/GENERIC.MP real mem = 4264022016 (4066MB) avail mem = 4046548992 (3859MB) random: good seed from bootblocks mainbus0 at root: SolidRun CN9130 Clearfog Base psci0 at mainbus0: PSCI 1.1, SMCCC 1.2, SYSTEM_SUSPEND efi0 at mainbus0: UEFI 2.8 efi0: Das U-Boot rev 0x20191000 smbios0 at efi0: SMBIOS 3.2 smbios0: vendor U-Boot version "2019.10-10.23.01-16008-gab5495dda2" date 01/19/2025 cpu0 at mainbus0 mpidr 0: ARM Cortex-A72 r0p3 cpu0: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu0: 512KB 64b/line 16-way L2 cache cpu0: CRC32,SHA2,SHA1,AES+PMULL,ASID16 cpu1 at mainbus0 mpidr 1: ARM Cortex-A72 r0p3 cpu1: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu1: 512KB 64b/line 16-way L2 cache cpu2 at mainbus0 mpidr 100: ARM Cortex-A72 r0p3 cpu2: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu2: 512KB 64b/line 16-way L2 cache cpu3 at mainbus0 mpidr 101: ARM Cortex-A72 r0p3 cpu3: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu3: 512KB 64b/line 16-way L2 cache "psci-area" at mainbus0 not configured apm0 at mainbus0 simplebus0 at mainbus0: "ap807" simplebus1 at simplebus0: "config-space" ampintc0 at simplebus1 nirq 352, ncpu 4 ipi: 0, 1, 2: "interrupt-controller" ampintcmsi0 at ampintc0: nspi 32 ampintcmsi1 at ampintc0: nspi 32 ampintcmsi2 at ampintc0: nspi 32 ampintcmsi3 at ampintc0: nspi 32 mvgicp0 at simplebus1 syscon0 at simplebus1: "system-controller" mvpinctrl0 at syscon0 mvclock0 at syscon0 mvgpio0 at syscon0 syscon1 at simplebus1: "system-controller" "thermal-sensor" at syscon1 not configured "clock-cpu" at syscon1 not configured agtimer0 at simplebus1: 25000 kHz "pmu" at simplebus1 not configured "odmi" at simplebus1 not configured "interrupt-controller" at simplebus1 not configured "interrupt-controller" at simplebus1 not configured "xor" at simplebus1 not configured "xor" at simplebus1 not configured "xor" at simplebus1 not configured "xor" at simplebus1 not configured "uio-xor0" at simplebus1 not configured "uio-xor1" at simplebus1 not configured "uio-xor2" at simplebus1 not configured "uio-xor3" at simplebus1 not configured com0 at simplebus1: dw16550 com0: console "watchdog" at simplebus1 not configured sdhc0 at simplebus1 sdhc0: SDHC 3.00, 400 MHz base clock sdmmc0 at sdhc0: 8-bit, sd high-speed, mmc high-speed, ddr52, dma "musdk_cma" at simplebus1 not configured simplebus2 at mainbus0: "cp0" simplebus3 at simplebus2: "config-space" mvicu0 at simplebus3 syscon2 at simplebus3: "system-controller" mvclock1 at syscon2 mvpinctrl1 at syscon2 mvgpio1 at syscon2 mvgpio2 at syscon2 syscon3 at simplebus3: "system-controller" mvtemp0 at syscon3 sxitwi0 at simplebus3 iic0 at sxitwi0 "atmel,24c02" at iic0 addr 0x53 not configured pcagpio0 at iic0 addr 0x20 "microchip,mcp3021" at iic0 addr 0x4c not configured "atmel,24c02" at iic0 addr 0x52 not configured sxitwi1 at simplebus3 iic1 at sxitwi1 mvppc0 at simplebus3 "uio_pp_0" at simplebus3 not configured "phy" at simplebus3 not configured mvmdio0 at simplebus3: "mdio" mvrtc0 at simplebus3 "pcie-mac-reset" at simplebus3 not configured "utmi" at simplebus3 not configured xhci0 at simplebus3, xHCI 1.0 usb0 at xhci0: USB revision 3.0 uhub0 at usb0 configuration 1 interface 0 "Generic xHCI root hub" rev 3.00/1.00 addr 1 xhci1 at simplebus3, xHCI 1.0 usb1 at xhci1: USB revision 3.0 uhub1 at usb1 configuration 1 interface 0 "Generic xHCI root hub" rev 3.00/1.00 addr 1 ahci0 at simplebus3: AHCI 1.0 scsibus0 at ahci0: 32 targets "xor" at simplebus3 not configured "xor" at simplebus3 not configured "cp0_uio_xor0" at simplebus3 not configured "cp0_uio_xor1" at simplebus3 not configured "spi" at simplebus3 not configured com1 at simplebus3: dw16550 mvrng0 at simplebus3 sdhc1 at simplebus3 sdhc1: SDHC 3.00, 400 MHz base clock sdmmc1 at sdhc1: 4-bit, sd high-speed, mmc high-speed, dma "crypto" at simplebus3 not configured "uio_sam" at simplebus3 not configured mvpp0 at mvppc0: address d0:63:b4:XX:XX:XX mvpp1 at mvppc0: address d0:63:b4:XX:XX:XX eephy0 at mvpp1 phy 1: 88E1512, rev. 1 mvpp2 at mvppc0: address d0:63:b4:XX:XX:XX eephy1 at mvpp2 phy 0: 88E1512, rev. 1 dwpcie0 at simplebus2 dwpcie0: can't initialize hardware "regulator-1-8" at mainbus0 not configured "regulator-vhv-1-8" at mainbus0 not configured "regulator-usb3-vbus0" at mainbus0 not configured sfp0 at mainbus0 gpiokeys0 at mainbus0: "Rear Button" "rfkill-m2-gnss" at mainbus0 not configured "rfkill-m2-wwan" at mainbus0 not configured scsibus1 at sdmmc0: 2 targets, initiator 0 sd0 at scsibus1 targ 1 lun 0: <Samsung, 8GTF4R, 0000> removable sd0: 7456MB, 512 bytes/sector, 15269888 sectors scsibus2 at sdmmc1: 2 targets, initiator 0 sd1 at scsibus2 targ 1 lun 0: <Sandisk, SC16G, 0080> removable sd1: 15193MB, 512 bytes/sector, 31116288 sectors vscsi0 at root scsibus3 at vscsi0: 256 targets softraid0 at root scsibus4 at softraid0: 256 targets root on sd1a (d8a0bcb7c6f9b7b6.a) swap on sd1b dump on sd1b usbdevs: Controller /dev/usb0: addr 01: 0000:0000 Generic, xHCI root hub super speed, self powered, config 1, rev 1.00 driver: uhub0 Controller /dev/usb1: addr 01: 0000:0000 Generic, xHCI root hub super speed, self powered, config 1, rev 1.00 driver: uhub1 beaker$ netstat -m 1766 mbufs in use: 1755 mbufs allocated to data 1 mbuf allocated to packet headers 10 mbufs allocated to socket names and addresses 802/1152 mbuf 2048 byte clusters in use (current/peak) 0/0 mbuf 2112 byte clusters in use (current/peak) 0/24 mbuf 4096 byte clusters in use (current/peak) 0/8 mbuf 8192 byte clusters in use (current/peak) 0/0 mbuf 9216 byte clusters in use (current/peak) 0/0 mbuf 12288 byte clusters in use (current/peak) 0/0 mbuf 16384 byte clusters in use (current/peak) 0/0 mbuf 65536 byte clusters in use (current/peak) 2496/2928/131072 Kbytes allocated to network (current/peak/max) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines 0 defrag mbuf allocation 124524 prepend mbuf allocation 0 pullup mbuf allocation 0 pullup memory copy 0 pulldown mbuf allocation 0 pulldown memory copy beaker$ systat -b mbufs 2 users Load 0.00 0.00 0.00 beaker.hensen.home. 21:32:08 IFACE RING LIVELOCKS SIZE ALIVE LWM HWM CWM System mbufs 0 256 1761 116 mcl2k 2048 790 144 mcl4k 4096 0 3 mcl8k 8192 0 1 lo0 mvpp0 mvpp1 mvpp2 enc0 pflog0 beaker$ kstat (only showing mvpp1 and mvpp2) mvpp1:0:rxq:0 packets: 24032476 packets bytes: 27277348845 bytes fdrops: 83 packets qdrops: 0 packets errors: 0 packets qlen: 0 packets enqueues: 8085391 dequeues: 8075293 mvpp1:0:txq:0 packets: 10805241 packets bytes: 5928636020 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 511 packets oactive: false oactives: 34 mvpp2:0:rxq:0 packets: 10786275 packets bytes: 5913729610 bytes fdrops: 743 packets qdrops: 0 packets errors: 0 packets qlen: 0 packets enqueues: 6090474 dequeues: 6089608 mvpp2:0:txq:0 packets: 23964924 packets bytes: 27198133431 bytes qdrops: 6965 packets errors: 0 packets qlen: 0 packets maxqlen: 511 packets oactive: false oactives: 2 INFO: Status: Enabled for 3 days 08:24:36 Debug: err Interface Stats for mvpp1 IPv4 IPv6 Bytes In 11990125287 14603118594 Bytes Out 3320805820 2407858299 Packets In Passed 11181536 12406543 Blocked 119190 714 Packets Out Passed 6289851 4240975 Blocked 33416 42 State Table Total Rate current entries 274 half-open tcp 0 searches 68368250 236.2/s inserts 472450 1.6/s removals 472176 1.6/s Counters match 700336 2.4/s bad-offset 0 0.0/s fragment 0 0.0/s short 72797 0.3/s normalize 7 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 3 0.0/s proto-cksum 33 0.0/s state-mismatch 241 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s translate 0 0.0/s no-route 0 0.0/s # # See pf.conf(5) and /etc/examples/pf.conf # # Set some macros for the interfaces int_if = "mvpp2" ext_if = "mvpp1" # Table of addresses we shouldn't be seeing from the internet. table <aliens> { 0.0.0.0/8 10.0.0.0/8 127.0.0.0/8 169.254.0.0/16 \ 172.16.0.0/12 192.0.0.0/24 192.0.2.0/24 224.0.0.0/3 \ 192.168.0.0/16 198.18.0.0/15 198.51.100.0/24 \ 203.0.113.0/24 ::1/128 fec0::/10 2002::/24 \ 2002:7f00::/24 2002:e000::/20 2002:ff00::/24 \ ::ffff:0:0/96 fd00::/8 } # Set some policies set block-policy return set loginterface $ext_if set skip on lo # Queue to limit to reality of its connection queue inq on $int_if bandwidth 900M max 950M flows 2048 quantum 1280 \ qlimit 255 default queue ouq on $ext_if bandwidth 780M max 825M flows 2048 quantum 1280 \ qlimit 255 default # Screen quickly obvious spoofed packets antispoof quick for { ($int_if) ($ext_if) } block drop in quick on $ext_if from <aliens> to any block out log quick on $ext_if from any to <aliens> # Port build user does not need network block out log quick proto {tcp udp} user _pbuild # NAT for the gateway match out on $ext_if inet from !($ext_if:network) to any nat-to ($ext_if:0) # Rate limit icmp6 packets match proto { icmp6 icmp } all max-pkt-rate 100/1 # Default block rule block all # we aren't blocking out pass out pass in on $int_if # Rules for NDP pass inet6 proto icmp6 all icmp6-type neighbradv no state pass in inet6 proto icmp6 all icmp6-type \ { neighbrsol routeradv } keep state ( if-bound ) # Rules for incoming echo requests pass in on $ext_if inet6 proto icmp6 from any to \ { ($ext_if) ($int_if:network) } icmp6-type echoreq max-pkt-rate 100/10 pass in on $ext_if inet proto icmp from any to ($ext_if) \ icmp-type echoreq keep state ( if-bound ) max-pkt-rate 100/10 # Rules for DHCPv6 pass in on $ext_if inet6 proto udp from any port dhcpv6-server \ to ($ext_if:0) port dhcpv6-client keep state ( if-bound ) mvpp1: flags=a48843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6TEMP,AUTOCONF6,AUTOCONF4> mtu 1500 lladdr d0:63:b4:xx:xx:xx index 2 priority 0 llprio 3 groups: egress media: Ethernet autoselect (1000baseT full-duplex,master) status: active mvpp2: flags=248c43<UP,BROADCAST,RUNNING,OACTIVE,SIMPLEX,MULTICAST,AUTOCONF6TEMP,AUTOCONF6> mtu 1500 lladdr d0:63:b4:xx:xx:xx index 3 priority 0 llprio 3 media: Ethernet autoselect (1000baseT full-duplex) status: active
