any difference if you connect via a switch as a test?
On 2026/10/14 08:58, Lars Reunamo wrote:
> >Synopsis: bse(4) receive throughput around 0.3 Mbit/s with Starlink Mini
> >Category: kernel
> >Environment:
> System : OpenBSD 7.9
> Details : OpenBSD 7.9 (GENERIC.MP) #16: Mon Sep 28 02:46:43 MDT 2026
>
> [email protected]:/usr/src/sys/arch/arm64/compile/GENERIC.MP
>
> Architecture: OpenBSD.arm64
> Machine : arm64
>
> Raspberry Pi 4 Model B Rev 1.5, 2 GB RAM, BCM2711/GENET v5.
> PHY: BCM54210E, rev. 2 (brgphy0).
> Boot: U-Boot 2025.07; full boot dmesg and usbdevs are below.
> Stock GENERIC.MP kernel, no local kernel or driver modifications.
> All syspatches 001 through 034 are installed. The machine was
> rebooted after installing them on 2026-10-14.
> Ethernet: bse0, MTU 1500, autoselect, 1000baseT full-duplex.
> Direct wired connection to Starlink Mini, currently using its
> public IPv4/bypass configuration and DHCP. IPv6 autoconf is active.
> curl 8.19.0 with LibreSSL 4.3.0 was used for the latest tests.
>
> >Description:
> The receive throughput of this RPi4/OpenBSD system is consistently
> around 0.2-0.3 Mbit/s when directly connected to Starlink Mini's
> Ethernet port. The link reports 1000baseT full-duplex and there
> are no interface input/output errors or collisions.
>
> This is reproducible across several download servers, IPv4 and
> IPv6, and HTTPS and SSH/WireGuard traffic. The problem predates
> the upgrade to 7.9 and persists after that upgrade, syspatching
> and rebooting. All rates below distinguish bytes/s from bits/s.
>
> Latest measurements, 2026-10-14, after syspatch and reboot:
>
> CDN IPv4, https://cdn.openbsd.org/pub/OpenBSD/7.9/arm64/bsd.rd
> HTTP 206; 375321 bytes in 15.002606 s; 25017 bytes/s.
> Same CDN IPv6:
> HTTP 206; 376900 bytes in 15.002815 s; 25121 bytes/s.
> Separate mirror, IPv4:
> https://ftp.eu.openbsd.org/pub/OpenBSD/7.9/arm64/bsd.rd
> HTTP 206; 409600 bytes in 12.010057 s; 34104 bytes/s.
>
> These are bounded range downloads. curl exit 28 is expected
> here because the deliberately short time limit expires before
> the requested 1 MiB has been transferred. The measured rates
> are 0.200-0.273 Mbit/s, not 25-34 Mbit/s.
>
> Earlier investigation on 2026-09-03/04 found:
>
> - One TCP download: about 34.2 kB/s.
> - Four simultaneous TCP downloads: about 34.7 kB/s TOTAL.
> - SSH/WireGuard data into this server: about 29.1 kB/s.
> - IPv4 about 34.7 kB/s, IPv6 about 32.8 kB/s.
> - No increase in TCP retransmissions during the controlled
> download, and no PF queue or bandwidth-shaping rules.
> - CPU mostly idle during the tests.
> - Captured large RX segments at bse0 had timestamp spacing
> around 33.554 ms, or multiples of that interval. These are
> software capture timestamps, not a measurement of the
> actual arrival time of frames on the wire.
>
> Control tests reported during that investigation:
>
> - Another computer using the same Starlink Ethernet port and
> cable achieved about 90 Mbit/s.
> - Devuan Linux on the same RPi4 achieved about 100 Mbit/s.
> - The same RPi4/OpenBSD on the house Ethernet network downloaded
> at about 6 MB/s (48 Mbit/s).
> - Starlink's own speed test reported about 186 Mbit/s down,
> 38 Mbit/s up and 44 ms latency.
> - Disabling Starlink Public IP and trying DHCP/NAT did not
> remove the slow throughput.
> - Forcing 100baseTX full-duplex improved throughput to about
> 119 kB/s, but did not restore normal performance. This was
> only a temporary test; the interface is back on autoselect.
> A duplex mismatch is a possible confounder in that test.
>
> Thus a general Starlink service bandwidth limit does not fit
> the control results. The symptom appears to depend on the
> OpenBSD GENET/PHY path and this link partner/traffic pattern.
> A specific driver or PHY cause has NOT been demonstrated.
>
> >How-To-Repeat:
> On the affected stock OpenBSD RPi4, boot with bse0 directly
> connected to Starlink Mini. Use the normal DHCP configuration:
>
> /etc/hostname.bse0:
> inet autoconf
> inet6 autoconf
>
> Verify that the link and default route have come up:
>
> ifconfig bse0
> netstat -rn -f inet
> netstat -in -I bse0
>
> A minimal current download reproducer, requiring installed curl:
>
> /usr/local/bin/curl -4 --silent --show-error --fail \
> --connect-timeout 5 --max-time 15 --max-filesize 1048576 \
> --range 0-1048575 --output /dev/null \
> --write-out 'HTTP=%{http_code} bytes=%{size_download}
> time=%{time_total}s speed=%{speed_download} B/s\n' \
> https://cdn.openbsd.org/pub/OpenBSD/7.9/arm64/bsd.rd
>
> Repeat with -6 for IPv6, and with the alternative mirror:
> https://ftp.eu.openbsd.org/pub/OpenBSD/7.9/arm64/bsd.rd
>
> The expected result, based on the same-port/same-cable control
> tests, is throughput well above 0.3 Mbit/s. On the affected
> system the command prints rates near those shown above and
> times out before receiving the full 1 MiB.
>
> >Fix:
> No verified fix or patch is available from me.
>
> A source comparison suggests a possible lead: explicit RX
> interrupt threshold and ring timeout initialization.
>
> OpenBSD sys/dev/ic/bcmgenet.c rev. 1.12 (2026-01-24) writes
> GENET_TX_DMA_MBUF_DONE_THRES but does not explicitly write the
> corresponding RX threshold or RX ring timeout. The RX register
> definitions are also absent from bcmgenetreg.h. This was checked
> against the public source on 2026-10-14; I have NOT boot-tested
> a -current kernel.
>
> NetBSD added RX/TX interrupt mitigation on 2021-03-08:
>
> https://github.com/NetBSD/src/commit/7b6ab7c6d3273ce6b0bd23b327ad32dea1358dfe
> Its genet_set_rxthresh(sc, qid, 57, 10) explicitly programs the
> RX packet threshold and timeout (requested 57 us, rounded to
> seven 8.192 us ticks).
>
> Linux bcmgenet_set_rx_coalesce() also explicitly writes both
> settings; the non-adaptive defaults are one packet and zero
> timeout. Its source describes a timeout tick of 8.192 us.
>
> The observed 33.554 ms spacing is numerically close to
> 4096 * 8.192 us = 33.554432 ms. This is a hypothesis only:
> the live RX timeout register value has not been confirmed,
> nor has a test kernel demonstrated that this initialization
> difference causes or fixes the problem.
>
> FreeBSD's current if_genet.c also lacks the corresponding
> explicit RX initialization. I have not tested FreeBSD on this
> hardware and do not claim that it fixes this symptom.
>
> Relevant public sources:
> https://github.com/openbsd/src/blob/master/sys/dev/ic/bcmgenet.c
> https://github.com/openbsd/src/blob/master/sys/dev/ic/bcmgenetreg.h
> https://github.com/NetBSD/src/blob/trunk/sys/dev/ic/bcmgenet.c
>
> https://github.com/torvalds/linux/blob/master/drivers/net/ethernet/broadcom/genet/bcmgenet.c
>
> https://github.com/freebsd/freebsd-src/blob/main/sys/arm64/broadcom/genet/if_genet.c
>
> I checked the 7.9 errata through 034 and the published changes
> toward 8.0; I found no listed fix addressing this symptom. The
> public OpenBSD bcmgenet.c still shows the initialization
> difference described above.
>
> Testing limitation: all Raspberry Pis available to me are in
> production running an EV charging/OCPP service. I cannot
> currently boot-test a custom kernel, a snapshot, or disruptive
> link/PHY changes, and have no spare RPi for such tests. I can
> provide additional read-only diagnostics where practical.
>
> I would appreciate a review of whether the RX register
> initialization deserves a change, or whether another known
> GENET/PHY issue better explains these observations.
>
> dmesg:
> OpenBSD 7.9 (GENERIC.MP) #16: Mon Sep 28 02:46:43 MDT 2026
>
> [email protected]:/usr/src/sys/arch/arm64/compile/GENERIC.MP
> real mem = 2066153472 (1970MB)
> avail mem = 1959325696 (1868MB)
> random: good seed from bootblocks
> mainbus0 at root: Raspberry Pi 4 Model B Rev 1.5
> efi0 at mainbus0: UEFI 2.11
> efi0: Das U-Boot rev 0x20250700
> smbios0 at efi0: SMBIOS 3.7.0
> smbios0: vendor U-Boot version "2025.07" date 07/01/2025
> smbios0: raspberrypi Raspberry Pi 4 Model B Rev 1.5
> cpu0 at mainbus0 mpidr 0: ARM Cortex-A72 r0p3
> cpu0: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
> cpu0: 1024KB 64b/line 16-way L2 cache
> cpu0: CRC32,ASID16
> cpu1 at mainbus0 mpidr 1: ARM Cortex-A72 r0p3
> cpu1: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
> cpu1: 1024KB 64b/line 16-way L2 cache
> cpu2 at mainbus0 mpidr 2: ARM Cortex-A72 r0p3
> cpu2: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
> cpu2: 1024KB 64b/line 16-way L2 cache
> cpu3 at mainbus0 mpidr 3: ARM Cortex-A72 r0p3
> cpu3: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
> cpu3: 1024KB 64b/line 16-way L2 cache
> "linux,cma" at mainbus0 not configured
> "nvram" at mainbus0 not configured
> "nvram" at mainbus0 not configured
> apm0 at mainbus0
> simplefb0 at mainbus0: 1920x1200, 32bpp
> wsdisplay0 at simplefb0 mux 1: console (std, vt100 emulation)
> wsdisplay0: screen 1-5 added (std, vt100 emulation)
> "system" at mainbus0 not configured
> "axi" at mainbus0 not configured
> simplebus0 at mainbus0: "soc"
> bcmclock0 at simplebus0
> bcmmbox0 at simplebus0
> bcmgpio0 at simplebus0
> bcmaux0 at simplebus0
> ampintc0 at simplebus0 nirq 256, ncpu 4 ipi 0: "interrupt-controller"
> bcmtmon0 at simplebus0
> bcmdmac0 at simplebus0: DMA0 DMA2 DMA4 DMA5 DMA6 DMA7 DMA8 DMA9 DMA10
> pluart0 at simplebus0: rev 3, 32 byte fifo
> "interrupt-controller" at simplebus0 not configured
> bcmdog0 at simplebus0
> bcmirng0 at simplebus0
> "firmware" at simplebus0 not configured
> "power" at simplebus0 not configured
> "mailbox" at simplebus0 not configured
> sdhc0 at simplebus0
> sdhc0: SDHC 3.00, 250 MHz base clock
> sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed
> "gpiomem" at simplebus0 not configured
> "fb" at simplebus0 not configured
> simplebus1 at simplebus0: "nvmem"
> "nvmem_otp" at simplebus1 not configured
> "nvmem_cust" at simplebus1 not configured
> "nvmem_priv" at simplebus1 not configured
> simplebus2 at mainbus0: "clocks"
> "clk-osc" at simplebus2 not configured
> "clk-usb" at simplebus2 not configured
> "phy" at mainbus0 not configured
> "clk-27M" at mainbus0 not configured
> "clk-108M" at mainbus0 not configured
> simplebus3 at mainbus0: "emmc2bus"
> sdhc1 at simplebus3
> sdhc1: SDHC 3.00, 100 MHz base clock
> sdmmc1 at sdhc1: 8-bit, sd high-speed, mmc high-speed, ddr52, dma
> "pmu" at mainbus0 not configured
> agtimer0 at mainbus0: 54000 kHz
> simplebus4 at mainbus0: "scb"
> bcmpcie0 at simplebus4
> pci0 at bcmpcie0
> ppb0 at pci0 dev 0 function 0 "Broadcom BCM2711" rev 0x20
> pci1 at ppb0 bus 1
> xhci0 at pci1 dev 0 function 0 "VIA VL805 xHCI" rev 0x01: intx, xHCI 1.0
> usb0 at xhci0: USB revision 3.0
> uhub0 at usb0 configuration 1 interface 0 "VIA xHCI root hub" rev 3.00/1.00
> addr 1
> bse0 at simplebus4: address 2c:cf:67:0e:24:23
> brgphy0 at bse0 phy 1: BCM54210E, rev. 2
> "codec" at simplebus4 not configured
> "dma" at simplebus4 not configured
> gpioleds0 at mainbus0: "ACT", "PWR"
> "cam_dummy_reg" at mainbus0 not configured
> "regulator-cam1" at mainbus0 not configured
> "regulator-sd-io-1v8" at mainbus0 not configured
> "regulator-sd-vcc" at mainbus0 not configured
> "fixedregulator_3v3" at mainbus0 not configured
> "fixedregulator_5v0" at mainbus0 not configured
> "zone_dma" at mainbus0 not configured
> simplebus5 at mainbus0: "v3dbus"
> scsibus0 at sdmmc1: 2 targets, initiator 0
> sd0 at scsibus0 targ 1 lun 0: <SD/MMC, USD00, 0010> removable
> sd0: 30250MB, 512 bytes/sector, 61952000 sectors
> uhub1 at uhub0 port 1 configuration 1 interface 0 "VIA Labs USB2.0 Hub" rev
> 2.10/4.21 addr 2
> bwfm0 at sdmmc0 function 1
> manufacturer 0x02d0, product 0xa9a6 at sdmmc0 function 2 not configured
> manufacturer 0x02d0, product 0xa9a6 at sdmmc0 function 3 not configured
> uhidev0 at uhub1 port 1 configuration 1 interface 0 "Microsoft Wired Keyboard
> 400" rev 1.10/1.10 addr 3
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd0 at ukbd0: console keyboard, using wsdisplay0
> vscsi0 at root
> scsibus1 at vscsi0: 256 targets
> softraid0 at root
> scsibus2 at softraid0: 256 targets
> root on sd0a (38f1aa3aa8c2cb0e.a) swap on sd0b dump on sd0b
> WARNING: CHECK AND RESET THE DATE!
> gpio0 at bcmgpio0: 58 pins
> bwfm0: address 2c:cf:67:0e:24:24
>
> usbdevs:
> Controller /dev/usb0:
> addr 01: 1106:0000 VIA, xHCI root hub
> super speed, self powered, config 1, rev 1.00
> driver: uhub0
> addr 02: 2109:3431 VIA Labs, USB2.0 Hub
> high speed, self powered, config 1, rev 4.21
> driver: uhub1
> addr 03: 045e:0752 Microsoft, Wired Keyboard 400
> low speed, power 100 mA, config 1, rev 1.10
> driver: uhidev0
>
