Hello, I have discovered a replicable crash in pool_do_get on an OpenBSD machine being used as a server for iperf3 tests within a private network. I first encountered on 6.5-release, then upgraded to 6.8-current to see if that would resolve it but I can still cause the panic. It appears to happen between 500 and 800 seconds into an iperf3 stress test; system uptime does not seem to be a factor. Topology of the network is as follows:
host720 (iperf3 server on openbsd 6.8) <-> L2 switch <-> L3 gateway <-> L2 switch <-> host710 (iperf3 client on openbsd 6.8) iperf3 server command: host720# iperf3 -s iperf3 client command: host710# iperf3 -t 0 -c 10.107.20.10 If any additional ddb commands need to be run besides the ones listed below I can easily replicate the crash for them; this is my first time submitting a bug report to the list and I can't sendbug(1) from this particular box. Cheers, Troy Martin kazin...@cordilon.net ---- dmesg (from serial console): OpenBSD 6.8 (GENERIC) #5: Mon Feb 22 04:04:49 MST 2021 r...@syspatch-68-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC real mem = 1056899072 (1007MB) avail mem = 1009983488 (963MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xe1000 (10 entries) bios0: vendor innotek GmbH version "VirtualBox" date 12/01/2006 bios0: innotek GmbH VirtualBox acpi0 at bios0: ACPI 4.0 acpi0: sleep states S0 S5 acpi0: tables DSDT FACP APIC SSDT acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz, 3696.50 MHz, 06-9e-0a cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SS E,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,RDRA ND,NXE,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,AVX2,INVPCID,RDSEED,CLFLUSHOPT,MD_CLEAR,L 1DF,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: CPU supports MTRRs but not enabled by BIOS cpu0: apic clock running at 1000MHz cpu0: mwait min=64, max=64 ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 20, 24 pins, remapped acpiprt0 at acpi0: bus 0 (PCI0) acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001 acpiac0 at acpi0: AC unit online acpicpu0 at acpi0: C1(@1 halt!) acpivideo0 at acpi0: GFX0 cpu0: using VERW MDS workaround (except on vmm entry) pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02 pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00 pciide0 at pci0 dev 1 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility wd0 at pciide0 channel 0 drive 0: <VBOX HARDDISK> wd0: 128-sector PIO, LBA, 16384MB, 33554432 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 atapiscsi0 at pciide0 channel 1 drive 0 scsibus1 at atapiscsi0: 2 targets cd0 at scsibus1 targ 0 lun 0: <VBOX, CD-ROM, 1.0> removable cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2 vga1 at pci0 dev 2 function 0 "InnoTek VirtualBox Graphics Adapter" rev 0x00 wsdisplay at vga1 not configured em0 at pci0 dev 3 function 0 "Intel 82540EM" rev 0x02: apic 1 int 19, address 08:00:27:46:e1:02 "InnoTek VirtualBox Guest Service" rev 0x00 at pci0 dev 4 function 0 not configured auich0 at pci0 dev 5 function 0 "Intel 82801AA AC97" rev 0x01: apic 1 int 21, ICH ac97: codec id 0x83847600 (SigmaTel STAC9700) audio0 at auich0 ohci0 at pci0 dev 6 function 0 "Apple Intrepid USB" rev 0x00: apic 1 int 22, version 1.0 piixpm0 at pci0 dev 7 function 0 "Intel 82371AB Power" rev 0x08: apic 1 int 23 iic0 at piixpm0 em1 at pci0 dev 8 function 0 "Intel 82540EM" rev 0x02: apic 1 int 16, address 08:00:27:43:c6:70 ehci0 at pci0 dev 11 function 0 "Intel 82801FB USB" rev 0x00: apic 1 int 19 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com0: console pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0 mux 1 pms0 at pckbc0 (aux slot) wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 usb1 at ohci0: USB revision 1.0 uhub1 at usb1 configuration 1 interface 0 "Apple OHCI root hub" rev 1.00/1.00 addr 1 vscsi0 at root scsibus2 at vscsi0: 256 targets softraid0 at root scsibus3 at softraid0: 256 targets root on wd0a (891d4cfcef7cbcc1.a) swap on wd0b dump on wd0b WARNING: / was not properly unmounted Automatic boot in progress: starting file system checks. /dev/wd0a (891d4cfcef7cbcc1.a): 1805 files, 47809 used, 172894 free (102 frags, 21599 blocks, 0.0% fragmentation) /dev/wd0a (891d4cfcef7cbcc1.a): MARKING FILE SYSTEM CLEAN /dev/wd0k (891d4cfcef7cbcc1.k): 3 files, 4 used, 1536851 free (19 frags, 192104 blocks, 0.0% fragmentation) /dev/wd0k (891d4cfcef7cbcc1.k): MARKING FILE SYSTEM CLEAN /dev/wd0d (891d4cfcef7cbcc1.d): UNREF FILE I=3 OWNER=root MODE=100600 /dev/wd0d: SIZE=131072 MTIME=Apr 7 23:26 2021 (CLEARED) /dev/wd0d (891d4cfcef7cbcc1.d): FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) /dev/wd0d (891d4cfcef7cbcc1.d): SUMMARY INFORMATION BAD (SALVAGED) /dev/wd0d (891d4cfcef7cbcc1.d): BLK(S) MISSING IN BIT MAPS (SALVAGED) /dev/wd0d (891d4cfcef7cbcc1.d): 6 files, 5 used, 292930 free (34 frags, 36612 blocks, 0.0% fragmentation) /dev/wd0d (891d4cfcef7cbcc1.d): MARKING FILE SYSTEM CLEAN /dev/wd0f (891d4cfcef7cbcc1.f): 17332 files, 591646 used, 209217 free (1225 frags, 25999 blocks, 0.2% fragmentation) /dev/wd0f (891d4cfcef7cbcc1.f): MARKING FILE SYSTEM CLEAN /dev/wd0g (891d4cfcef7cbcc1.g): 9186 files, 126172 used, 153819 free (4571 frags, 18656 blocks, 1.6% fragmentation) /dev/wd0g (891d4cfcef7cbcc1.g): MARKING FILE SYSTEM CLEAN /dev/wd0h (891d4cfcef7cbcc1.h): 5276 files, 53297 used, 899766 free (118 frags, 112456 blocks, 0.0% fragmentation) /dev/wd0h (891d4cfcef7cbcc1.h): MARKING FILE SYSTEM CLEAN /dev/wd0j (891d4cfcef7cbcc1.j): 2 files, 524425 used, 2171438 free (14 frags, 271428 blocks, 0.0% fragmentation) /dev/wd0j (891d4cfcef7cbcc1.j): MARKING FILE SYSTEM CLEAN /dev/wd0i (891d4cfcef7cbcc1.i): 1 files, 1 used, 713462 free (14 frags, 89181 blocks, 0.0% fragmentation) /dev/wd0i (891d4cfcef7cbcc1.i): MARKING FILE SYSTEM CLEAN /dev/wd0e (891d4cfcef7cbcc1.e): 501 files, 31335 used, 387240 free (88 frags, 48394 blocks, 0.0% fragmentation) /dev/wd0e (891d4cfcef7cbcc1.e): MARKING FILE SYSTEM CLEAN pf enabled starting network add net 0.0.0.0/0: gateway 10.107.20.254 reordering libraries: done. starting early daemons: syslogd pflogd ntpd. starting RPC daemons:. savecore: no core dump checking quotas: done. clearing /tmp kern.securelevel: 0 -> 1 creating runtime link editor directory cache. preserving editor files. starting network daemons: sshd smtpd sndiod. starting local daemons: cron. Wed Apr 7 20:24:44 PDT 2021 ---- Scrollback: [ 5] 550.00-551.01 sec 41.0 MBytes 342 Mbits/sec [ 5] 551.01-552.00 sec 40.0 MBytes 338 Mbits/sec [ 5] 552.00-553.00 sec 40.6 MBytes 341 Mbits/sec [ 5] 553.00-554.01 sec 40.7 MBytes 340 Mbits/sec [ 5] 554.01-555.00 sec 40.3 MBytes 339 Mbits/sec [ 5] 555.00-556.00 sec 40.6 MBytes 341 Mbits/sec [ 5] 556.00-557.00 sec 41.0 MBytes 344 Mbits/sec panic: pool_do_get: mcl2k2 free list modified: page 0xfffffd80308e8000; item addr 0xfffffd80308ee6c0; offset 0x0=0x2e14627000851e2 != 0x131b283cee7e51e2 Stopped at db_enter+0x10: popq %rbp TID PID UID PRFLAGS PFLAGS CPU COMMAND db_enter() at db_enter+0x10 panic(ffffffff81e63d13) at panic+0x12a pool_do_get(ffffffff82186c08,2,ffff800014e12ca4) at pool_do_get+0x321 pool_get(ffffffff82186c08,2) at pool_get+0x75 m_clget(0,2,802) at m_clget+0x18a em_get_buf(ffff80000002ee00,f2) at em_get_buf+0x59 em_rxfill(ffff80000002ee00) at em_rxfill+0xc4 em_intr(ffff80000009c000) at em_intr+0xaa intr_handler(ffff800014e12e90,ffff800000074280) at intr_handler+0x38 Xintr_ioapic_level16_untramp() at Xintr_ioapic_level16_untramp+0x1a3 Xspllower() at Xspllower+0x19 softintr_dispatch(0) at softintr_dispatch+0xc4 Xsoftclock() at Xsoftclock+0x1f acpicpu_idle() at acpicpu_idle+0x25d end trace frame: 0xffff800014e13110, count: 0 https://www.openbsd.org/ddb.html describes the minimum info required in bug reports. Insufficient info makes it difficult to find and fix bugs. ddb> ps PID TID PPID UID S FLAGS WAIT COMMAND 15220 216128 79774 0 3 0x83 select iperf3 79774 401118 78090 0 3 0x10008b pause ksh 78090 334892 63275 0 3 0x82 select sshd 80035 223623 1 0 3 0x100098 poll cron 7448 383767 1 110 3 0x100090 poll sndiod 67776 402027 1 99 3 0x100090 poll sndiod 34532 491704 52998 95 3 0x100092 kqread smtpd 25805 456398 52998 103 3 0x100092 kqread smtpd 94953 279979 52998 95 3 0x100092 kqread smtpd 75900 72327 52998 95 3 0x100092 kqread smtpd 525 2320 52998 95 3 0x100092 kqread smtpd 24155 480349 52998 95 3 0x100092 kqread smtpd 52998 134926 1 0 3 0x100080 kqread smtpd 63275 292624 1 0 3 0x80 select sshd 45721 215373 1 0 3 0x100080 poll ntpd 74593 426430 57619 83 3 0x100092 poll ntpd 57619 442923 1 83 3 0x100092 poll ntpd 62053 193710 18976 74 3 0x100092 bpf pflogd 18976 132414 1 0 3 0x80 netio pflogd 45540 370213 38999 73 3 0x100090 kqread syslogd 38999 119947 1 0 3 0x100082 netio syslogd 40523 382349 19856 115 3 0x100092 kqread slaacd 15649 313003 19856 115 3 0x100092 kqread slaacd 19856 154355 1 0 3 0x100080 kqread slaacd 99341 142700 0 0 3 0x14200 bored smr 60709 394946 0 0 3 0x14200 pgzero zerothread 75632 416007 0 0 3 0x14200 aiodoned aiodoned 76858 398479 0 0 3 0x14200 syncer update 98827 503027 0 0 3 0x14200 cleaner cleaner 10641 280519 0 0 3 0x14200 reaper reaper 76881 73775 0 0 3 0x14200 pgdaemon pagedaemon 57951 480893 0 0 3 0x14200 bored crynlk 49825 183771 0 0 3 0x14200 bored crypto 11173 279315 0 0 3 0x14200 usbtsk usbtask 8587 406910 0 0 3 0x14200 usbatsk usbatsk 8370 446960 0 0 3 0x40014200 acpi0 acpi0 59659 307189 0 0 2 0x14200 softnet 80651 258220 0 0 3 0x14200 bored systqmp 24726 343136 0 0 3 0x14200 bored systq 21646 421831 0 0 3 0x40014200 bored softclock *86057 50562 0 0 7 0x40014200 idle0 1 289975 0 0 3 0x82 wait init 0 0 -1 0 3 0x10200 scheduler swapper ddb> show registers rdi 0xffffffff820ba288 kprintf_mutex rsi 0x5 rbp 0xffff800014e12b40 rbx 0xffff800014e12b50 rdx 0x3fd rcx 0x4800000000001812 rax 0x1 r8 0xffff800014e12b00 r9 0 r10 0xff2c77b6a77211dd r11 0xcc76eb5cbe87dbfe r12 0x3000000008 r13 0xffff800014e12bf0 r14 0x100 r15 0xffffffff81e63d13 cy_pio_rec+0x137e8 rip 0xffffffff81991370 db_enter+0x10 cs 0x8 rflags 0x282 rsp 0xffff800014e12b40 ss 0x10 db_enter+0x10: popq %rbp