Hello, here I'm getting a kernel panic on a PC Engines APU1D4:
https://www.pcengines.ch/apu1d4.htm The user scenario is: I'm running a Dovecot server, and I have an SSH connection open via wireless interface. There is a usb stick connected that operates as softraid CRYPTO. Tmux is running. Then, eventually, a kernel panic occurs overnight. I tested this multiple times. The environment is OpenBSD 6.8 and latest patches installed (stable). Following the instructions in http://www.openbsd.org/ddb.html the crash happens for cpu 0 in > Stopped at db_enter+0x10: popq %rbp and cpu 1 in > Stopped at x86_ipi_db+0x12: leave Thank you very much for your support regarding this problem. Fox Steward === HERE ARE THE LOGS === dmesg ----- OpenBSD 6.8 (GENERIC.MP) #5: Mon Feb 22 04:36:10 MST 2021 r...@syspatch-68-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 4259577856 (4062MB) avail mem = 4115435520 (3924MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fe42040 (9 entries) bios0: vendor coreboot version "v4.12.0.5" date 09/25/2020 bios0: PC Engines apu1 acpi0 at bios0: ACPI 6.0 acpi0: sleep states S0 S1 S3 S4 S5 acpi0: tables DSDT FACP SSDT TPM2 APIC HEST SSDT SSDT HPET acpi0: wakeup devices AGPB(S4) HDMI(S4) PBR4(S4) PBR5(S4) PBR6(S4) PBR7(S4) PE20(S4) PE21(S4) PE22(S4) PE23(S4) PIBR(S4) UOH1(S3) UOH2(S3) UOH3(S3) UOH4(S3) UOH5(S3) [...] acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD G-T40E Processor, 1000.12 MHz, 14-02-00 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,SSSE3,CX16,POPCNT,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,IBS,SKINIT,ITSC cpu0: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 16-way L2 cache cpu0: 8 4MB entries fully associative cpu0: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 200MHz cpu0: mwait min=64, max=64, IBE cpu1 at mainbus0: apid 1 (application processor) cpu1: AMD G-T40E Processor, 1000.01 MHz, 14-02-00 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,SSSE3,CX16,POPCNT,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,IBS,SKINIT,ITSC cpu1: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 16-way L2 cache cpu1: 8 4MB entries fully associative cpu1: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative cpu1: smt 0, core 1, package 0 ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 21, 24 pins acpihpet0 at acpi0: 14318180 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus -1 (AGPB) acpiprt2 at acpi0: bus -1 (HDMI) acpiprt3 at acpi0: bus 1 (PBR4) acpiprt4 at acpi0: bus 2 (PBR5) acpiprt5 at acpi0: bus 3 (PBR6) acpiprt6 at acpi0: bus -1 (PBR7) acpiprt7 at acpi0: bus -1 (PE20) acpiprt8 at acpi0: bus -1 (PE21) acpiprt9 at acpi0: bus -1 (PE22) acpiprt10 at acpi0: bus -1 (PE23) acpiprt11 at acpi0: bus 4 (PIBR) acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001 "PRP0001" at acpi0 not configured "PRP0001" at acpi0 not configured "PRP0001" at acpi0 not configured "PRP0001" at acpi0 not configured "PRP0001" at acpi0 not configured "PRP0001" at acpi0 not configured "PRP0001" at acpi0 not configured acpicmos0 at acpi0 "BOOT0000" at acpi0 not configured acpicpu0 at acpi0: C2(0@100 io@0x841), C1(@1 halt!), PSS acpicpu1 at acpi0: C2(0@100 io@0x841), C1(@1 halt!), PSS acpitz0 at acpi0: critical temperature is 110 degC cpu0: 1000 MHz: speeds: 1000 800 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "AMD 14h Host" rev 0x00 ppb0 at pci0 dev 4 function 0 "AMD 14h PCIE" rev 0x00: msi pci1 at ppb0 bus 1 re0 at pci1 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E (0x2c00), msi, address 00:0d:b9:52:5b:24 rgephy0 at re0 phy 7: RTL8169S/8110S/8211 PHY, rev. 4 ppb1 at pci0 dev 5 function 0 "AMD 14h PCIE" rev 0x00: msi pci2 at ppb1 bus 2 re1 at pci2 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E (0x2c00), msi, address 00:0d:b9:52:5b:25 rgephy1 at re1 phy 7: RTL8169S/8110S/8211 PHY, rev. 4 ppb2 at pci0 dev 6 function 0 "AMD 14h PCIE" rev 0x00: msi pci3 at ppb2 bus 3 re2 at pci3 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E (0x2c00), msi, address 00:0d:b9:52:5b:26 rgephy2 at re2 phy 7: RTL8169S/8110S/8211 PHY, rev. 4 ahci0 at pci0 dev 17 function 0 "ATI SBx00 SATA" rev 0x40: apic 2 int 19, AHCI 1.2 ahci0: port 0: 3.0Gb/s scsibus1 at ahci0: 32 targets sd0 at scsibus1 targ 0 lun 0: <ATA, SATA SSD, SBFM> t10.ATA_SATA_SSD_22C7070505AE00012653 sd0: 15272MB, 512 bytes/sector, 31277232 sectors, thin ohci0 at pci0 dev 18 function 0 "ATI SB700 USB" rev 0x00: apic 2 int 18, version 1.0, legacy support ehci0 at pci0 dev 18 function 2 "ATI SB700 USB2" rev 0x00: apic 2 int 17 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 configuration 1 interface 0 "ATI EHCI root hub" rev 2.00/1.00 addr 1 ohci1 at pci0 dev 19 function 0 "ATI SB700 USB" rev 0x00: apic 2 int 18, version 1.0, legacy support ehci1 at pci0 dev 19 function 2 "ATI SB700 USB2" rev 0x00: apic 2 int 17 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 configuration 1 interface 0 "ATI EHCI root hub" rev 2.00/1.00 addr 1 piixpm0 at pci0 dev 20 function 0 "ATI SBx00 SMBus" rev 0x42: SMI iic0 at piixpm0 pcib0 at pci0 dev 20 function 3 "ATI SB700 ISA" rev 0x40 ppb3 at pci0 dev 20 function 4 "ATI SB600 PCI" rev 0x40 pci4 at ppb3 bus 4 ohci2 at pci0 dev 22 function 0 "ATI SB700 USB" rev 0x00: apic 2 int 18, version 1.0, legacy support ehci2 at pci0 dev 22 function 2 "ATI SB700 USB2" rev 0x00: apic 2 int 17 usb2 at ehci2: USB revision 2.0 uhub2 at usb2 configuration 1 interface 0 "ATI EHCI root hub" rev 2.00/1.00 addr 1 pchb1 at pci0 dev 24 function 0 "AMD 14h Link Cfg" rev 0x43 pchb2 at pci0 dev 24 function 1 "AMD 14h Address Map" rev 0x00 pchb3 at pci0 dev 24 function 2 "AMD 14h DRAM Cfg" rev 0x00 km0 at pci0 dev 24 function 3 "AMD 14h Misc Cfg" rev 0x00 pchb4 at pci0 dev 24 function 4 "AMD 14h CPU Power" rev 0x00 pchb5 at pci0 dev 24 function 5 "AMD 14h Reserved" rev 0x00 pchb6 at pci0 dev 24 function 6 "AMD 14h NB Power" rev 0x00 pchb7 at pci0 dev 24 function 7 "AMD 14h Reserved" rev 0x00 usb3 at ohci0: USB revision 1.0 uhub3 at usb3 configuration 1 interface 0 "ATI OHCI root hub" rev 1.00/1.00 addr 1 usb4 at ohci1: USB revision 1.0 uhub4 at usb4 configuration 1 interface 0 "ATI OHCI root hub" rev 1.00/1.00 addr 1 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com0: console com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo pcppi0 at isa0 port 0x61 spkr0 at pcppi0 lpt0 at isa0 port 0x378/4 irq 7 wbsio0 at isa0 port 0x2e/2: NCT5104D rev 0x53 usb5 at ohci2: USB revision 1.0 uhub5 at usb5 configuration 1 interface 0 "ATI OHCI root hub" rev 1.00/1.00 addr 1 vmm0 at mainbus0: SVM/RVI umass0 at uhub0 port 1 configuration 1 interface 0 "Generic Mass Storage Device" rev 2.00/5.00 addr 2 umass0: using SCSI over Bulk-Only scsibus2 at umass0: 2 targets, initiator 0 sd1 at scsibus2 targ 1 lun 0: <USB Mass, Storage Device, \\001 \\001> removable serial.XXX sd1: 30436MB, 512 bytes/sector, 62333952 sectors urtwn0 at uhub0 port 5 configuration 1 interface 0 "Realtek 802.11n WLAN Adapter" rev 2.00/2.00 addr 3 urtwn0: MAC/BB RTL8188CUS, RF 6052 1T1R, address 08:be:ac:0c:3f:d7 umass1 at uhub2 port 1 configuration 1 interface 0 "Generic Flash Card Reader/Writer" rev 2.01/1.00 addr 2 umass1: using SCSI over Bulk-Only scsibus3 at umass1: 2 targets, initiator 0 sd2 at scsibus3 targ 1 lun 0: <Multiple, Card Reader, 1.00> removable serial.XXX vscsi0 at root scsibus4 at vscsi0: 256 targets softraid0 at root scsibus5 at softraid0: 256 targets root on sd0a (024cbc614c17c0ed.a) swap on sd0b dump on sd0b Installed packages ------------------ # pkg_info athn-firmware-1.1p4 firmware binary images for athn(4) driver bzip2-1.0.8 block-sorting file compressor, unencumbered curl-7.72.0p0 transfer files with FTP, HTTP, HTTPS, etc. dovecot-2.3.13v0 compact IMAP/POP3 server libiconv-1.16p0 character set conversion library libsodium-1.0.18p1 library for network communications and cryptography lz4-1.9.2p0 fast BSD-licensed data compression nghttp2-1.41.0p1 library for HTTP/2 quirks-3.442 exceptions to pkg_add rules sqlite3-3.31.1p0 embedded SQL implementation urtwn-firmware-20180103p0 firmware binary images for urtwn(4) driver vmm-firmware-1.11.0p3 firmware binary images for vmm(4) driver xz-5.2.5 LZMA compression and decompression tools zstd-1.4.5p0 zstandard fast real-time compression algorithm Patches installed ----------------- # syspatch -l 001_bgpd 002_icmp6 003_tmux 004_wg 005_unwind 006_rpki 008_asn1 009_exit 010_smtpd 011_nd6 012_carp 013_libressl 014_pffrag 015_sshagent Kernel Panic ------------ ddb{0}> show panic bad dir ddb{1}> machine ddbcpu 0 Stopped at db_enter+0x10: popq %rbp db_enter() at db_enter+0x10 panic(ffffffff81e1644c) at panic+0x12a ufs_lookup() at ufs_lookup+0xed1 VOP_LOOKUP(fffffd817df6a750,ffff80002205cb70,ffff80002205cbc0) at VOP_LOOKUP+0x 46 vfs_lookup(ffff80002205cb40) at vfs_lookup+0x442 namei(ffff80002205cb40) at namei+0x2b5 dofstatat(ffff80002202b658,ffffff9c,89f4b4323c0,89f19638048,0) at dofstatat+0x8 c syscall(ffff80002205cdc0) at syscall+0x389 Xsyscall() at Xsyscall+0x128 end of kernel end trace frame: 0x7f7ffffdd500, count: 6 ddb{0}> trace db_enter() at db_enter+0x10 panic(ffffffff81e1644c) at panic+0x12a ufs_lookup() at ufs_lookup+0xed1 VOP_LOOKUP(fffffd817df6a750,ffff80002205cb70,ffff80002205cbc0) at VOP_LOOKUP+0x 46 vfs_lookup(ffff80002205cb40) at vfs_lookup+0x442 namei(ffff80002205cb40) at namei+0x2b5 dofstatat(ffff80002202b658,ffffff9c,89f4b4323c0,89f19638048,0) at dofstatat+0x8 c syscall(ffff80002205cdc0) at syscall+0x389 Xsyscall() at Xsyscall+0x128 end of kernel end trace frame: 0x7f7ffffdd500, count: -9 ddb{0}> machine ddbcpu 1 Stopped at x86_ipi_db+0x12: leave x86_ipi_db(ffff800021f38ff0) at x86_ipi_db+0x12 x86_ipi_handler() at x86_ipi_handler+0x80 Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23 acpicpu_idle() at acpicpu_idle+0x11f sched_idle(ffff800021f38ff0) at sched_idle+0x27e end trace frame: 0x0, count: 10 ddb{1}> trace x86_ipi_db(ffff800021f38ff0) at x86_ipi_db+0x12 x86_ipi_handler() at x86_ipi_handler+0x80 Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23 acpicpu_idle() at acpicpu_idle+0x11f sched_idle(ffff800021f38ff0) at sched_idle+0x27e end trace frame: 0x0, count: -5 ddb{1}> ps PID TID PPID UID S FLAGS WAIT COMMAND 81065 277639 78628 0 7 0x2 perl 78628 333242 13898 0 3 0x10008a pause sh 13898 267297 49444 0 3 0x10008a pause sh 49444 305841 61165 0 3 0x100090 piperd cron 30273 413960 60164 1001 3 0x100083 select ssh 11914 338018 91348 95 3 0x100092 kqread smtpd 91840 459472 91348 103 3 0x100092 kqread smtpd 76892 77916 91348 95 2 0x100492 smtpd 86842 481309 91348 95 3 0x100092 kqread smtpd 63115 413834 91348 95 3 0x100092 kqread smtpd 3878 97823 91348 95 3 0x100092 kqread smtpd 91348 137768 1 0 3 0x100080 kqread smtpd 64494 343758 67927 518 3 0x92 kqread stats 3037 430886 95210 0 3 0x100083 ttyin ksh 50808 137877 70510 0 3 0x100083 kqread tail 70510 232635 95210 0 3 0x10008b pause ksh 68437 230481 67927 0 3 0x92 kqread config 17934 290959 67927 0 3 0x92 kqread log 59081 240736 67927 518 3 0x92 kqread anvil 67927 243101 1 0 3 0x80 kqread dovecot 63607 250242 1 0 3 0x100080 kqread httpd 83733 260466 1 67 3 0x100092 kqread httpd 98936 493541 1 67 3 0x100092 kqread httpd 34174 58222 66899 1002 3 0x100083 ttyin ksh 66899 13393 95210 0 3 0x10008b pause ksh 60164 234315 76675 1001 3 0x10008b pause sh 76675 229189 55152 1001 3 0x10008b pause ksh 55152 261192 95210 0 3 0x10008b pause ksh 95210 129903 1 0 2 0x100480 tmux 55839 288972 0 0 3 0x14200 bored srdis 93620 342493 1 0 3 0x100083 ttyin ksh 61165 108085 1 0 3 0x100098 poll cron 19551 416652 1 99 3 0x100090 poll sndiod 82665 127493 1 110 3 0x100090 poll sndiod 68711 469926 1 0 3 0x80 select sshd 67697 313366 75418 74 2 0x100492 pflogd 75418 20408 1 0 3 0x80 netio pflogd 92478 244921 20829 73 2 0x100090 syslogd 20829 391744 1 0 3 0x100082 netio syslogd 31529 258323 4649 115 3 0x100092 kqread slaacd 47793 12758 4649 115 3 0x100092 kqread slaacd 4649 283643 1 0 3 0x100080 kqread slaacd 58486 259045 0 0 3 0x14200 bored smr 11950 381702 0 0 3 0x14200 pgzero zerothread 52091 64691 0 0 3 0x14200 aiodoned aiodoned 91065 336069 0 0 2 0x14200 update 64441 448754 0 0 3 0x14200 cleaner cleaner 2295 360418 0 0 3 0x14200 reaper reaper 66625 257917 0 0 3 0x14200 pgdaemon pagedaemon 89694 319941 0 0 3 0x14200 bored crynlk 9361 194373 0 0 3 0x14200 bored crypto 37010 297798 0 0 2 0x14200 sensors 38485 88124 0 0 2 0x14200 usbtask 77551 233513 0 0 3 0x14200 usbatsk usbatsk 28282 98477 0 0 2 0x40014200 acpi0 *88538 275627 0 0 7 0x40014200 idle1 6636 475999 0 0 2 0x14200 softnet
