>Synopsis: Frequent crashes to ddb{0}> prompt
>Category: <PR category (one line)>
>Environment:
System : OpenBSD 7.6
Details : OpenBSD 7.6 (GENERIC.MP) #338: Mon Sep 30 08:55:35 MDT
2024
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
Architecture: OpenBSD.amd64
Machine : amd64
>Description:
Frequent crashes to ddb{0}> prompt.
Sorry, I think this might come through twice - the earlier one through sendbug
didn't send - but then now seems to have gone... but this has more information.
Strangely, it can work fine for weeks, then can happen several times in quick
succession - for example, 18:42, 19:05 and 19:58 tonight.
I'm running a router on PC Engines APU2 hardware. It's a pretty simple setup
with a few interfaces, PF firewall and a PPPoE interface.
I have tried two different apu2 machines and have had the same problem on both.
I run a pair of the same apu2 machines as firewalls (one in the same 1U rack
case as the one experiencing the problem) and have not had this issue on those
- the only difference there is the PPPoE interface.
Also, it generally seems to have this output in the logs before the problem
occurs:
Dec 11 19:08:43 edge /bsd: pppoe0: received unexpected PADO
Dec 11 19:08:43 edge last message repeated 10 times
Dec 11 19:08:45 edge /bsd: pppoe0: host unique tag found, but it belongs to a
connection in state 3
Dec 11 19:08:45 edge /bsd: pppoe: received PADO but could not find request for
it
Dec 11 19:08:45 edge /bsd: pppoe0: host unique tag found, but it belongs to a
connection in state 3
Dec 11 19:08:45 edge /bsd: pppoe: received PADO but could not find request for
it
Dec 11 19:08:45 edge /bsd: pppoe0: host unique tag found, but it belongs to a
connection in state 3
Dec 11 19:08:45 edge /bsd: pppoe: received PADO but could not find request for
it
Dec 11 19:08:45 edge /bsd: pppoe0: host unique tag found, but it belongs to a
connection in state 3
Dec 11 19:08:45 edge /bsd: pppoe: received PADO but could not find request for
it
Dec 11 19:08:45 edge /bsd: pppoe0: host unique tag found, but it belongs to a
connection in state 3
Dec 11 19:08:45 edge /bsd: pppoe: received PADO but could not find request for
it
Dec 11 19:08:45 edge /bsd: pppoe0: host unique tag found, but it belongs to a
connection in state 3
Dec 11 19:08:45 edge /bsd: pppoe: received PADO but could not find request for
it
Dec 11 19:08:45 edge /bsd: pppoe0: host unique tag found, but it belongs to a
connection in state 3
Dec 11 19:08:45 edge /bsd: pppoe: received PADO but could not find request for
it
Dec 11 19:08:45 edge /bsd: pppoe0: host unique tag found, but it belongs to a
connection in state 3
Dec 11 19:08:45 edge /bsd: pppoe: received PADO but could not find request for
it
Dec 11 19:08:45 edge /bsd: pppoe0: host unique tag found, but it belongs to a
connection in state 3
Dec 11 19:08:45 edge /bsd: pppoe: received PADO but could not find request for
it
Dec 11 19:08:45 edge /bsd: pppoe0: host unique tag found, but it belongs to a
connection in state 3
Dec 11 19:08:45 edge /bsd: pppoe: received PADO but could not find request for
it
Dec 11 19:08:45 edge /bsd: pppoe0: host unique tag found, but it belongs to a
connection in state 3
Dec 11 19:08:45 edge /bsd: pppoe: received PADO but could not find request for
it
Dec 11 19:08:45 edge /bsd: pppoe0: host unique tag found, but it belongs to a
connection in state 3
Dec 11 19:08:45 edge /bsd: pppoe: received PADO but could not find request for
it
Dec 11 19:08:51 edge reorder_kernel: kernel relinking done
One capture:
ddb{0}> trace
mtx_enter(0) at mtx_enter+0x65
Xsoftclock() at Xsoftclock+0x27
acpicpu_idle() at acpicpu_idle+0x131
sched_idle(ffffffff82755ff0) at sched_idle+0x298
end trace frame: 0x0, count: -4
ddb{0}> print
ffffffff818f5e95
ddb{0}> examine
mtx_enter+0x65: 141d3b4c
Another capture - with the commands from the bug report man page:
ddb{0}> show panic
*cpu0: uvm_fault(0xffffffff828cd410, 0x687863f8, 0, 1) -> e
ddb{0}> trace
db_read_bytes(687863f8,8,ffff80002a098f58) at db_read_bytes+0x43
db_get_value(687863f8,8,0) at db_get_value+0x43
db_stack_trace_print(ffffffff81fe4e4f,0,e,ffffffff8237040f,ffffffff81eddec0) at
db_stack_trace_print+0x31d
db_trap(6,0) at db_trap+0x10e
db_ktrap(6,0,ffff80002a099140) at db_ktrap+0x155
kerntrap(ffff80002a099140) at kerntrap+0xcb
alltraps_kern_meltdown() at alltraps_kern_meltdown+0x7b
end of kernel
end trace frame: 0x68786400, count: -7
ddb{0}> ps
PID TID PPID UID S FLAGS WAIT COMMAND
74508 273993 42026 0 3 0x100083 ttyin ksh
42026 511107 59535 1000 3 0xb1 kqread sudo
59535 304419 60587 1000 3 0xb3 kqread sudo
60587 497508 52690 1000 3 0x10008b sigsusp ksh
52690 337460 80404 1000 3 0x98 kqread sshd-session
80404 284420 31219 0 3 0x92 kqread sshd-session
23271 210395 315 1000 3 0x100083 ttyin ksh
315 141749 20773 1000 3 0x98 kqread sshd-session
20773 431825 31219 0 3 0x92 kqread sshd-session
95143 96674 1 0 3 0x100083 ttyin getty
13242 340775 1 0 3 0x100098 kqread cron
59564 313751 1 826 3 0x100083 piperd logger
47663 429950 1 826 3 0x83 thrsleep telegraf
47663 37370 1 826 3 0x4000083 thrsleep telegraf
47663 57022 1 826 3 0x4000083 thrsleep telegraf
47663 23278 1 826 3 0x4000083 thrsleep telegraf
47663 50205 1 826 3 0x4000083 thrsleep telegraf
47663 14397 1 826 3 0x4000083 thrsleep telegraf
47663 251080 1 826 3 0x4000083 thrsleep telegraf
47663 40078 1 826 3 0x4000083 thrsleep telegraf
47663 388963 1 826 3 0x4000083 thrsleep telegraf
47663 57537 1 826 3 0x4000083 thrsleep telegraf
47663 299376 1 826 3 0x4000083 thrsleep telegraf
47663 103797 1 826 3 0x4000083 thrsleep telegraf
47663 147238 1 826 3 0x4000083 kqread telegraf
47663 173070 1 826 3 0x4000083 thrsleep telegraf
10524 47107 1 99 3 0x1100090 kqread sndiod
21508 39113 1 110 3 0x100090 kqread sndiod
7222 130837 19243 95 3 0x1100092 kqread smtpd
9936 424118 19243 103 3 0x1100092 kqread smtpd
13980 198314 19243 95 3 0x1100092 kqread smtpd
42737 131028 19243 95 3 0x100092 kqread smtpd
36435 475456 19243 95 3 0x1100092 kqread smtpd
28529 439675 19243 95 3 0x1100092 kqread smtpd
19243 344395 1 0 3 0x100080 kqread smtpd
37547 425351 1 77 3 0x1100090 kqread dhcpd
31219 388499 1 0 3 0x88 kqread sshd
43513 434829 1 0 3 0x100080 kqread ntpd
88292 440946 32662 83 3 0x100092 kqread ntpd
32662 455124 1 83 3 0x1100092 kqread ntpd
65437 419274 65505 74 3 0x1100092 bpf pflogd
65505 458724 1 0 3 0x80 sbwait pflogd
27771 66469 24422 73 3 0x1100090 kqread syslogd
24422 217823 1 0 3 0x100082 sbwait syslogd
14670 376782 1 0 3 0x100080 kqread resolvd
1027 462447 10711 77 3 0x100092 kqread dhcpleased
32728 53838 10711 77 3 0x100092 kqread dhcpleased
10711 257371 1 0 3 0x80 kqread dhcpleased
98731 51884 57393 115 3 0x100092 kqread slaacd
38794 524240 57393 115 3 0x100092 kqread slaacd
57393 497983 1 0 3 0x100080 kqread slaacd
58386 209750 0 0 3 0x14200 bored smr
81598 377130 0 0 3 0x14200 pgzero zerothread
59092 504852 0 0 3 0x14200 aiodoned aiodoned
84528 370345 0 0 3 0x14200 syncer update
63818 195751 0 0 3 0x14200 cleaner cleaner
68904 393167 0 0 3 0x14200 reaper reaper
82643 283919 0 0 3 0x14200 pgdaemon pagedaemon
93133 162363 0 0 3 0x14200 mmctsk sdmmc0
6023 370311 0 0 3 0x14200 usbtsk usbtask
9840 398974 0 0 3 0x14200 usbatsk usbatsk
80908 191277 0 0 3 0x40014200 acpi0 acpi0
88121 36765 0 0 7 0x40014200 idle3
67562 512711 0 0 3 0x40014200 idle2
62787 381340 0 0 7 0x40014200 idle1
853 119827 0 0 3 0x14200 bored sensors
9846 122280 0 0 3 0x14200 bored softnet3
37949 36964 0 0 3 0x14200 bored softnet2
2822 272163 0 0 3 0x14200 bored softnet1
2189 278310 0 0 7 0x14200 softnet0
39002 210612 0 0 3 0x14200 bored systqmp
61901 240656 0 0 3 0x14200 bored systq
86466 235456 0 0 3 0x14200 tmoslp softclockmp
71550 510525 0 0 3 0x40014200 tmoslp softclock
*29302 125796 0 0 7 0x40014200 idle0
1 258542 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb{0}> show registers
rdi 0x687863f8
rsi 0x8
rbp 0xffff80002a098f40
rbx 0xe
rdx 0xffff80002a098f58
rcx 0x4f
rax 0x68786400
r8 0xffff80002a098f70
r9 0
r10 0x17a115daf90ef1d1
r11 0xe914a7ba61d44b57
r12 0x8
r13 0
r14 0x8
r15 0
rip 0xffffffff8187bf73 db_read_bytes+0x43
cs 0x8
rflags 0x10246 __ALIGN_SIZE+0xf246
rsp 0xffff80002a098f20
ss 0x10
db_read_bytes+0x43: movq 0(%rdi),%rax
ddb{0}>
>How-To-Repeat:
Not able to re-produce - it just happens sometimes.
>Fix:
Have to connect to the serial console and reboot.
SENDBUG: Run sendbug as root if this is an ACPI report!
SENDBUG: dmesg and usbdevs are attached.
SENDBUG: Feel free to delete or use the -D flag if they contain sensitive
information.
dmesg:
OpenBSD 7.6 (GENERIC.MP) #338: Mon Sep 30 08:55:35 MDT 2024
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2112446464 (2014MB)
avail mem = 2025299968 (1931MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7ee97040 (13 entries)
bios0: vendor coreboot version "v4.19.0.1" date 01/31/2023
bios0: PC Engines apu2
acpi0 at bios0: ACPI 6.0
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP SSDT MCFG TPM2 APIC HEST SSDT SSDT DRTM HPET
acpi0: wakeup devices PBR4(S4) PBR5(S4) PBR6(S4) PBR7(S4) PBR8(S4) UOH1(S3)
UOH2(S3) UOH3(S3) UOH4(S3) UOH5(S3) UOH6(S3) XHC0(S4)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimcfg0 at acpi0
acpimcfg0: addr 0xf8000000, bus 0-63
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD GX-412TC SOC, 998.17 MHz, 16-30-01, patch 07030105
cpu0: cpuid 1
edx=178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT>
ecx=36d8220b<SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C>
cpu0: cpuid 6 eax=4<ARAT> ecx=1<EFFFREQ>
cpu0: cpuid 7.0 ebx=8<BMI1>
cpu0: cpuid d.1 eax=1<XSAVEOPT>
cpu0: cpuid 80000001 edx=2fd3fbff<NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG>
ecx=1d4037ff<LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3>
cpu0: cpuid 80000007 edx=33d9<HWPSTATE,ITSC>
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 2-way I-cache, 2MB 64b/line
16-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD GX-412TC SOC, 998.26 MHz, 16-30-01, patch 07030105
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: AMD GX-412TC SOC, 998.27 MHz, 16-30-01, patch 07030105
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: AMD GX-412TC SOC, 998.34 MHz, 16-30-01, patch 07030105
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 21, 24 pins
ioapic1 at mainbus0: apid 5 pa 0xfec20000, version 21, 32 pins
acpihpet0 at acpi0: 14318180 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PBR4)
acpiprt2 at acpi0: bus 1 (PBR5)
acpiprt3 at acpi0: bus 2 (PBR6)
acpiprt4 at acpi0: bus 3 (PBR7)
acpiprt5 at acpi0: bus -1 (PBR8)
acpicpu0 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu1 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu2 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu3 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at acpi0 COM2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
amdgpio0 at acpi0 GPIO uid 0 addr 0xfed81500/0x300 irq 7, 184 pins
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"BOOT0000" at acpi0 not configured
acpitz0 at acpi0: critical temperature is 115 degC
cpu0: 998 MHz: speeds: 1000 800 600 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "AMD 16h Root Complex" rev 0x00
vendor "AMD", unknown product 0x1567 (class system subclass IOMMU, rev 0x00) at
pci0 dev 0 function 2 not configured
pchb1 at pci0 dev 2 function 0 "AMD 16h Host" rev 0x00
ppb0 at pci0 dev 2 function 2 "AMD 16h PCIE" rev 0x00: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel I211" rev 0x03: msi, address
00:0d:b9:46:cd:9c
ppb1 at pci0 dev 2 function 3 "AMD 16h PCIE" rev 0x00: msi
pci2 at ppb1 bus 2
em1 at pci2 dev 0 function 0 "Intel I211" rev 0x03: msi, address
00:0d:b9:46:cd:9d
ppb2 at pci0 dev 2 function 4 "AMD 16h PCIE" rev 0x00: msi
pci3 at ppb2 bus 3
em2 at pci3 dev 0 function 0 "Intel I211" rev 0x03: msi, address
00:0d:b9:46:cd:9e
ccp0 at pci0 dev 8 function 0 "AMD 16h Crypto" rev 0x00: msix
xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msix, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "AMD xHCI root hub" rev 3.00/1.00
addr 1
ahci0 at pci0 dev 17 function 0 "AMD Hudson-2 SATA" rev 0x40: apic 4 int 19,
AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, Samsung SSD 840, EXT4> naa.5002538844584d30
sd0: 114473MB, 512 bytes/sector, 234441648 sectors, thin
ehci0 at pci0 dev 19 function 0 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 18
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00
addr 1
piixpm0 at pci0 dev 20 function 0 "AMD Hudson-2 SMBus" rev 0x42: SMI
iic0 at piixpm0
iic1 at piixpm0
iic1: addr 0x4c 3e=00 48=00 4a=00 4e=00 fc=00 fe=00 words 00=ffff 01=ffff
02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
pcib0 at pci0 dev 20 function 3 "AMD Hudson-2 LPC" rev 0x11
sdhc0 at pci0 dev 20 function 7 "AMD Bolton SD/MMC" rev 0x01: apic 4 int 16
sdhc0: SDHC 2.00, 50 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma
pchb2 at pci0 dev 24 function 0 "AMD 16h Link Cfg" rev 0x00
pchb3 at pci0 dev 24 function 1 "AMD 16h Address Map" rev 0x00
pchb4 at pci0 dev 24 function 2 "AMD 16h DRAM Cfg" rev 0x00
km0 at pci0 dev 24 function 3 "AMD 16h Misc Cfg" rev 0x00
pchb5 at pci0 dev 24 function 4 "AMD 16h CPU Power" rev 0x00
pchb6 at pci0 dev 24 function 5 "AMD 16h Misc Cfg" rev 0x00
isa0 at pcib0
isadma0 at isa0
com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
intr_establish: pic ioapic0 pin 7: can't share type 3 with 2
wbsio0 at isa0 port 0x2e/2: NCT5104D rev 0x53
vmm0 at mainbus0: SVM/RVI
uhub2 at uhub1 port 1 configuration 1 interface 0 "Advanced Micro Devices Hub"
rev 2.00/0.18 addr 2
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (30a95afa2e188e6c.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
usbdevs:
Controller /dev/usb0:
addr 01: 1022:0000 AMD, xHCI root hub
super speed, self powered, config 1, rev 1.00
driver: uhub0
Controller /dev/usb1:
addr 01: 1022:0000 AMD, EHCI root hub
high speed, self powered, config 1, rev 1.00
driver: uhub1
addr 02: 0438:7900 Advanced Micro Devices, Hub
high speed, self powered, config 1, rev 0.18
driver: uhub2
