Hi,
One of my OpenBSD boxes sometimes gets in a weird locked-up or
almost-locked-up state. I'm wondering what I can do to debug it
further next time it happens.
It feels like swap thrashing, but top reports plenty of memory free.
Symptoms:
1. top reports lots of free memory, small act/tot and cache amounts,
and some swap space used.
See example output of "top -ud1" below. Note I ran pkill -9 firefox
before running that command, so apparently it's taking a while even
for kill -9 to have an effect.
It's weird the act/tot amounts are so small when free is so high.
I would also expect the cache number to be much higher. E.g. on
this occasion, I was running "git annex fsck", which reads plenty
of data from disk. With my vague understanding of the vm system I
would expect this to result in lots being cached.
3. Hard disk light is constantly active. (Swapping?)
2. System is completely or almost completely unresponsive.
Usually when this happens the X11 display appears completely frozen
and I can't ssh to the machine, switch to another virtual terminal,
or otherwise interact. (I think pinging the box still works in
this case.) I will sometimes notice the system slowing down for
a minute or so, and hard disk activity increasing, before it
completely grinds to a halt.
The most recent time I got lucky, and I could still use ssh and run
simple commands like top and dmesg. After several minutes, I even
saw the X11 display update. My tmux-in-xterm windows were still
trying to redraw themselves for several minutes after being resized:
a few lines were displayed but many were still missing.
Just now, about 40 minutes after the lock-up, my system started
responding quickly again, as if nothing was ever wrong. Maybe it
hepled that I killed some processes including firefox, but I did
that probably more than 10 minutes ago.
below dmesg is from when it was in the almost-locked-up state. I
think the filesystem full messages at the end can be ignored; that
was an issue I resolved earlier. I'm pretty sure I've observed this
on 7.4 and/or 7.5 before I switched to current.
--
James
angel ~ $ top -ud1
load averages: 8.17, 10.86, 10.03 angel.falsifian.org 18:44:22
200 processes: 1 starting, 6 running, 171 idle, 1 stopped, 18 dead, 3 on
processor up 0 days 04:10:10
CPU0 states: 7.6% user, 0.0% nice, 8.4% sys, 6.6% spin, 2.6% intr, 74.8%
idle
CPU1 states: 10.1% user, 0.0% nice, 10.6% sys, 4.8% spin, 0.0% intr, 74.5%
idle
CPU2 states: 9.8% user, 0.0% nice, 10.3% sys, 4.9% spin, 0.0% intr, 75.0%
idle
Memory: Real: 88K/1197M act/tot Free: 14G Cache: 165M Swap: 1046M/32G
PID UID PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
77712 1000 62 0 39M 20K run/1 - 3:21 30.52% tmux
79301 1000 60 0 1752K 20K run/1 - 0:42 29.59% sshd-session
96911 48 62 0 10M 20K run/2 - 1:30 26.56% unwind
9794 35 61 0 133M 3156K onproc/2 - 8:12 25.20% Xorg
59778 1000 10 0 610M 20M run/1 thrdeat 19:05 10.84% firefox
27864 1000 18 0 1308K 4K sleep/0 sigsusp 0:31 4.98% sh
48161 1000 -6 0 1308K 4K sleep/2 piperd 0:31 4.59% sh
dmesg:
OpenBSD 7.5-current (GENERIC.MP) #77: Sun May 19 16:08:08 MDT 2024
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17160474624 (16365MB)
avail mem = 16619134976 (15849MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf0100 (59 entries)
bios0: vendor Award Software International, Inc. version "F7" date 11/20/2009
bios0: Gigabyte Technology Co., Ltd. GA-MA790XT-UD4P
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP SSDT HPET MCFG TAMG APIC
acpi0: wakeup devices PCI0(S5) USB0(S3) USB1(S3) USB2(S3) USB3(S3) USB4(S3)
USB5(S3) USB6(S3) SBAZ(S4) P2P_(S5) PCE2(S4) PCE3(S4) PCE4(S4) PCE5(S4)
PCE6(S4) PCE7(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpihpet0 at acpi0: 14318180 Hz
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD Phenom(tm) II X3 710 Processor, 2611.93 MHz, 10-04-02, patch 010000db
cpu0: cpuid 1
edx=178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT>
ecx=802009<SSE3,MWAIT,CX16,POPCNT>
cpu0: cpuid 80000001
edx=efd3fbff<NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,3DNOW2,3DNOW>
ecx=37ff<LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT>
cpu0: cpuid 80000007 edx=1f9<HWPSTATE,ITSC>
cpu0: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache
cpu0: 512KB 64b/line 16-way L2 cache
cpu0: smt 0, core 0, package 0
cpu0: AMD erratum 721 detected and fixed
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 200MHz
cpu0: mwait min=64, max=64, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD Phenom(tm) II X3 710 Processor, 2611.94 MHz, 10-04-02, patch 010000db
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: AMD Phenom(tm) II X3 710 Processor, 2612.02 MHz, 10-04-02, patch 010000db
cpu2: smt 0, core 2, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 21, 24 pins, remapped
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (P2P_)
acpiprt2 at acpi0: bus 1 (PCE2)
acpiprt3 at acpi0: bus -1 (PCE3)
acpiprt4 at acpi0: bus 2 (PCE4)
acpiprt5 at acpi0: bus -1 (PCE5)
acpiprt6 at acpi0: bus -1 (PCE6)
acpiprt7 at acpi0: bus -1 (PCE7)
acpiprt8 at acpi0: bus -1 (PCE8)
acpiprt9 at acpi0: bus -1 (PCE9)
acpiprt10 at acpi0: bus 3 (PCEA)
acpiprt11 at acpi0: bus -1 (PCEB)
acpiprt12 at acpi0: bus -1 (PCEC)
acpibtn0 at acpi0: PWRB
acpipci0 at acpi0 PCI0
acpicmos0 at acpi0
com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
"PNP0C14" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!), PSS
acpicpu1 at acpi0: C1(@1 halt!), PSS
acpicpu2 at acpi0: C1(@1 halt!), PSS
cpu0: 2611 MHz: speeds: 2600 1900 1400 800 MHz
pci0 at mainbus0 bus 0
0:0:0: mem address conflict 0xe0000000/0x20000000
pchb0 at pci0 dev 0 function 0 "ATI RD780 HT-PCIE" rev 0x00
ppb0 at pci0 dev 2 function 0 "ATI RD790 PCIE" rev 0x00
pci1 at ppb0 bus 1
radeondrm0 at pci1 dev 0 function 0 "ATI Radeon HD 5450" rev 0x00
drm0 at radeondrm0
radeondrm0: msi
azalia0 at pci1 dev 0 function 1 "ATI Radeon HD 5470 Audio" rev 0x00: msi
azalia0: no supported codecs
ppb1 at pci0 dev 4 function 0 "ATI RD790 PCIE" rev 0x00
pci2 at ppb1 bus 2
jmb0 at pci2 dev 0 function 0 "JMicron JMB363 IDE/SATA" rev 0x02
ahci0 at jmb0: apic 2 int 16, AHCI 1.0
scsibus1 at ahci0: 32 targets
pciide0 at jmb0: DMA, channel 0 wired to native-PCI, channel 1 wired to
native-PCI
pciide0: using apic 2 int 16 for native-PCI interrupt
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
ppb2 at pci0 dev 10 function 0 "ATI RD790 PCIE" rev 0x00
pci3 at ppb2 bus 3
re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x03: RTL8168D/8111D (0x2800),
msi, address 00:24:1d:12:df:b1
rgephy0 at re0 phy 7: RTL8169S/8110S/8211 PHY, rev. 2
ahci1 at pci0 dev 17 function 0 "ATI SBx00 SATA" rev 0x00: apic 2 int 22, AHCI
1.1
ahci1: port 2: 3.0Gb/s
ahci1: port 4: 3.0Gb/s
scsibus2 at ahci1: 32 targets
sd0 at scsibus2 targ 2 lun 0: <ATA, WDC WDS100T2B0A, 4150> naa.5001b448b53807dd
sd0: 953869MB, 512 bytes/sector, 1953525168 sectors, thin
sd1 at scsibus2 targ 4 lun 0: <ATA, Hitachi HDS72107, GK8O> naa.5000cca215c1867e
sd1: 715404MB, 512 bytes/sector, 1465149168 sectors
ohci0 at pci0 dev 18 function 0 "ATI SB700 USB" rev 0x00: apic 2 int 16,
version 1.0, legacy support
ohci1 at pci0 dev 18 function 1 "ATI SB700 USB" rev 0x00: apic 2 int 16,
version 1.0, legacy support
ehci0 at pci0 dev 18 function 2 "ATI SB700 USB2" rev 0x00: apic 2 int 17
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "ATI EHCI root hub" rev 2.00/1.00
addr 1
ohci2 at pci0 dev 19 function 0 "ATI SB700 USB" rev 0x00: apic 2 int 18,
version 1.0, legacy support
ohci3 at pci0 dev 19 function 1 "ATI SB700 USB" rev 0x00: apic 2 int 18,
version 1.0, legacy support
ehci1 at pci0 dev 19 function 2 "ATI SB700 USB2" rev 0x00: apic 2 int 19
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "ATI EHCI root hub" rev 2.00/1.00
addr 1
piixpm0 at pci0 dev 20 function 0 "ATI SBx00 SMBus" rev 0x3a: SMI
iic0 at piixpm0
spdmem0 at iic0 addr 0x50: 4GB DDR3 SDRAM PC3-10600
spdmem1 at iic0 addr 0x51: 4GB DDR3 SDRAM PC3-10600
spdmem2 at iic0 addr 0x52: 4GB DDR3 SDRAM PC3-10600
spdmem3 at iic0 addr 0x53: 4GB DDR3 SDRAM PC3-10600
pciide1 at pci0 dev 20 function 1 "ATI SB700 IDE" rev 0x00: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
azalia1 at pci0 dev 20 function 2 "ATI SBx00 HD Audio" rev 0x00: apic 2 int 16
azalia1: codecs: Realtek ALC885
audio0 at azalia1
pcib0 at pci0 dev 20 function 3 "ATI SB700 ISA" rev 0x00
ppb3 at pci0 dev 20 function 4 "ATI SB600 PCI" rev 0x00
pci4 at ppb3 bus 4
"TI TSB43AB23 FireWire" rev 0x00 at pci4 dev 14 function 0 not configured
ohci4 at pci0 dev 20 function 5 "ATI SB700 USB" rev 0x00: apic 2 int 18,
version 1.0, legacy support
pchb1 at pci0 dev 24 function 0 "AMD 10h HyperTransport" rev 0x00
pchb2 at pci0 dev 24 function 1 "AMD 10h Address Map" rev 0x00
pchb3 at pci0 dev 24 function 2 "AMD 10h DRAM Cfg" rev 0x00
km0 at pci0 dev 24 function 3 "AMD 10h Misc Cfg" rev 0x00
pchb4 at pci0 dev 24 function 4 "AMD 10h Link Cfg" rev 0x00
usb2 at ohci0: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "ATI OHCI root hub" rev 1.00/1.00
addr 1
usb3 at ohci1: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "ATI OHCI root hub" rev 1.00/1.00
addr 1
usb4 at ohci2: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "ATI OHCI root hub" rev 1.00/1.00
addr 1
usb5 at ohci3: USB revision 1.0
uhub5 at usb5 configuration 1 interface 0 "ATI OHCI root hub" rev 1.00/1.00
addr 1
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
it0 at isa0 port 0x2e/2: IT8720F rev 5, EC port 0x228
usb6 at ohci4: USB revision 1.0
uhub6 at usb6 configuration 1 interface 0 "ATI OHCI root hub" rev 1.00/1.00
addr 1
vmm0 at mainbus0: SVM/RVI
uhub7 at uhub1 port 4 configuration 1 interface 0 "Genesys Logic USB2.0 Hub"
rev 2.00/77.63 addr 2
uhidev0 at uhub7 port 1 configuration 1 interface 0 "Kensington Kensington
Expert Mouse" rev 1.10/2.01 addr 3
uhidev0: iclass 3/1
ums0 at uhidev0: 4 buttons, Z dir
wsmouse0 at ums0 mux 0
uhidev1 at uhub7 port 2 configuration 1 interface 0 "ZSA Ergodox EZ" rev
1.10/0.01 addr 4
uhidev1: iclass 3/1
ukbd0 at uhidev1: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard
uhidev2 at uhub7 port 2 configuration 1 interface 1 "ZSA Ergodox EZ" rev
1.10/0.01 addr 4
uhidev2: iclass 3/0, 5 report ids
uhid0 at uhidev2 reportid 3: input=2, output=0, feature=0
ucc0 at uhidev2 reportid 4: 668 usages, 20 keys, array
wskbd1 at ucc0 mux 1
ukbd1 at uhidev2 reportid 5: 128 variable keys, 0 key codes
wskbd2 at ukbd1 mux 1
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
sd2 at scsibus4 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006>
sd2: 391167MB, 512 bytes/sector, 801111803 sectors
root on sd2a (676fc55ae679a444.a) swap on sd2b dump on sd2b
radeondrm0: CEDAR
radeon_audio_component_init: stub
radeondrm0: 1920x1080, 32bpp
wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using wskbd0
wskbd1: connecting to wsdisplay0
wskbd2: connecting to wsdisplay0
wsdisplay0: screen 1-5 added (std, vt100 emulation)
uid 1000 on /mnt/nature/ext2: file system full
uid 1000 on /mnt/nature/ext2: file system full
uid 1000 on /mnt/nature/ext2: file system full
uid 1000 on /mnt/nature/ext2: file system full
