Hello.
I use my OBSD machine to test some server on another machine. They are
connected by pathcord, 1Gbit network cards are used.
Test program (uses kqueue) do many (I want thousands) connections to
server. Write query, read answer.
And it tries to keep that much connections by doing as much new
connections as needed.
When number of connections kept below 100 - all ok. But if I raise
them (upto about 500-1000) the program start these connections, do
some write/read (show about 10-20 successful reads) and the kernel
hangs. 1-2 sec after start.
Tweaks - kern.maxfiles=16384 and openfiles-cur/max=8192 for my user.
Info from ddb (see dmesg below):
ddb> show panic
the kernel did not panic
ddb> trace
Debugger(0,3f8,0,0,1) at Debugger+0x4
comintr(d1571000) at comintr+0x287
Xrecurse_legacy4() at Xrecurse_legacy4+0xb3
--- interrupt ---
pool_do_get(d0a10b60,0,0,0,60) at pool_do_get+0x2c2
pool_get(d0a10b60,0,80000000,0,0) at pool_get+0x54
m_gethdr(1,1,80000000,369e99,0) at m_gethdr+0x39
m_clget(0,1,d1526054,800,d03e1aeb) at m_clget+0x10a
re_newbuf(d1526000,10,d999eb48,d02b30cc,d1526000) at re_newbuf+0x35
re_rx_list_fill(d1526000,20,60,58,d1520010) at re_rx_list_fill+0x21
re_rxeof(d1526000,d9799800,3e,10,10) at re_rxeof+0x37c
re_intr(d1526000) at re_intr+0x12a
Xrecurse_legacy11() at Xrecurse_legacy11+0xb7
--- interrupt ---
filt_soread(d9a5bdc0,0,0,d9a5bd98,d9a5bd98) at filt_soread+0x1
selwakeup(d9a5bdbc,d9b08300,d9b08200,d9b08300,d9a5bd98) at selwakeup+0x22
sowakeup(d9a5bd4c,d9a5bd98,14,d999ed24,1) at sowakeup+0x1d
tcp_input(d9b08300,14,0,0,6) at tcp_input+0x26ac
ipv4_input(d9b08300,0,d999ede8,d0202089,d03d0058) at ipv4_input+0x42a
ipintr(d03d0058,d09e0010,10,d5d10010,d09e72c0) at ipintr+0x49
Bad frame pointer: 0xd999ede8
ddb> ps
PID PPID PGRP UID S FLAGS WAIT COMMAND
3410 32488 3410 1000 2 0x4000 penetrator
16754 6068 16754 1000 3 0x44180 poll systat
6068 20285 6068 1000 3 0x4080 pause ksh
20285 13637 13637 1000 3 0x180 select sshd
13637 9091 13637 0 3 0x4080 netio sshd
921 24774 921 1000 3 0x4080 poll top
24774 7958 24774 1000 3 0x4080 pause ksh
7958 18572 18572 1000 3 0x180 select sshd
18572 9091 18572 0 3 0x4080 netio sshd
20295 16560 16560 1000 3 0x180 netio ftpd
16560 28104 16560 0 3 0x4080 netio ftpd
32488 24805 32488 1000 3 0x4080 pause ksh
24805 1162 1162 1000 3 0x180 select sshd
1162 9091 1162 0 3 0x4080 netio sshd
11793 1 11793 0 3 0x40180 select sendmail
24133 1 24133 0 2 0x4080 getty
5061 1 5061 0 3 0x4080 ttyin getty
16343 1 16343 0 3 0x4080 ttyin getty
27708 1 27708 0 3 0x4080 ttyin getty
21353 1 21353 0 3 0x4080 ttyin getty
25731 1 25731 0 3 0x4080 ttyin getty
4928 1 4928 0 3 0x80 select cron
4928 1 4928 0 3 0x80 select cron
9091 1 9091 0 3 0x80 select sshd
18814 30428 30428 70 3 0x180 select named
30428 1 30428 0 3 0x180 netio named
9309 32415 32415 74 3 0x180 bpf pflogd
32415 1 32415 0 3 0x80 netio pflogd
18677 8041 8041 73 3 0x180 poll syslogd
8041 1 8041 0 3 0x88 netio syslogd
15 0 0 0 3 0x100200 aiodoned aiodoned
14 0 0 0 3 0x100200 syncer update
13 0 0 0 3 0x100200 cleaner cleaner
12 0 0 0 3 0x100200 reaper reaper
11 0 0 0 3 0x100200 pgdaemon pagedaemon
10 0 0 0 3 0x100200 bored crypto
9 0 0 0 3 0x100200 pftm pfpurge
8 0 0 0 3 0x100200 usbevt usb1
7 0 0 0 3 0x100200 usbtsk usbtask
6 0 0 0 3 0x100200 usbevt usb0
5 0 0 0 3 0x100200 apmev apm0
4 0 0 0 3 0x100200 bored syswq
3 0 0 0 3 0x40100200 idle0
* 2 0 0 0 7 0x100200 kmthread
1 0 1 0 3 0x4080 wait init
0 -1 0 0 3 0x80200 scheduler swapper
ddb> show all pools
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
inpcbpl 228 3415 0 2402 61 0 61 61 0 8 0
plimitpl 148 25 0 11 1 0 1 1 0 8 0
synpl 192 7 0 7 1 0 1 1 0 8 1
tcpqepl 16 315 0 99 1 0 1 1 0 13 0
tcpcbpl 400 3258 0 2254 109 6 103 103 0 8 1
rtentpl 116 35 0 0 2 0 2 2 0 8 0
pfosfp 28 814 0 407 3 0 3 3 0 8 0
pfosfpen 108 1392 0 696 30 11 19 19 0 8 0
pfstateitempl 12 3332 0 1357 6 0 6 6 0 8 0
pfstatekeypl 72 3332 0 1357 47 11 36 36 0 8 0
pfstatepl 212 3332 0 1334 106 0 106 106 0 527 0
pfrulepl 1148 13 0 11 5 0 5 5 0 8 3
dirhash 1024 151 0 14 35 0 35 35 0 128 0
dino1pl 128 1891 0 46 60 0 60 60 0 8 0
ffsino 184 1891 0 46 84 0 84 84 0 8 0
nchpl 88 3221 0 78 69 0 69 69 0 8 0
vnodes 156 1902 0 0 77 0 77 77 0 8 0
namei 1024 8341 0 8341 2 0 2 2 0 8 2
wdcspl 96 15243 0 15243 1 0 1 1 0 8 1
sigapl 324 290 0 259 3 0 3 3 0 8 0
knotepl 64 6424 0 4464 32 0 32 32 0 8 0
kqueuepl 192 8 0 7 1 0 1 1 0 8 0
pipepl 72 162 0 156 1 0 1 1 0 8 0
fdescpl 300 291 0 259 3 0 3 3 0 8 0
filepl 88 6673 0 5573 25 0 25 25 0 8 0
lockfpl 56 6 0 4 1 0 1 1 0 8 0
pcredpl 20 304 0 259 1 0 1 1 0 8 0
sessionpl 48 27 0 6 1 0 1 1 0 8 0
pgrppl 24 67 0 43 1 0 1 1 0 8 0
ucredpl 80 57 0 43 1 0 1 1 0 8 0
zombiepl 72 259 0 259 1 0 1 1 0 8 1
processpl 64 304 0 259 1 0 1 1 0 8 0
procpl 316 304 0 259 4 0 4 4 0 8 0
sockpl 212 3516 0 2478 56 0 56 56 0 8 0
mcl2k 2048 56587 2451736 54237 1175 0 1175 1175 4 3072 0
mbpl 256 2581219 0 2577872 211 0 211 211 1 384 1
bufpl 172 15230 0 13456 78 0 78 78 0 8 0
anonpl 12 24559 0 19175 16 0 16 16 0 24 0
amappl 44 12998 0 11131 21 0 21 21 0 45 0
aobjpl 44 1 0 0 1 0 1 1 0 8 0
vmmpekpl 88 734 0 677 2 0 2 2 0 8 0
vmmpepl 88 36760 0 34255 55 0 55 55 0 179 0
vmsppl 180 290 0 259 2 0 2 2 0 8 0
pmappl 72 290 0 259 1 0 1 1 0 8 0
extentpl 20 258 0 207 1 0 1 1 0 8 0
phpool 48 1595 0 6 19 0 19 19 0 8 0
ddb> show bcstats
Current Buffer Cache status:
numbufs 1774, freebufs 1774
bufpages 7016, freepages 0, dirtypages 4
pendingreads 0, pendingwrites 0
ddb> show extents
extent `swap0x0000' (0x0 - 0x102db), flags=0
0x0 - 0x0
extent `swapmap' (0x1 - 0x7fffffff), flags=0
0x1 - 0x102dc
extent `ppb1 pcipmem' (0x0 - 0xffffffff), flags=0
0x0 - 0xe45fffff
0xe4700000 - 0xffffffff
extent `ppb1 pcimem' (0x0 - 0xffffffff), flags=0
0x0 - 0xfc8fffff
0xfc9ff800 - 0xfc9ff8ff
0xfc9ffc00 - 0xfc9ffcff
0xfca00000 - 0xffffffff
extent `ppb1 pciio' (0x0 - 0xffffffff), flags=0
0x0 - 0xcfff
0xd400 - 0xd4ff
0xd800 - 0xd8ff
0xe000 - 0xffffffff
extent `ppb0 pcipmem' (0x0 - 0xffffffff), flags=0
0x0 - 0xe46fffff
0xe8000000 - 0xefffffff
0xf4800000 - 0xffffffff
extent `ppb0 pcimem' (0x0 - 0xffffffff), flags=0
0x0 - 0xfc9fffff
0xfd000000 - 0xfdffffff
0xfeb00000 - 0xffffffff
extent `pcimem' (0x0 - 0xffffffff), flags=0
0x1000 - 0x1fffffff
0xe4600000 - 0xf47fffff
0xf8000000 - 0xfbffffff
0xfc900000 - 0xfeafffff
0xffb80000 - 0xffbfffff
0xfff00000 - 0xffffffff
extent `pciio' (0x0 - 0xffffffff), flags=0
0xd000 - 0xdfff
0xe800 - 0xe8ff
0xef00 - 0xef5f
0xef80 - 0xefaf
0xffa0 - 0xffaf
0x10000 - 0xffffffff
extent `PCI I/O port space' (0x0 - 0xffff), flags=0
0xd400 - 0xd4ff
0xd800 - 0xd8ff
0xe800 - 0xe8ff
0xef00 - 0xef5f
0xef80 - 0xefaf
0xffa0 - 0xffaf
extent `PCI I/O memory space' (0x0 - 0xffffffff), flags=0
0xe8000000 - 0xefffffff
0xf8000000 - 0xfbffffff
0xfc9ff800 - 0xfc9ff8ff
0xfc9ffc00 - 0xfc9ffcff
0xfd000000 - 0xfdffffff
0xfec00000 - 0xfecfffff
0xfee00000 - 0xfeefffff
0xfffe0000 - 0xffffffff
extent `iomem' (0x0 - 0xffffffff), flags=3<NOCOALESCE,FIXED>
0x400 - 0xfff
0x2000 - 0x219b
0x8000 - 0x9efff
0xa0000 - 0xbffff
0xc0000 - 0xcbbff
0xf0000 - 0xfffff
0x100000 - 0x1ffbffff
0xfc9ff800 - 0xfc9ff8ff
0xffb00000 - 0xffbfffff
extent `ioport' (0x0 - 0xffff), flags=3<NOCOALESCE,FIXED>
0x0 - 0xb
0x40 - 0x43
0x60 - 0x60
0x61 - 0x61
0x64 - 0x64
0x80 - 0x8e
0xc0 - 0xd7
0x1f0 - 0x1f7
0x2f8 - 0x2ff
0x378 - 0x37b
0x3c0 - 0x3cf
0x3d0 - 0x3df
0x3f6 - 0x3f6
0x3f8 - 0x3ff
0x400 - 0x47f
0x4d0 - 0x4d1
0xd800 - 0xd8ff
0xe800 - 0xe8ff
0xef00 - 0xef3f
0xef40 - 0xef5f
0xef80 - 0xef9f
0xefa0 - 0xefaf
0xffa0 - 0xffaf
ddb> show malloc
Type InUse MemUse HighUse Limit Requests Type Lim Kern Lim
devbuf 526 524K 524K 39321K 604 0 0
pcb 38 5K 5K 39321K 114 0 0
routetbl 76 4K 5K 39321K 220 0 0
ifaddr 53 13K 13K 39321K 54 0 0
sysctl 2 0K 0K 39321K 2 0 0
ioctlops 0 0K 4K 39321K 1449 0 0
mount 12 5K 5K 39321K 12 0 0
vnodes 1447 90K 90K 39321K 1451 0 0
UFS quota 1 16K 16K 39321K 1 0 0
UFS mount 21 40K 40K 39321K 21 0 0
shm 2 0K 0K 39321K 2 0 0
VM map 2 0K 0K 39321K 2 0 0
sem 2 0K 0K 39321K 2 0 0
dirhash 60 12K 12K 39321K 66 0 0
file desc 5 12K 16K 39321K 23 0 0
proc 11 4K 4K 39321K 11 0 0
NFS srvsock 1 0K 0K 39321K 1 0 0
NFS daemon 1 0K 0K 39321K 1 0 0
in_multi 34 1K 1K 39321K 34 0 0
ether_multi 8 0K 0K 39321K 8 0 0
ISOFS mount 1 16K 16K 39321K 1 0 0
MSDOSFS mount 1 8K 8K 39321K 1 0 0
ttys 420 308K 308K 39321K 420 0 0
exec 0 0K 1K 39321K 269 0 0
pfkey data 1 0K 0K 39321K 2 0 0
xform_data 0 0K 0K 39321K 36 0 0
pagedep 1 4K 4K 39321K 1 0 0
inodedep 1 16K 16K 39321K 1 0 0
newblk 1 0K 0K 39321K 1 0 0
VM swap 7 26K 26K 39321K 7 0 0
UVM amap 1890 263K 263K 39321K 13233 0 0
UVM aobj 2 1K 1K 39321K 2 0 0
USB 29 2K 2K 39321K 29 0 0
USB device 10 3K 3K 39321K 10 0 0
memdesc 1 4K 4K 39321K 1 0 0
crypto data 1 1K 1K 39321K 1 0 0
ip6_options 2 0K 0K 39321K 6 0 0
NDP 8 0K 0K 39321K 10 0 0
temp 52 10K 14K 39321K 10266 0 0
NTFS hash tables 1 16K 16K 39321K 1 0 0
AGP Memory 1 0K 0K 39321K 1 0 0
ddb> show map
MAP 0xd05670b4: [0xf24ec83->0x8bf04d01]
#ent=1569444401, sz=609520108, ref=604292876, version=3238536260, flags
=0x8910d9ac
uvm_fault(0xd0a09280, 0x768dc000, 0, 1) -> e
kernel: page fault trap, code=0
Faulted in DDB; continuing...
ddb> show registers
ds 0xd0c00010 end+0x13802c
es 0xd0c00010 end+0x13802c
fs 0x58
gs 0xd0be0010 end+0x11802c
edi 0xd150c960 end+0xa4497c
esi 0xd15710ac end+0xaa90c8
ebp 0xd999e960 end+0x8ed697c
ebx 0xf9
edx 0x3f8
ecx 0xd1571000 end+0xaa901c
eax 0x1
eip 0xd05670b4 Debugger+0x4
cs 0x50
eflags 0x202
esp 0xd999e960 end+0x8ed697c
ss 0xd9990010 end+0x8ec802c
Debugger+0x4: popl %ebp
ddb> show uvmexp
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
126367 VM pages: 6642 active, 2259 inactive, 0 wired, 106099 free
min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
pages 0 anon, 0 vnode, 0 vtext
freemin=4212, free-target=5616, inactive-target=0, wired-max=42122
faults=232055, traps=67583, intrs=198464, ctxswitch=49343 fpuswitch=467
softint=75622, syscalls=386758, swapins=0, swapouts=0, kmapent=27
fault counts:
noram=0, noanon=0, pgwait=0, pgrele=0
ok relocks(total)=3577(3577), anget(retries)=199263(0), amapcopy=16223
neighbor anon/obj pg=1919/34212, gets(lock/unlock)=14114/3577
cases: anon=194615, anoncow=4648, obj=12879, prcopy=1235, przero=18676
daemon and swap counts:
woke=0, revs=0, scans=0, obscans=0, anscans=0
busy=0, freed=0, reactivate=0, deactivate=0
pageouts=0, pending=0, nswget=0
nswapdev=1, nanon=0, nanonneeded=0 nfreeanon=0
swpages=66267, swpginuse=0, swpgonly=0 paging=0
kernel pointers:
objs(kern)=0xd09e7280
ddb> boot sync
panic: mtx_enter: locking against myself
Stopped at Debugger+0x4: popl %ebp
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
ddb> trace
Debugger(d08bf8bc,d999e394,d0202f2e,d999e394,0) at Debugger+0x4
panic(d0202f2e,d999e3b4,d03e9f48,d0a10b60,0) at panic+0x5d
mtx_enter(d0a10b60,0,0,d9a77324,40) at mtx_enter+0x5a
pool_get(d0a10b60,0,0,d9a77324,5a8) at pool_get+0x48
m_gethdr(1,2,0,d999e484,2) at m_gethdr+0x39
tcp_output(d9a77324,d9a5be6c,14,d999e5f0,1) at tcp_output+0x754
tcp_input(d9b08600,14,0,0,d999e68c) at tcp_input+0x2711
ipv4_input(d9b08600,0,d999e6b4,d0202089,58) at ipv4_input+0x42a
ipintr(58,10,d9990010,d03a0010,d099d040) at ipintr+0x49
Bad frame pointer: 0xd999e6b4
OpenBSD 4.8-current (GENERIC) #0: Sat Sep 11 22:36:53 YEKST 2010
r...@obsd.my.domain:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel Pentium III ("GenuineIntel" 686-class) 798 MHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PSE36,MMX,FXSR,SSE
real mem = 536178688 (511MB)
avail mem = 517439488 (493MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 11/14/00, BIOS32 rev. 0 @
0xfda74, SMBIOS rev. 2.3 @ 0xf0ea0 (56 entries)
bios0: vendor Intel Corp. version "EA81510A.86A.0040.P09.0011141019"
date 11/14/2000
bios0: Intel Corporation D815EEA
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
acpi at bios0 function 0x0 not configured
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf2f70/224 (12 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82371FB ISA" rev 0x00)
pcibios0: PCI bus #2 is the last bus
bios0: ROM list: 0xc0000/0xbc00
cpu0 at mainbus0: (uniprocessor)
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 82815 Host" rev 0x02
intelagp0 at pchb0
agp0 at intelagp0: aperture at 0xf8000000, size 0x2400000
ppb0 at pci0 dev 1 function 0 "Intel 82815 AGP" rev 0x02
pci1 at ppb0 bus 2
vga1 at pci1 dev 0 function 0 "NVIDIA GeForce2 MX" rev 0xa1
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb1 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x02
pci2 at ppb1 bus 1
re0 at pci2 dev 10 function 0 "D-Link DGE-528T" rev 0x10: RTL8110S
(0x0400), irq 11, address 00:15:e9:f0:70:27
rgephy0 at re0 phy 7: RTL8169S/8110S PHY, rev. 0
rl0 at pci2 dev 13 function 0 "Realtek 8139" rev 0x10: irq 11, address
00:50:22:bb:05:b9
rlphy0 at rl0 phy 0: RTL internal PHY
ichpcib0 at pci0 dev 31 function 0 "Intel 82801BA LPC" rev 0x02:
24-bit timer at 3579545Hz
pciide0 at pci0 dev 31 function 1 "Intel 82801BA IDE" rev 0x02: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <ST340014A>
wd0: 16-sector PIO, LBA48, 38165MB, 78163247 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
pciide0: channel 1 ignored (disabled)
uhci0 at pci0 dev 31 function 2 "Intel 82801BA USB" rev 0x02: irq 10
ichiic0 at pci0 dev 31 function 3 "Intel 82801BA SMBus" rev 0x02: irq 6
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 256MB SDRAM non-parity PC133CL2
spdmem1 at iic0 addr 0x51: 256MB SDRAM non-parity PC133CL2
uhci1 at pci0 dev 31 function 4 "Intel 82801BA USB" rev 0x02: irq 9
auich0 at pci0 dev 31 function 5 "Intel 82801BA AC97" rev 0x02: irq 6, ICH2 AC97
ac97: codec id 0x41445360 (Analog Devices AD1885)
ac97: codec features headphone, Analog Devices Phat Stereo
audio0 at auich0
isa0 at ichpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb1 at uhci1: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
biomask f765 netmask ff65 ttymask ffff
mtrr: Pentium Pro MTRR support
softraid0 at root
root on wd0a swap on wd0b dump on wd0b
WARNING: / was not properly unmounted
--
antonvm
