cri on jul 11 snapshots now, have gone thru i think 2 or 3 snapshot iterations since ~early/mid june.
first recorded/noticed incident of the 'uvm_mapent_alloc: out of static map entries' jobby was jun.16th while running a DEBUG.MP kernel i had made in attempt to catch more info on a panic we had seen (kernel/5799). we never did get that panic again, and had seen a few weird silent lockups. i didn't have syslog setup to retain logs forever back then so maybe there were UVM_amap hits and i just didn't see them. i've checked our sudo logs (those we do have long back in syslog) and the only changes that appear significant (around before jun.16) are the addition of the ices-0.4 pkg and some pf.conf edits. anyway, the DEBUG.MP kernel wasn't making life any better so we went back to the GENERIC.MP kernel from the same snapshot we were still riding on (feb.22); things were cool from then (jun.16) to jun.18 where we got the UVM_amap hit. a few ppl suggested to me offline to keep chunkin' thru snapshots so right now we're at jul 11th ones and everything has been peachy (i made a comment in the recent rtorrent misc@ thread regarding this). until weird o'clock this morning. power went out at ~midnight EDT last night and came back on ~4:45 EDT. UVM_amap is off to the races today! i've got a script i made who runs sysctl kern.malloc.kmemstat on all the critters in there and peeps to stdout when something goes up and what it is at -- i pipe that to logger on startup. http://www.ice-nine.org/jrrs/UVM.png is made from those syslog messages you can see in there the few times it has shot up like skyrockets. of note is that i seem to get the kprintf before the 'maxused' hits 'limit', per sysctl or vmstat. maybe that's not noteworthy tho.. anyway, where it gets calm on the 12th is us doing the jul.11 snapshots and popping corks and thinking the dragons were slain; we made it through monday (whereas the previous monday was nasty), but tuesday hit and we went bonkers; then after the box came back up this morning, the UVM_amap was going up very high before anyone logged in. Jul 16 04:57:20 iorek vmstat_watcher: vmstat_watcher: name: UVM_amap memuse: 395000 maxused: 3026000 limit: 39322000 Jul 16 05:37:03 iorek sshd[16018]: Accepted publickey for jrrs from 192.168.7.18 port 12655 ssh2 Jul 16 05:37:04 iorek vmstat_watcher: vmstat_watcher: name: UVM_amap memuse: 803000 maxused: 3026000 limit: 39322000 # i fix some crap i broke in pf.conf and logout Jul 16 05:45:55 iorek vmstat_watcher: vmstat_watcher: name: UVM_amap memuse: 953000 maxused: 3026000 limit: 39322000 Jul 16 06:28:50 iorek vmstat_watcher: vmstat_watcher: name: UVM_amap memuse: 10280000 maxused: 10325000 limit: 39322000 Jul 16 07:30:54 iorek vmstat_watcher: vmstat_watcher: name: UVM_amap memuse: 15285000 maxused: 18052000 limit: 39322000 Jul 16 07:46:01 iorek vmstat_watcher: vmstat_watcher: name: UVM_amap memuse: 14776000 maxused: 18052000 limit: 39322000 Jul 16 07:46:07 iorek sshd[925]: Accepted password for XXXX from XXXXX port 4484 ssh2 and keeps going up from there until it is bouncing offa ~34073000. those #s correspond to the most recent spike in the graph linked above. so barring anything else, i'll try to go to the jul.15th snapshots tonight. i have NO IDEA how to discern what it is that is tying up so much memory. i can certainly see the usage go up if i do things like make new screen(1) sessions or terminals or run stuff, but if there is any way to see who is pissing things off so much, i could maybe be more useful. i took a quick peek at systat(1) and on the iostat page i see: STATS 23770 numbufs 23769 freebufs 95057 numbufpages 0 numfreepages 24 numdirtypages 95029 numcleanpages 18014398509481983K pendingwrites 0 pendingreads 11 numwrites 0 numreads 159 cachehits where that number is very stupid high, but i checked a few other machines who aren't giving me trouble and they're showing '*', so could be that means nothing. we used to have an mfs /tmp; i did away with that when we went to the jul.11 snapshots, so that doesn't seem to be directly related. other than that about the only non-standard thing that leaps to mind about the system is having done: # sh ./MAKEDEV pty{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15} because we have run out of them a few times prior and /etc/ttys has all those bastards. here's a vmstat -m: --- Memory statistics by bucket size Size In Use Free Requests HighWater Couldfree 16 76971 29781 139521406 1280 524682 32 2434 3326 12302459 640 53921 64 4510 14754 21418581 320 1041099 128 4277 619 2706743 160 1748 256 979 109 4783949 80 17926 512 1923 181 1735210 40 326563 1024 3801 3 33657 20 3448 2048 543 3 8640 10 3118 4096 891 1 11917 5 9867 8192 536 8 2240 5 1136 16384 383 0 4338 5 0 32768 8 0 74 5 0 65536 20 0 407 5 0 131072 97 0 337 5 0 262144 33 0 204 5 0 524288 26 0 70 5 0 Memory usage type by bucket size Size Type(s) 16 devbuf, pcb, routetbl, ifaddr, sysctl, vnodes, UFS mount, sem, dirhash, ACPI, in_multi, exec, pfkey data, xform_data, VM swap, UVM amap, UVM aobj, USB, packet tags, temp 32 devbuf, pcb, routetbl, ifaddr, sysctl, sem, dirhash, ACPI, proc, VFS cluster, in_multi, ether_multi, exec, pfkey data, xform_data, VM swap, UVM amap, USB, crypto data, IPsec creds, ip6_options, temp 64 devbuf, pcb, routetbl, ifaddr, vnodes, UFS mount, sem, dirhash, ACPI, in_multi, pfkey data, xform_data, UVM amap, USB, packet tags, NDP, temp 128 devbuf, routetbl, ifaddr, iov, vnodes, UFS mount, dirhash, ACPI, NFS srvsock, ttys, exec, pfkey data, tdb, xform_data, inodedep, UVM amap, USB, USB device, crypto data, ip6_options, NDP, temp 256 devbuf, routetbl, ifaddr, sysctl, ioctlops, vnodes, UFS mount, shm, VM map, dirhash, ACPI, file desc, NFS daemon, pfkey data, tdb, inodedep, newblk, UVM amap, USB, temp 512 devbuf, pcb, ifaddr, sysctl, ioctlops, iov, UFS mount, shm, dirhash, ACPI, file desc, proc, ttys, exec, pfkey data, tdb, xform_data, UVM amap, USB device, crypto data, temp 1024 devbuf, ioctlops, mount, dirhash, ACPI, file desc, ttys, exec, tdb, UVM amap, UVM aobj, crypto data, temp 2048 devbuf, ifaddr, sysctl, ioctlops, namecache, UFS mount, proc, xform_data, VM swap, UVM amap, temp 4096 devbuf, ioctlops, UFS mount, pagedep, UVM amap, memdesc, temp 8192 devbuf, UFS mount, MSDOSFS mount, UVM amap, temp 16384 devbuf, NFS node, namecache, UFS quota, UFS mount, ISOFS mount, inodedep, indirdep, UVM amap, temp 32768 devbuf, UVM amap 65536 UVM amap 131072 namecache, UVM amap 262144 UFS mount, UVM amap 524288 VM swap, UVM amap Memory statistics by type Type Kern Type InUse MemUse HighUse Limit Requests Limit Limit Size(s) devbuf 2718 1109K 1173K 39322K 2850 0 0 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768 pcb 110 9K 9K 39322K 11514 0 0 16,32,64,512 routetbl 210 17K 17K 39322K 4807 0 0 16,32,64,128,256 ifaddr 113 24K 24K 39322K 115 0 0 16,32,64,128,256,512,2048 sysctl 3 3K 3K 39322K 5 0 0 16,32,256,512,2048 ioctlops 0 0K 4K 39322K 5963 0 0 256,512,1024,2048,4096 iov 0 0K 1K 39322K 900 0 0 128,512 mount 7 7K 7K 39322K 7 0 0 1024 NFS node 1 16K 16K 39322K 1 0 0 16384 vnodes 112 11K 179K 39322K 6968 0 0 16,64,128,256 namecache 3 146K 146K 39322K 3 0 0 2048,16384,131072 UFS quota 1 16K 16K 39322K 1 0 0 16384 UFS mount 29 203K 203K 39322K 29 0 0 16,64,128,256,512,2048,4096,8192,16384,262144 shm 2 1K 1K 39322K 2 0 0 256,512 VM map 3 1K 1K 39322K 3 0 0 256 sem 4 1K 1K 39322K 4 0 0 16,32,64 dirhash 48 11K 47K 39322K 2205 0 0 16,32,64,128,256,512,1024 ACPI 856 55K 64K 39322K 4250 0 0 16,32,64,128,256,512,1024 file desc 28 13K 24K 39322K 3017 0 0 256,512,1024 proc 38 6K 6K 39322K 38 0 0 32,512,2048 VFS cluster 0 0K 1K 39322K 454 0 0 32 NFS srvsock 1 1K 1K 39322K 1 0 0 128 NFS daemon 1 1K 1K 39322K 1 0 0 256 in_multi 98 4K 4K 39322K 98 0 0 16,32,64 ether_multi 8 1K 1K 39322K 8 0 0 32 ISOFS mount 1 16K 16K 39322K 1 0 0 16384 MSDOSFS mount 1 8K 8K 39322K 1 0 0 8192 ttys 6000 3750K 3750K 39322K 6000 0 0 128,512,1024 exec 0 0K 3K 39322K 2784610 0 0 16,32,128,512,1024 pfkey data 21 2K 3K 39322K 2186 0 0 16,32,64,128,256,512 tdb 11 10K 17K 39322K 348 0 0 128,256,512,1024 xform_data 18 1K 3K 39322K 38298 0 0 16,32,64,128,512,2048 pagedep 1 4K 4K 39322K 1 0 0 4096 inodedep 1 16K 18K 39322K 346 0 0 128,256,16384 newblk 1 1K 1K 39322K 1 0 0 256 indirdep 0 0K 32K 39322K 212 0 0 16384 VM swap 7 587K 587K 39322K 7 0 0 16,32,2048,524288 UVM amap 86185 36264K 37113K 39322K168256628 0 0 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072,262144,524288 UVM aobj 2 2K 2K 39322K 2 0 0 16,1024 USB 43 4K 4K 39322K 43 0 0 16,32,64,128,256 USB device 12 5K 5K 39322K 12 0 0 128,512 memdesc 1 4K 4K 39322K 1 0 0 4096 crypto data 42 8K 13K 39322K 1132 0 0 32,128,512,1024 IPsec creds 36 2K 2K 39322K 904 0 0 32 packet tags 0 0K 1K 39322K 60368 0 0 16,64 ip6_options 549 69K 69K 39322K 562 0 0 32,128 NDP 21 2K 3K 39322K 25 0 0 64,128 temp 169 13K 45K 39322K 11335400 0 0 16,32,64,128,256,512,1024,2048,4096,8192,16384 Memory Totals: In Use Free Requests 42410K 1764K 182530332 Memory resource pool statistics Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle extentpl 20 209 0 189 1 0 1 1 0 8 0 phpool 32 1346 0 476 7 0 7 7 0 8 0 pmappl 84 1704093 0 1703940 4 0 4 4 0 8 0 vmsppl 188 1704093 0 1703940 8 0 8 8 0 8 0 vmmpepl 88 147886152 0 147851389 4994 4032 962 989 0 179 179 vmmpekpl 88 4436695 0 4436621 2 0 2 2 0 8 0 aobjpl 52 1 0 0 1 0 1 1 0 8 0 amappl 44 54083282 0 54055198 3113 2698 415 419 0 45 16 anonpl 16 109391204 0 109331937 2130 1597 533 809 0 244 173 bufpl 144 6158899 0 6135905 4540 3671 869 888 0 8 5 mbpl 256 1748512 0 1748038 44 0 44 44 1 384 10 mclpl 2048 467298 0 466978 191 0 191 191 4 3072 29 sockpl 212 1930669 0 1930420 14 0 14 14 0 8 0 procpl 344 1704112 0 1703940 17 0 17 17 0 8 1 processpl 20 1704112 0 1703940 1 0 1 1 0 8 0 zombiepl 72 1703940 0 1703940 1 0 1 1 0 8 1 ucredpl 80 5546 0 5421 3 0 3 3 0 8 0 pgrppl 24 6284 0 6181 1 0 1 1 0 8 0 sessionpl 48 3498 0 3414 2 0 2 2 0 8 1 pcredpl 24 1704112 0 1703940 2 0 2 2 0 8 0 lockfpl 52 442207 0 442192 1 0 1 1 0 8 0 filepl 88 6760026 0 6759416 14 0 14 14 0 8 0 fdescpl 296 1704094 0 1703940 13 0 13 13 0 8 1 pipepl 72 948160 0 948101 2 0 2 2 0 8 0 kqueuepl 192 7 0 7 1 0 1 1 0 8 1 knotepl 64 14 0 14 1 0 1 1 0 8 1 sigapl 316 1704093 0 1703940 14 0 14 14 0 8 1 wqtasks 20 43014 0 43014 1 0 1 1 0 8 1 scxspl 132 1608106 0 1608106 1 0 1 1 0 8 1 pfiaddrpl 100 184 0 0 5 0 5 5 0 8 0 namei 1024 9701823 0 9701823 3 0 3 3 0 8 3 vnodes 148 5927 0 0 220 0 220 220 0 8 0 nchpl 72 38541 0 35578 53 0 53 53 0 8 0 ffsino 184 248951 0 243033 290 21 269 269 0 8 0 dino1pl 128 34021 0 33550 245 203 42 125 0 8 8 dino2pl 256 214930 0 209483 826 476 350 351 0 8 4 pagedeppl 68 7308 0 7307 3 0 3 3 0 8 2 inodedeppl 84 21147 0 21143 106 98 8 59 0 8 7 newblkpl 36 26461 0 26461 1 0 1 1 0 8 1 bmsafemappl 32 7243 0 7243 1 0 1 1 0 8 1 allocdirectpl 76 11081 0 11078 14 6 8 11 0 8 7 indirdeppl 28 317 0 316 1 0 1 1 0 8 0 allocindirpl 60 15380 0 15376 41 33 8 33 0 8 7 freefragpl 36 853 0 853 1 0 1 1 0 8 1 freeblkspl 168 12298 0 12298 206 198 8 109 0 8 8 freefilepl 28 13459 0 13459 20 12 8 20 0 8 8 diraddpl 32 11342 0 11341 3 0 3 3 0 8 2 mkdirpl 28 1924 0 1924 2 0 2 2 0 8 2 dirrempl 32 16760 0 16760 23 15 8 23 0 8 8 newdirblkpl 16 5 0 5 1 0 1 1 0 8 1 dirhash 1024 4261 0 4051 128 0 128 128 0 128 75 semapl 68 2 0 0 1 0 1 1 0 8 0 semupl 100 158 0 157 1 0 1 1 0 8 0 pfrulepl 848 334 0 11 84 0 84 84 0 8 2 pfstatepl 192 13358 0 13322 26 0 26 26 0 477 22 pfstatekeypl 72 14171 0 14135 10 2 8 10 0 8 5 pfstateitempl 12 14171 0 14135 2 0 2 2 0 8 1 pfaltqpl 224 24 0 8 1 0 1 1 0 8 0 pfpooladdrpl 68 8 0 0 1 0 1 1 0 8 0 pfrktable 1240 1054 0 1034 9 0 9 9 0 334 1 pfrkentry 96 4185 0 2168 52 0 52 52 0 4762 3 pfrkentry2 96 30 0 5 1 0 1 1 0 8 0 pfosfpen 108 2088 0 1392 41 22 19 19 0 8 0 pfosfp 28 1221 0 814 3 0 3 3 0 8 0 strprocpl 2268 4 0 3 3 0 3 3 0 8 2 strpolpl 28 1 0 0 1 0 1 1 0 8 0 rtentpl 116 106 0 16 3 0 3 3 0 8 0 ipqepl 20 504 0 504 1 0 1 1 0 8 1 ipqpl 24 252 0 252 1 0 1 1 0 8 1 tcpcbpl 400 19122 0 19005 14 0 14 14 0 8 1 tcpqepl 16 6617 0 6617 1 0 1 1 0 13 1 sackhlpl 20 160 0 160 1 0 1 1 0 162 1 synpl 184 7566 0 7566 1 0 1 1 0 8 1 plimitpl 152 1703806 0 1703755 3 0 3 3 0 8 1 inpcbpl 220 1919164 0 1919016 10 0 10 10 0 8 1 ipsec policy 212 45 0 31 1 0 1 1 0 8 0 cryptop 48 30726 0 30726 1 0 1 1 0 8 1 cryptodesc 56 61452 0 61452 1 0 1 1 0 8 1 In use 13928K, total allocated 32756K; utilization 42.5% -- jared
