On 2026/02/12 09:38, K R wrote:
> On Wed, Feb 11, 2026 at 10:50 AM Claudio Jeker <[email protected]>
> wrote:
> >
> > On Wed, Feb 11, 2026 at 10:24:20AM -0300, K R wrote:
> > > Same panic, this time with show malloc included. Please let me know
> > > if you need additional ddb commands next time.
> > >
> > > Thanks,
> > > --Kor
> > >
> > > ddb{1}> show panic
> > > ddb{1}> *cpu0: malloc: out of space in kmem_map
> >
> >
> > Something is using all memory in kmem_map and then the system goes boom.
> > It is not malloc, the show malloc does not show any bucket that consumes a
> > lot of mem.
> >
> > show all pools is another place memory may hide. Since multi page pools
>
> Thanks, I'll submit show all pools (and perhaps show uvmexp?) -- the
> machine paniced again, waiting for the ddb commands to be run by the
> remote admin.
>
> > use the kmem_map as well.
> >
> > You can actually watch this during runtime and look if something is slowly
> > growing upwards. vmstat -m is great for that.
>
> It may be unrelated, but it caught my eye that the only pool with Fail
> request if this:
>
> Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg
> Idle
> pfstate 384 27749088 7613228 7622 391809 390986 823 10001 0 8
> 8
pfctl -si might help too
> dmesg was showing:
> uvm_mapent_alloc: out of static map entries
>
> >
> > > ddb{1}> tr
> > > ddb{1}> savectx() at savectx+0xae
> > > end of kernel
> > > end trace frame: 0x7ae419eb6220, count: -1
> > >
> > > ddb{1}> ps
> > > ddb{1}> PID TID PPID UID S FLAGS WAIT COMMAND
> > > 56384 64771 1 0 3 0x100083 ttyin getty
> > > 72686 27501 1 0 3 0x100083 ttyin getty
> > > 36459 175760 1 0 3 0x100083 ttyin getty
> > > 81450 57092 1 0 3 0x100083 ttyin getty
> > > 96634 204314 1 0 3 0x100083 ttyin ksh
> > > 65510 363957 1 0 3 0x100098 kqread cron
> > > 88845 514315 1 10000 3 0x80 kqread python3.12
> > > 97326 174375 1 10000 3 0x80 kqread python3.12
> > > 22253 67759 1 10000 3 0x80 kqread python3.12
> > > 35444 44336 1 10000 3 0x90 kqread python3.12
> > > 29275 40912 1 10000 3 0x90 kqread python3.12
> > > 35724 333760 1 10000 3 0x80 kqread python3.12
> > > 50769 288439 1 10000 3 0x90 kqread python3.12
> > > 82066 241649 1 10000 3 0x10 netlock python3.12
> > > 53406 363520 1 10000 3 0x80 kqread python3.12
> > > 74524 458639 1 10000 3 0x90 kqread python3.12
> > > 74524 199419 1 10000 3 0x4000090 fsleep python3.12
> > > 74524 83014 1 10000 3 0x4000090 fsleep python3.12
> > > *26047 417104 1 76 7 0x1000010 p0f3
> > > 763 241728 1 760 3 0x90 kqread snmpd
> > > 57219 438582 3052 95 3 0x1100092 kqread smtpd
> > > 38463 126657 3052 103 3 0x1100092 kqread smtpd
> > > 64172 119575 3052 95 3 0x1100092 kqread smtpd
> > > 51265 104678 3052 95 3 0x100092 kqread smtpd
> > > 43007 287543 3052 95 3 0x1100092 kqread smtpd
> > > 40890 233284 3052 95 3 0x1100092 kqread smtpd
> > > 3052 172226 1 0 3 0x100080 kqread smtpd
> > > 26543 98368 1 0 3 0x88 kqread sshd
> > > 79246 67776 0 0 3 0x14200 acct acct
> > > 4238 13076 1 0 3 0x100080 kqread ntpd
> > > 68994 428671 87754 83 3 0x100092 kqread ntpd
> > > 87754 161036 1 83 3 0x1100092 kqread ntpd
> > > 30143 296065 1 53 3 0x1000090 kqread unbound
> > > 6513 123759 16181 74 3 0x1100092 bpf pflogd
> > > 16181 56823 1 0 3 0x80 sbwait pflogd
> > > 77125 81173 61465 73 3 0x1100090 kqread syslogd
> > > 61465 258272 1 0 3 0x100082 sbwait syslogd
> > > 88652 122618 0 0 3 0x14200 bored smr
> > > 17119 301982 0 0 3 0x14200 pgzero zerothread
> > > 75306 268311 0 0 3 0x14200 aiodoned aiodoned
> > > 89902 288787 0 0 3 0x14200 syncer update
> > > 77149 53678 0 0 3 0x14200 cleaner cleaner
> > > 74601 396045 0 0 3 0x14200 reaper reaper
> > > 14050 464621 0 0 3 0x14200 pgdaemon pagedaemon
> > > 59034 421709 0 0 3 0x14200 bored wsdisplay0
> > > 42208 103791 0 0 3 0x14200 usbtsk usbtask
> > > 3252 461912 0 0 3 0x14200 usbatsk usbatsk
> > > 58242 495231 0 0 3 0x40014200 acpi0 acpi0
> > > 57561 206381 0 0 3 0x40014200 idle1
> > > 61023 369667 0 0 3 0x14200 bored softnet1
> > > 34197 326659 0 0 3 0x14200 netlock softnet0
> > > 52256 164467 0 0 3 0x14200 bored systqmp
> > > 71398 21045 0 0 7 0x14200 systq
> > > 6890 354256 0 0 3 0x14200 tmoslp softclockmp
> > > 60049 384374 0 0 3 0x40014200 tmoslp softclock
> > > 74022 123588 0 0 3 0x40014200 idle0
> > > 1 198910 0 0 3 0x82 wait init
> > > 0 0 -1 0 3 0x10200 scheduler swapper
> > >
> > > ddb{1}> show reg
> > > rdi 0xffffffff829f04f8 kprintf_mutex
> > > rsi 0x5
> > > rbp 0xffff80002ddbdc10
> > > rbx 0
> > > rdx 0
> > > rcx 0x1900 __ALIGN_SIZE+0x900
> > > rax 0x3c
> > > r8 0x70000 acpi_pdirpa+0x5be71
> > > r9 0xffff80002dc3b000
> > > r10 0
> > > r11 0x986d6894b8c166b
> > > r12 0
> > > r13 0
> > > r14 0xffff80002dd302b8
> > > r15 0
> > > rip 0xffffffff823723ee savectx+0xae
> > > cs 0x8
> > > rflags 0x46
> > > rsp 0xffff80002ddbdb90
> > > ss 0x10
> > > savectx+0xae: movl $0,%gs:0x688
> > >
> > > ddb{1}> show malloc
> > > ddb{1}> Type InUse MemUse HighUse Limit Requests Type Lim
> > > devbuf 2194 5224K 5289K 186616K 8223 0
> > > pcb 17 8208K 12304K 186616K 45 0
> > > rtable 2173 57K 60K 186616K 52874 0
> > > pf 20 39K 55K 186616K 1586 0
> > > ifaddr 400 97K 97K 186616K 400 0
> > > ifgroup 27 1K 1K 186616K 30 0
> > > sysctl 4 1K 9K 186616K 10 0
> > > counters 54 35K 35K 186616K 54 0
> > > ioctlops 0 0K 4K 186616K 39816 0
> > > mount 6 6K 6K 186616K 6 0
> > > vnodes 1263 79K 79K 186616K 1339 0
> > > UFS quota 1 32K 32K 186616K 1 0
> > > UFS mount 25 65K 65K 186616K 25 0
> > > shm 2 1K 1K 186616K 2 0
> > > VM map 2 1K 1K 186616K 2 0
> > > sem 2 0K 0K 186616K 2 0
> > > dirhash 351 68K 68K 186616K 381 0
> > > ACPI 3761 457K 633K 186616K 18596 0
> > > file desc 12 20K 21K 186616K 34 0
> > > proc 96 76K 93K 186616K 4935 0
> > > NFS srvsock 1 0K 0K 186616K 1 0
> > > NFS daemon 1 16K 16K 186616K 1 0
> > > in_multi 531 29K 29K 186616K 531 0
> > > ether_multi 130 8K 8K 186616K 130 0
> > > ISOFS mount 1 32K 32K 186616K 1 0
> > > MSDOSFS mount 1 16K 16K 186616K 1 0
> > > ttys 37 97K 97K 186616K 37 0
> > > exec 0 0K 1K 186616K 39087 0
> > > fusefs mount 1 32K 32K 186616K 1 0
> > > tdb 3 0K 0K 186616K 3 0
> > > VM swap 8 582K 584K 186616K 10 0
> > > UVM amap 4623 584K 1015K 186616K 346048 0
> > > UVM aobj 3 2K 2K 186616K 3 0
> > > pinsyscall 68 136K 210K 186616K 108216 0
> > > USB 21 15K 15K 186616K 25 0
> > > USB device 8 0K 0K 186616K 8 0
> > > USB HC 1 0K 0K 186616K 1 0
> > > memdesc 1 4K 4K 186616K 1 0
> > > crypto data 1 1K 1K 186616K 1 0
> > > ip6_options 1 0K 3K 186616K 21040 0
> > > NDP 5 0K 16K 186616K 134 0
> > > temp 10 8622K 8751K 186616K 9681105 0
> > > kqueue 37 70K 80K 186616K 1586 0
> > > SYN cache 2 16K 16K 186616K 2 0
> > >
> > > On Mon, Jan 19, 2026 at 4:23 PM K R <[email protected]> wrote:
> > > >
> > > > >Synopsis: panic: malloc: out of space in kmem_map
> > > > >Category: kernel amd64
> > > > >Environment:
> > > > System : OpenBSD 7.8
> > > > Details : OpenBSD 7.8 (GENERIC.MP) #1: Sat Nov 29 11:02:59
> > > > MST 2025
> > > >
> > > > [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > > >
> > > > Architecture: OpenBSD.amd64
> > > > Machine : amd64
> > > > >Description:
> > > >
> > > > The machine is running 7.8 + syspatches under VMware:
> > > >
> > > > hw.model=Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
> > > > hw.vendor=VMware, Inc.
> > > > hw.product=VMware20,1
> > > > hw.physmem=4277600256
> > > > hw.ncpufound=2
> > > > hw.ncpuonline=2
> > > >
> > > > and panics with a out of space in kmem_map message. Panic, trace and
> > > > ps shown below.
> > > >
> > > > I wish I could show malloc, but the machine is in a remote location
> > > > and these are the only ddb commands I got before the operator decided
> > > > to reboot.
> > > >
> > > > panic: malloc: out of space in kmem_map
> > > > Stopped at db_enter+0x14: popq %rbp
> > > > TID PID UID PRFLAGS PFLAGS CPU COMMAND
> > > > *327273 39043 0 0x14000 0x200 0 systq
> > > >
> > > > db_enter() at db_enter+0x14
> > > > panic(ffffffff82573eac) at panic+0xd5
> > > > malloc(2a39,2,9) at malloc+0x823
> > > > vmt_nicinfo_task(ffff8000000f8800) at vmt_nicinfo_task+0xec
> > > > taskq_thread(ffffffff82a19e10) at taskq_thread+0x129
> > > > end trace frame: 0x0, count: -5
> > > >
> > > > PID TID PPID UID S FLAGS WAIT COMMAND
> > > > 34434 429242 1 0 3 0x100083 ttyin getty
> > > > 45351 273621 1 0 3 0x100083 ttyin getty
> > > > 15766 13242 1 0 3 0x100083 ttyin getty
> > > > 22501 485732 1 0 3 0x100083 ttyin getty
> > > > 21121 14373 1 0 3 0x100083 ttyin getty
> > > > 80812 223396 1 0 3 0x100098 kqread cron
> > > > 38632 393850 1 10000 3 0x80 kqread python3.12
> > > > 50241 286369 1 10000 3 0x80 kqread python3.12
> > > > 47425 216199 1 10000 3 0x80 kqread python3.12
> > > > 15348 391586 1 10000 3 0x90 kqread python3.12
> > > > 83699 242757 1 10000 3 0x90 kqread python3.12
> > > > 85859 155143 1 10000 3 0x80 kqread python3.12
> > > > 140 96058 1 10000 3 0x90 kqread python3.12
> > > > 16478 159685 1 10000 3 0x90 kqread python3.12
> > > > 83476 226912 1 10000 3 0x80 kqread python3.12
> > > > 90068 368113 1 10000 3 0x90 kqread python3.12
> > > > 48780 36449 1 76 3 0x1000090 kqread p0f3
> > > > 41298 290255 1 760 3 0x90 kqread snmpd
> > > > 47065 410042 45934 95 3 0x1100092 kqread smtpd
> > > > 69131 288318 45934 103 3 0x1100092 kqread smtpd
> > > > 16340 95197 45934 95 3 0x1100092 kqread smtpd
> > > > 93858 467609 45934 95 3 0x100092 kqread smtpd
> > > > 77301 381360 45934 95 3 0x1100092 kqread smtpd
> > > > 21497 499144 45934 95 3 0x1100092 kqread smtpd
> > > > 45934 163643 1 0 3 0x100080 kqread smtpd
> > > > 16761 447799 1 0 3 0x88 kqread sshd
> > > > 57214 310491 0 0 3 0x14200 acct acct
> > > > 56721 278490 1 0 3 0x100080 kqread ntpd
> > > > 57480 393701 1368 83 3 0x100092 kqread ntpd
> > > > 1368 281100 1 83 3 0x1100092 kqread ntpd
> > > > 24741 184818 1 53 3 0x1000090 kqread unbound
> > > > 74565 391331 50900 74 3 0x1100092 bpf pflogd
> > > > 50900 22496 1 0 3 0x80 sbwait pflogd
> > > > 65059 173120 1614 73 3 0x1100090 kqread syslogd
> > > > 1614 223274 1 0 3 0x100082 sbwait syslogd
> > > > 12330 136338 0 0 3 0x14200 bored smr
> > > > 60396 73572 0 0 3 0x14200 pgzero zerothread
> > > > 46408 208812 0 0 3 0x14200 aiodoned aiodoned
> > > > 44729 344674 0 0 3 0x14200 syncer update
> > > > 61833 363291 0 0 3 0x14200 cleaner cleaner
> > > > 52556 361252 0 0 3 0x14200 reaper reaper
> > > > 64026 456140 0 0 3 0x14200 pgdaemon pagedaemon
> > > > 75515 242523 0 0 3 0x14200 bored wsdisplay0
> > > > 14784 395040 0 0 3 0x14200 usbtsk usbtask
> > > > 78465 209741 0 0 3 0x14200 usbatsk usbatsk
> > > > 70654 374635 0 0 3 0x40014200 acpi0 acpi0
> > > > 48248 77950 0 0 7 0x40014200 idle1
> > > > 21581 78258 0 0 3 0x14200 bored softnet1
> > > > 42528 246111 0 0 3 0x14200 netlock softnet0
> > > > 84149 341522 0 0 3 0x14200 bored systqmp
> > > > *39043 327273 0 0 7 0x14200 systq
> > > > 50129 384305 0 0 3 0x14200 netlock softclockmp
> > > > 86142 318003 0 0 3 0x40014200 tmoslp softclock
> > > > 95618 290560 0 0 3 0x40014200 idle0
> > > > 1 184077 0 0 3 0x82 wait init
> > > > 0 0 -1 0 3 0x10200 scheduler swapper
> > > >
> > > > >How-To-Repeat:
> > > >
> > > > It seems to be related to VMWare when the machine is under
> > > > medium/heavy network traffic. Other baremetal machines with similar
> > > > daemons/traffic work just fine.
> > > >
> > > > Any command (vmstat, systat, etc), while the machine is alive, that
> > > > could help?
> > > >
> > > > Thanks,
> > > > --Kor
> > > >
> > > > >Fix:
> > > >
> > > > Unknown.
> > >
> >
> > --
> > :wq Claudio
>
