On Wed, Feb 11, 2026 at 10:24:20AM -0300, K R wrote:
> Same panic, this time with show malloc included. Please let me know
> if you need additional ddb commands next time.
>
> Thanks,
> --Kor
>
> ddb{1}> show panic
> ddb{1}> *cpu0: malloc: out of space in kmem_map
Something is using all memory in kmem_map and then the system goes boom.
It is not malloc, the show malloc does not show any bucket that consumes a
lot of mem.
show all pools is another place memory may hide. Since multi page pools
use the kmem_map as well.
You can actually watch this during runtime and look if something is slowly
growing upwards. vmstat -m is great for that.
> ddb{1}> tr
> ddb{1}> savectx() at savectx+0xae
> end of kernel
> end trace frame: 0x7ae419eb6220, count: -1
>
> ddb{1}> ps
> ddb{1}> PID TID PPID UID S FLAGS WAIT COMMAND
> 56384 64771 1 0 3 0x100083 ttyin getty
> 72686 27501 1 0 3 0x100083 ttyin getty
> 36459 175760 1 0 3 0x100083 ttyin getty
> 81450 57092 1 0 3 0x100083 ttyin getty
> 96634 204314 1 0 3 0x100083 ttyin ksh
> 65510 363957 1 0 3 0x100098 kqread cron
> 88845 514315 1 10000 3 0x80 kqread python3.12
> 97326 174375 1 10000 3 0x80 kqread python3.12
> 22253 67759 1 10000 3 0x80 kqread python3.12
> 35444 44336 1 10000 3 0x90 kqread python3.12
> 29275 40912 1 10000 3 0x90 kqread python3.12
> 35724 333760 1 10000 3 0x80 kqread python3.12
> 50769 288439 1 10000 3 0x90 kqread python3.12
> 82066 241649 1 10000 3 0x10 netlock python3.12
> 53406 363520 1 10000 3 0x80 kqread python3.12
> 74524 458639 1 10000 3 0x90 kqread python3.12
> 74524 199419 1 10000 3 0x4000090 fsleep python3.12
> 74524 83014 1 10000 3 0x4000090 fsleep python3.12
> *26047 417104 1 76 7 0x1000010 p0f3
> 763 241728 1 760 3 0x90 kqread snmpd
> 57219 438582 3052 95 3 0x1100092 kqread smtpd
> 38463 126657 3052 103 3 0x1100092 kqread smtpd
> 64172 119575 3052 95 3 0x1100092 kqread smtpd
> 51265 104678 3052 95 3 0x100092 kqread smtpd
> 43007 287543 3052 95 3 0x1100092 kqread smtpd
> 40890 233284 3052 95 3 0x1100092 kqread smtpd
> 3052 172226 1 0 3 0x100080 kqread smtpd
> 26543 98368 1 0 3 0x88 kqread sshd
> 79246 67776 0 0 3 0x14200 acct acct
> 4238 13076 1 0 3 0x100080 kqread ntpd
> 68994 428671 87754 83 3 0x100092 kqread ntpd
> 87754 161036 1 83 3 0x1100092 kqread ntpd
> 30143 296065 1 53 3 0x1000090 kqread unbound
> 6513 123759 16181 74 3 0x1100092 bpf pflogd
> 16181 56823 1 0 3 0x80 sbwait pflogd
> 77125 81173 61465 73 3 0x1100090 kqread syslogd
> 61465 258272 1 0 3 0x100082 sbwait syslogd
> 88652 122618 0 0 3 0x14200 bored smr
> 17119 301982 0 0 3 0x14200 pgzero zerothread
> 75306 268311 0 0 3 0x14200 aiodoned aiodoned
> 89902 288787 0 0 3 0x14200 syncer update
> 77149 53678 0 0 3 0x14200 cleaner cleaner
> 74601 396045 0 0 3 0x14200 reaper reaper
> 14050 464621 0 0 3 0x14200 pgdaemon pagedaemon
> 59034 421709 0 0 3 0x14200 bored wsdisplay0
> 42208 103791 0 0 3 0x14200 usbtsk usbtask
> 3252 461912 0 0 3 0x14200 usbatsk usbatsk
> 58242 495231 0 0 3 0x40014200 acpi0 acpi0
> 57561 206381 0 0 3 0x40014200 idle1
> 61023 369667 0 0 3 0x14200 bored softnet1
> 34197 326659 0 0 3 0x14200 netlock softnet0
> 52256 164467 0 0 3 0x14200 bored systqmp
> 71398 21045 0 0 7 0x14200 systq
> 6890 354256 0 0 3 0x14200 tmoslp softclockmp
> 60049 384374 0 0 3 0x40014200 tmoslp softclock
> 74022 123588 0 0 3 0x40014200 idle0
> 1 198910 0 0 3 0x82 wait init
> 0 0 -1 0 3 0x10200 scheduler swapper
>
> ddb{1}> show reg
> rdi 0xffffffff829f04f8 kprintf_mutex
> rsi 0x5
> rbp 0xffff80002ddbdc10
> rbx 0
> rdx 0
> rcx 0x1900 __ALIGN_SIZE+0x900
> rax 0x3c
> r8 0x70000 acpi_pdirpa+0x5be71
> r9 0xffff80002dc3b000
> r10 0
> r11 0x986d6894b8c166b
> r12 0
> r13 0
> r14 0xffff80002dd302b8
> r15 0
> rip 0xffffffff823723ee savectx+0xae
> cs 0x8
> rflags 0x46
> rsp 0xffff80002ddbdb90
> ss 0x10
> savectx+0xae: movl $0,%gs:0x688
>
> ddb{1}> show malloc
> ddb{1}> Type InUse MemUse HighUse Limit Requests Type Lim
> devbuf 2194 5224K 5289K 186616K 8223 0
> pcb 17 8208K 12304K 186616K 45 0
> rtable 2173 57K 60K 186616K 52874 0
> pf 20 39K 55K 186616K 1586 0
> ifaddr 400 97K 97K 186616K 400 0
> ifgroup 27 1K 1K 186616K 30 0
> sysctl 4 1K 9K 186616K 10 0
> counters 54 35K 35K 186616K 54 0
> ioctlops 0 0K 4K 186616K 39816 0
> mount 6 6K 6K 186616K 6 0
> vnodes 1263 79K 79K 186616K 1339 0
> UFS quota 1 32K 32K 186616K 1 0
> UFS mount 25 65K 65K 186616K 25 0
> shm 2 1K 1K 186616K 2 0
> VM map 2 1K 1K 186616K 2 0
> sem 2 0K 0K 186616K 2 0
> dirhash 351 68K 68K 186616K 381 0
> ACPI 3761 457K 633K 186616K 18596 0
> file desc 12 20K 21K 186616K 34 0
> proc 96 76K 93K 186616K 4935 0
> NFS srvsock 1 0K 0K 186616K 1 0
> NFS daemon 1 16K 16K 186616K 1 0
> in_multi 531 29K 29K 186616K 531 0
> ether_multi 130 8K 8K 186616K 130 0
> ISOFS mount 1 32K 32K 186616K 1 0
> MSDOSFS mount 1 16K 16K 186616K 1 0
> ttys 37 97K 97K 186616K 37 0
> exec 0 0K 1K 186616K 39087 0
> fusefs mount 1 32K 32K 186616K 1 0
> tdb 3 0K 0K 186616K 3 0
> VM swap 8 582K 584K 186616K 10 0
> UVM amap 4623 584K 1015K 186616K 346048 0
> UVM aobj 3 2K 2K 186616K 3 0
> pinsyscall 68 136K 210K 186616K 108216 0
> USB 21 15K 15K 186616K 25 0
> USB device 8 0K 0K 186616K 8 0
> USB HC 1 0K 0K 186616K 1 0
> memdesc 1 4K 4K 186616K 1 0
> crypto data 1 1K 1K 186616K 1 0
> ip6_options 1 0K 3K 186616K 21040 0
> NDP 5 0K 16K 186616K 134 0
> temp 10 8622K 8751K 186616K 9681105 0
> kqueue 37 70K 80K 186616K 1586 0
> SYN cache 2 16K 16K 186616K 2 0
>
> On Mon, Jan 19, 2026 at 4:23 PM K R <[email protected]> wrote:
> >
> > >Synopsis: panic: malloc: out of space in kmem_map
> > >Category: kernel amd64
> > >Environment:
> > System : OpenBSD 7.8
> > Details : OpenBSD 7.8 (GENERIC.MP) #1: Sat Nov 29 11:02:59 MST
> > 2025
> >
> > [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> >
> > Architecture: OpenBSD.amd64
> > Machine : amd64
> > >Description:
> >
> > The machine is running 7.8 + syspatches under VMware:
> >
> > hw.model=Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
> > hw.vendor=VMware, Inc.
> > hw.product=VMware20,1
> > hw.physmem=4277600256
> > hw.ncpufound=2
> > hw.ncpuonline=2
> >
> > and panics with a out of space in kmem_map message. Panic, trace and
> > ps shown below.
> >
> > I wish I could show malloc, but the machine is in a remote location
> > and these are the only ddb commands I got before the operator decided
> > to reboot.
> >
> > panic: malloc: out of space in kmem_map
> > Stopped at db_enter+0x14: popq %rbp
> > TID PID UID PRFLAGS PFLAGS CPU COMMAND
> > *327273 39043 0 0x14000 0x200 0 systq
> >
> > db_enter() at db_enter+0x14
> > panic(ffffffff82573eac) at panic+0xd5
> > malloc(2a39,2,9) at malloc+0x823
> > vmt_nicinfo_task(ffff8000000f8800) at vmt_nicinfo_task+0xec
> > taskq_thread(ffffffff82a19e10) at taskq_thread+0x129
> > end trace frame: 0x0, count: -5
> >
> > PID TID PPID UID S FLAGS WAIT COMMAND
> > 34434 429242 1 0 3 0x100083 ttyin getty
> > 45351 273621 1 0 3 0x100083 ttyin getty
> > 15766 13242 1 0 3 0x100083 ttyin getty
> > 22501 485732 1 0 3 0x100083 ttyin getty
> > 21121 14373 1 0 3 0x100083 ttyin getty
> > 80812 223396 1 0 3 0x100098 kqread cron
> > 38632 393850 1 10000 3 0x80 kqread python3.12
> > 50241 286369 1 10000 3 0x80 kqread python3.12
> > 47425 216199 1 10000 3 0x80 kqread python3.12
> > 15348 391586 1 10000 3 0x90 kqread python3.12
> > 83699 242757 1 10000 3 0x90 kqread python3.12
> > 85859 155143 1 10000 3 0x80 kqread python3.12
> > 140 96058 1 10000 3 0x90 kqread python3.12
> > 16478 159685 1 10000 3 0x90 kqread python3.12
> > 83476 226912 1 10000 3 0x80 kqread python3.12
> > 90068 368113 1 10000 3 0x90 kqread python3.12
> > 48780 36449 1 76 3 0x1000090 kqread p0f3
> > 41298 290255 1 760 3 0x90 kqread snmpd
> > 47065 410042 45934 95 3 0x1100092 kqread smtpd
> > 69131 288318 45934 103 3 0x1100092 kqread smtpd
> > 16340 95197 45934 95 3 0x1100092 kqread smtpd
> > 93858 467609 45934 95 3 0x100092 kqread smtpd
> > 77301 381360 45934 95 3 0x1100092 kqread smtpd
> > 21497 499144 45934 95 3 0x1100092 kqread smtpd
> > 45934 163643 1 0 3 0x100080 kqread smtpd
> > 16761 447799 1 0 3 0x88 kqread sshd
> > 57214 310491 0 0 3 0x14200 acct acct
> > 56721 278490 1 0 3 0x100080 kqread ntpd
> > 57480 393701 1368 83 3 0x100092 kqread ntpd
> > 1368 281100 1 83 3 0x1100092 kqread ntpd
> > 24741 184818 1 53 3 0x1000090 kqread unbound
> > 74565 391331 50900 74 3 0x1100092 bpf pflogd
> > 50900 22496 1 0 3 0x80 sbwait pflogd
> > 65059 173120 1614 73 3 0x1100090 kqread syslogd
> > 1614 223274 1 0 3 0x100082 sbwait syslogd
> > 12330 136338 0 0 3 0x14200 bored smr
> > 60396 73572 0 0 3 0x14200 pgzero zerothread
> > 46408 208812 0 0 3 0x14200 aiodoned aiodoned
> > 44729 344674 0 0 3 0x14200 syncer update
> > 61833 363291 0 0 3 0x14200 cleaner cleaner
> > 52556 361252 0 0 3 0x14200 reaper reaper
> > 64026 456140 0 0 3 0x14200 pgdaemon pagedaemon
> > 75515 242523 0 0 3 0x14200 bored wsdisplay0
> > 14784 395040 0 0 3 0x14200 usbtsk usbtask
> > 78465 209741 0 0 3 0x14200 usbatsk usbatsk
> > 70654 374635 0 0 3 0x40014200 acpi0 acpi0
> > 48248 77950 0 0 7 0x40014200 idle1
> > 21581 78258 0 0 3 0x14200 bored softnet1
> > 42528 246111 0 0 3 0x14200 netlock softnet0
> > 84149 341522 0 0 3 0x14200 bored systqmp
> > *39043 327273 0 0 7 0x14200 systq
> > 50129 384305 0 0 3 0x14200 netlock softclockmp
> > 86142 318003 0 0 3 0x40014200 tmoslp softclock
> > 95618 290560 0 0 3 0x40014200 idle0
> > 1 184077 0 0 3 0x82 wait init
> > 0 0 -1 0 3 0x10200 scheduler swapper
> >
> > >How-To-Repeat:
> >
> > It seems to be related to VMWare when the machine is under
> > medium/heavy network traffic. Other baremetal machines with similar
> > daemons/traffic work just fine.
> >
> > Any command (vmstat, systat, etc), while the machine is alive, that
> > could help?
> >
> > Thanks,
> > --Kor
> >
> > >Fix:
> >
> > Unknown.
>
--
:wq Claudio
