Hi,
I gathered more info, please see below. The observation is that bhyve
allocate kernel memory on the 2nd failing run. It looks like this
memory is lost until the next reboot because the attempt to unload the
vmm module fails (more details below).
BTW, I'm testing this on an up-to-date OpenIndiana.
On Mon, Jul 14, 2025 at 11:22:08PM +0200, Marcel Telka wrote:
> Hi,
>
> I'm trying to explore bhyve for the first time and so I tried some basic
> example from the bhyve(8) man page. Surprisingly, it does not work.
>
# reboot
...
# ps -elf | grep bhyve
# mdb -ke ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 530085 2070 13%
Boot pages 169 0 0%
ZFS File Data 878380 3431 21%
Anon 1339939 5234 32%
Exec and libs 4831 18 0%
Page cache 74943 292 2%
Free (cachelist) 72767 284 2%
Free (freelist) 1261250 4926 30%
Total 4162364 16259
Physical 4162363 16259
> # mkfile -n 1g image
> # bhyve -c 2 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,image \
> -l com1,stdio -H -P -m 1G vm
> virtual machine cannot be booted
# ps -elf | grep bhyve
0 S root 2491 2401 0 73 20 ? 455 ? 08:01:52 pts/1
0:00 grep bhyve
# mdb -ke ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 546514 2134 13%
Boot pages 169 0 0%
ZFS File Data 899780 3514 22%
Anon 1329041 5191 32%
Exec and libs 4802 18 0%
Page cache 78496 306 2%
Free (cachelist) 73948 288 2%
Free (freelist) 1229614 4803 30%
Total 4162364 16259
Physical 4162363 16259
> # bhyve -c 2 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,image \
> -l com1,stdio -H -P -m 1G vm
> vm exit[0]
> reason VMX
> rip 0x0000000000000000
> inst_length 0
> status 0
> exit_reason 33 (VM-entry failure due to invalid guest state)
> qualification 0x0000000000000000
> inst_type 0
> inst_error 0
> Abort (core dumped)
# ps -elf | grep bhyve
0 R root 2517 2401 0 63 20 ? 455 08:02:14 pts/1
0:00 grep bhyve
# mdb -ke ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 547803 2139 13%
Boot pages 169 0 0%
ZFS File Data 903686 3530 22%
VMM Memory 266240 1040 6%
Anon 1329034 5191 32%
Exec and libs 4802 18 0%
Page cache 78751 307 2%
Free (cachelist) 74013 289 2%
Free (freelist) 957866 3741 23%
Total 4162364 16259
Physical 4162363 16259
> # bhyve -c 2 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,image \
> -l com1,stdio -H -P -m 1G vm
> bhyve: could not activate CPU 0: Device busy
# ps -elf | grep bhyve
0 S root 2706 2401 0 65 20 ? 455 ? 08:09:56 pts/1
0:00 grep bhyve
# mdb -ke ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 569160 2223 14%
Boot pages 169 0 0%
ZFS File Data 966845 3776 23%
VMM Memory 266240 1040 6%
Anon 1348429 5267 32%
Exec and libs 5712 22 0%
Page cache 79339 309 2%
Free (cachelist) 82921 323 2%
Free (freelist) 843549 3295 20%
Total 4162364 16259
Physical 4162363 16259
# modinfo | grep vmm
189 fffffffff40c0000 35388 307 1 vmm (bhyve vmm)
# modunload -i 189
modunload: can't unload the module: Device busy
--
+-------------------------------------------+
| Marcel Telka e-mail: [email protected] |
| homepage: http://telka.sk/ |
+-------------------------------------------+
------------------------------------------
illumos: illumos-discuss
Permalink:
https://illumos.topicbox.com/groups/discuss/T0a2ef97770149180-M7728dc06a6aeeb429df74017
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription
