Hey all,
I was able to get another 21+ hours of uptime on idle with no issues, and I did
some more gaming last night and it's stable. I haven't experienced any of the
weird vm exits recently so that's good. There are probably cases where the vm
may exit but I'm not sure what those situations are given the recent stability.
I'll keep monitoring but I'm pretty happy with how everything turned out so far.
I've also debug the script a bit more to see why it wasn't shutting down. The
docs aren't explicit about it (or at least I haven't found a place yet for it),
but I think it may be related to the pid stuff. It doesn't seem that
rc.shutdown calls the stop command if it's simply defined given there are
performance optimizations to skip certain things even if KEYWORD: shutdown is
defined. So after doing a "ps -aux | grep gaming" and seeing what showed up, I
saw that there are 3-4 pids that exist with my current set up. daemon pid, main
bhyve pid, start.sh pid, etc. The pid that was getting written by daemon
previously was the pid of my start.sh script, but the pid of the bhyve process
that gets started by start.sh, so that would be a "grandchild of daemon", so I
think that's why originally it wasn't working for me. I now merged my
start/stop script logic into the main rc.d script, so daemon is directly
executing bhyve, and thus the child process recorded in the pidfile is correct.
The stop command can now load up the child pidfile, and properly call kill on
that pid directly. My script now works on shutdown properly. The only thing I
noticed was that even though I do now see Windows displaying the "shutting
down" message, it kinda seems like the system isn't necessarily obeying my
"sleep 10" call afterwards, which means it probably still isn't 100% a graceful
shutdown, but much better than before. There is also the bhyvectl --destroy ...
command after and I'm not sure if that's running. Below is the improved and
centralized rc.d script:
#!/bin/sh
# PROVIDE: vm_gaming
# REQUIRE: LOGIN
# KEYWORD: nojail shutdown
. /etc/rc.subr
name=vm_gaming
rcvar=vm_gaming_enable
vm_path="/atlantis/vms/gaming"
vm_name="gaming"
start_cmd="${name}_start"
stop_cmd="${name}_stop"
restart_cmd="${name}_restart"
pidfile="/var/run/${name}.child.pid"
vm_gaming_start()
{
daemon \
-o /var/log/${name}.log \
-p "${pidfile}" \
bhyve -AHPSw -c sockets=1,cores=16,threads=1 -m 32G \
-s 0,hostbridge \
-s 1,nvme,${vm_path}/disk0.img \
-s 3:0,passthru,3/0/0 \
-s 3:1,passthru,3/0/1 \
-s 13:0,passthru,13/0/0 \
-s 30,xhci,tablet \
-s 31,lpc \
-l
bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \
-o console=stdio \
${vm_name}
}
vm_gaming_stop()
{
# Send SIGTERM twice to make sure Windows listens to
# the ACPI shutdown signal.
pid="$(cat ${pidfile})"
kill ${pid}
kill ${pid}
# Wait a bit for the guest to shutdown properly before
# we continue shutting down the host.
sleep 10
bhyvectl --vm=${vm_name} --destroy
}
vm_gaming_restart()
{
# NOTE: AMD users will most likely experience the famous
# AMD Hardware Reset Bug. This means that after you reboot
# the guest, you most likely won't have video out. If this
# happens, you'll need to restart the host. Sometimes this
# command will work for AMD users though. So you can try a
# restart and see if it works.
vm_gaming_stop
vm_gaming_start
}
load_rc_config $name
: ${vm_gaming_enable:="NO"}
run_rc_command "$1"
Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email
On Wednesday, September 24th, 2025 at 09:05, Jonathan Vasquez <[email protected]>
wrote:
> Hello all,
>
> I was able to get another stable vm on idle (this time without leaving Task
> Manager
> opened) for another 22 hours. I also spent some time today improving my
> start/stop
> scripts. Using Corvin's suggestion about the "daemon" process, I was able to
> now successfully start the VM at boot time without blocking. I had to make
> some
> adjustments to my bhyve script so it doesn't use "-l com1,stdio", because
> during
> boot time, even with me making the script require "LOGIN syscons", it still
> would throw the following error:
>
> Unable to initialize backend 'stdio' for LPC device com1
> 100 Device emulation initialization error: Inappropriate ioctl for device
>
> Which prevented the VM from starting. I've switched to using "-o
> console=stdio"
> and that worked. I'm able to get regular console output, which allows the VM
> to always start up successfully, and also allows us to log to a file when we
> run
> our vm start script through daemon.
>
> I've also observed that there were a few times when I restarted the VM without
> restarting the host, 2-3 times, and each time the video output worked (meaning
> I wasn't affected by the AMD Hardware Reset Bug). However, after that I wasn't
> able to get that working anymore. So I would still recommend restarting the
> host.
>
> I added the following to the script:
>
> - Daemonized support so the VM starts up non-blocking. This also includes
> logging to a file at /var/log/vm_gaming.log. I didn't add -P and -p since the
> pid addresses looked off. We also don't seem to need it at the moment.
> - Added the restart command (with a note about the AMD issue).
> - Added some quality of life stuff like having some variables at the top
> to easily set your start/stop paths.
>
> Some other things are that if I do "service vm_gaming <some command>", the
>
> current console in tmux seems to get really laggy and the input gets weird. If
> I open another tmux window, then input is fine again. There are no other
> consequences other than laggy/wrong input in the current console. This doesn't
> seem to negatively affect anything when starting it automatically at boot time
> through rc.d.
>
> One last thing is that I noticed that when I do a "shutdown -r now", I don't
> see windows getting the ACPI shutdown signal, so I don't see Windows
> gracefully
> shutting down. I have read if you do "reboot" it will just send the SIGTERM
> signal,
> and kill everything afterwards, so the "shutdown" command should be used to
> give advance notice to processes. Given that my rc.d script has "shutdown" in
> the keywords list, I was expecting the "vm_gaming_stop" command to be ran
> automatically in order to gracefully shutdown the VM, and clean itself up. Is
> this
> not working because I'm not using -P / -p?
>
> I've updated my blog post to include all of these new scripts and
> modifications.
>
> I'll post the current scripts below. Thanks for reading.
>
> Jonathan
>
>
> ## Bhyve Scripts
>
> ### start script
>
> This is the main script that will start your vm. Save this as `start.sh`
> in a folder called `gaming` somewhere on your system:
>
> `#!/bin/sh VM_DIR="$(dirname $(realpath $0))" VM_NAME="$(basename $VM_DIR)"
> cd "$VM_DIR" bhyve -AHPSw -c sockets=1,cores=16,threads=1 -m 32G \\ -s
> 0,hostbridge \\ -s 1,nvme,disk0.img \\ -s 3:0,passthru,3/0/0 \\ -s
> 3:1,passthru,3/0/1 \\ -s 13:0,passthru,13/0/0 \\ -s 30,xhci,tablet \\ -s
> 31,lpc \\ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu
> \\ -o console=stdio \\ $VM_NAME # Exit the script here and leave some options
> for debugging more # ergonomically after the exit line. exit # Only use this
> when installing the VM for the first time: # VNC. Once you install Windows
> and enable RDP, you can turn this off. -s
> 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait \\ # The Windows ISO and the
> VirtIO Drivers ISO. # You can download the latest stable virtio drivers at: #
> https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso
> -s 4,ahci-cd,../files/Win10_22H2_English_x64v1_2023.iso \\ -s
> 5,ahci-cd,../files/virtio-win-0.1.271.iso \\ # If you want a network device
> without using virtio, you can use e1000. # You should install the virtio
> drivers and use virtio-net instead for # better performance. -s 2,e1000,tap0
> \\ -s 2,virtio-net,tap0 \\`
>
> ### stop script
>
> This is your stop script. Save this as `stop.sh` somewhere.
>
> `NOTE:` This is currently set up to kill any process with `bhyve` in
> the name. On this system I'm only running one bhyve VM, which is the gaming
> one.
> Adjust accordingly to restrict its scope.
>
> `#!/bin/sh # Send SIGTERM twice to make sure Windows listens to # the ACPI
> shutdown signal. pkill bhyve pkill bhyve # Wait a bit for the guest to
> shutdown properly before # we continue shutting down the host. sleep 10
> bhyvectl --vm=gaming --destroy`
>
> ### rc.d script
>
> This script will allow you to start the VM automatically at boot time. Save
> this
> file in your `/usr/local/etc/rc.d/` directory with the name `vm_gaming`.
> Make sure to adjust any paths to where you saved your start/stop scripts:
>
> `NOTE:` At the moment I noticed that if I do a `shutdown -r now`, I don't
> actually see Windows gracefully shutting down (like when I send the ACPI
> shutdown
> signal). Currently looking into this.
>
> `#!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdown .
> /etc/rc.subr name=vm_gaming rcvar=vm_gaming_enable start_cmd="${name}_start"
> stop_cmd="${name}_stop" restart_cmd="${name}_restart"
> path_to_start_script="/atlantis/vms/gaming/start.sh"
> path_to_stop_script="/atlantis/vms/gaming/stop.sh" vm_gaming_start() { daemon
> \\ -o /var/log/${name}.log \\ "${path_to_start_script}" } vm_gaming_stop() {
> "${path_to_stop_script}" } vm_gaming_restart() { # NOTE: AMD users will most
> likely experience the famous # AMD Hardware Reset Bug. This means that after
> you reboot # the guest, you most likely won't have video out. If this #
> happens, you'll need to restart the host. Sometimes this # command will work
> for AMD users though. So you can try a # restart and see if it works.
> vm_gaming_stop vm_gaming_start } load_rc_config $name :
> ${vm_gaming_enable:="NO"} run_rc_command "$1"`
