This is a hefty help email for Shamim Shahriar in regards to passing through Nvidia GPU to a bhyve VM.
Hi. I have recently gotten this mailing list message forwarded to me by Michael Dexter https://lists.freebsd.org/archives/freebsd-virtualization/2025-March/002818.html and it alone made me sign up here. So, here's a big mail containing a bunch of useful GPU passthrough info! To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config, let me help you out make it work correctly! Parts and short description: 1. Pass through the GPU audio and other devices (if exist) Modern GPUs usually create two PCI devices that you can see in pciconf - VGA and audio. They come on the same PCI function (i think it is called that), for example GPU 19/0/0 will also have 19/0/1 for audio and may have 19/0/2 for usb. 2. Give the GPU a VGA BIOS as PCI Option ROM GPUs need a VGA BIOS to work. In addition, Nvidia GPUs need a trimmed VGA BIOS since the PCI option rom that is used by the VM is nested inside the ROM file. (from now on i will refer to the vga bios as vbios) 3. Possibly, re-BAR in BIOS? Some UEFI firmware have a feature called re-BAR and "above 4G decoding / mining", it makes some GPUs in some configurations not work in passthru. 5. Possibly, Try newer EDK2 The edk2-bhyve package (provides UEFI firmware for bhyve) on the repos is still using the 2023 release. You can compile a 2025 version with my patch: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554 Full parts and HOWTOs: !!!! MISC GPU DEVICES As decribed before, pass through all the devices that your GPU creates in pciconf right after the GPU device on the same PCI function. All audio, USB, etc. devices belonging to the GPU should be passed through with it since they are a part of it, else the passthru will NOT work correctly. If that is not the case and there is only a single GPU VGA device with nothing else Nvidia to be found, skip this step. !!!! VBIOS You first need to get the pure vBIOS file. To do that you can either: 1. Use a Linux system and dump it with https://github.com/ilya-zlobintsev/LACT 2. Use a Windows system and dump it with https://www.techpowerup.com/gpuz 3. Download one that someone has dumped from their GPU using GPU-z from https://www.techpowerup.com/vgabios If you don't want to install a different OS on the PC you're currently running, we can just find and download a correct vBIOS from Techpowerup. But we stumble across an issue! You specified "NVidia Quadro RTX 6000/8000", so which GPU is it really? Those are two different GPUs with different specifications! Thankfully, we have your pciconf output which says "subdevice=0x13d8" and if we refer to Nvidias documentation, on page 7 over here: https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-1219.pdf we can see that it is a RTX 8000! Here is an unverified but most likely working vBIOS: https://www.techpowerup.com/vgabios/256271/256271 Now after we've gotten the vBIOS, we need to trim the start of the file to get to the embedded PCI Option ROM. For that we will use my tool "trim_nvidia_vbios" from https://github.com/9vlc/ptnotes located in progs/trim_nvidia_vbios Compile and run the resulting executable with the first argument being the downloaded vBIOS and the second argument being an output PCI OpROM. Example: $ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c $ ./trim.elf techpowerup.rom trimmed-vbios.rom And we come upon a road block: vm-bhyve does not support passing a PCI Option ROMs in "passthru=", so we're going to use "bhyve_options=" instead to define the GPU VGA device: bhyve_options="-s 6:0,passthru,19/0/0,rom=</path/to/trimmed-vbios.rom>" Other devices that do not require a PCI Option ROM can be added with "passthru=". These devices include the rest of the GPU pci functions like audio and usb. !!!! RE-BAR If after everything you do get the GPU to work in the VM at least in the edk2 firmware, Go to your regular UEFI BIOS setup utility and search for options titled "BAR", "PCI Re-size BAR", "re-BAR", "Above 4G Encoding" or "Above 4G for Mining". If you do see re-BAR but cannot disable it, you could try using https://github.com/DavidS95/Smokeless_UMAF to modify locked UEFI settings. Be warned though, smokeless is a heck of an unstable MESS. !!!! UPDATED EDK2 As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some rare cases, that might cause issues. I have submitted a package update bug report for sysutils/edk2 to FreeBSD bugzilla over here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554 You can download the patch, apply it to your ports tree, then compile and install the edk2 package with "FLAVOR=bhyve" to get the fresh UEFI firmware. ~~~~ Hope I didn't forget anything! I wish you success in the GPU passthrough journey!! - Alexey
