-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi Jason, welcome back :)
On Wed, Jul 08, 2020 at 06:34:51PM -0700, Jason M wrote: > I am currently researching and working on getting the Nvidia RTX20x0 series > GPU working within a VM using the Nvidia drivers on Qubes 4.1 (master > branches). > > I personally require access to the GPU Cuda cores for development > purposes. I have been able to successfully install and run the Nvidia > drivers within dom0 and the Nouveau drivers within the VM using either > Seabios or OVMF UEFI bios. > > Issues solved: > > - Running Nvidia GPU in dom0 with either Nouveau or Nvidia and Cuda > drivers > > - Running Nvidia GPU in HVM with Nouveau driver > > - Setting OVMF UEFI to boot HVM without persistent bios data > > - Using a Qubes template to pass options to libvirt > > Some of the issues I am currently working on are: > > - The big one; working on preventing Nvidia driver from failing due to > being run within a virtual environment. I have completed research on this > and start testing some theories this weekend. Most of the research tends > to support that the Nvidia drivers purposely fail to load and run within a > virtual environment. Yes, I think this is exactly the case. I haven't checked, but I wouldn't be surprised there is some fine print about which cards are supported in virtualized environment... > I don't know if that still holds true for RTX20X0 > cards, or even if the drivers fail to purposely load as that would be > illegal to do in many countries. In any event, some on the experiments > will be > attempting to hide the hypervisor, and kernel options related to PCI, > IOMMU and memory options like disabling encrypted memory on AMD processors. > > - Passing GPU as multi function device device. Not too sure if > multi-function is required. > > - Passing GPU ROM to HVM which may be required in some circumstances. I > do not have this issue as I am passing though the Nvidia GPU as a > secondary > card with all of its functions attached to the Xen pciback driver. The > GPU > ROM loaded within the VM does match the one that was loaded within dom0. > > Here are excerpts of the config file containing additional > configuration for GPU passthrough > (/etc/qubes/templates/libvirt/xen/by-name/fedora32-hvm.xml): > > ... <cpu mode='host-passthrough'> > <feature name='topoext' policy='require'/> > ... > </cpu> > > ... <features> > <xen><passthrough state='on'/> This one should be enabled automatically if there is PCI device included. > ... > </features> > > ... <hostdev mode='subsystem' type='pci' managed='yes' > permissive='yes'> > <source><address domain='0x0000' bus='0x2e' slot='0x00' > function='0x0 /></source> > *<!-- ENABLE MULTI-FUNCTION? -->* > <address type='pci' domain='0x0000' bus='0x00' slot='0x08' > function='0x0' multifunction='on' /> > *<!-- PASS GPU ROM VIA UEFI BIOS -->* > <rom bar='on' file='/etc/qubes/templates/nvidia-rom.bin' /> > </hostdev> Does rom option has any effect? I wouldn't expect this part of libvirt config to be supported with Xen. BTW there is a similar change for Intel graphics: https://github.com/QubesOS/qubes-vmm-xen/pull/79 Perhaps something similar would be needed here too? (...) > I am considering some of the following alternatives if I can not get the > Nvidia driver to work in VM: > > 1. Dom0less (static partitioning) to allow Xen to boot Qubes and a > static VM without the Dom0 control domain via DeviceTree which will > contain > dedicated hardware resources (CPU, RAM, etc). This is currently being > done > for the ARM architecture but there is not too much information out there > on > how to accomplish this for X86 processors. The current implementation > uses > U-boot [1] to load Xen, the device tree binary, dom0 kernel and ramdisk > [2-3]. U-boot may have support for X86 but it looks like it has not been > updated since 2015 [4]. The x86 equivalent of this feature is called "domB" and is at early design phase. > 2. Implementing some type of vchan/qrexec service within VM to access > Nvidia GPU cuda cores within dom0 (I do not require access to display). > Not sure what security implications this would have though. Depends on what you need cuda for exactly, but this is probably the hardest option from those listed here. You will need to both provide shared memory (or equivalent for data transfer) and properly maintain security boundaries. Remember that GPU can do DMA, and IOMMU (in Xen) filters it only at domain level - when GPU is attached to dom0, it can DMA to dom0. > 3. Nested virtualisation (Qubes in KVM). Yuk. Quickly tried and can't > get pass initial boot screen to login screen. Maybe there are drivers > missing in initramfs disk as I can get nested Fedora Xen to boot. May be > a > suitable stop-gap measure if proceeding with alternative #4 below. Qubes nested in KVM works, but depending on your hardware (Intel vs AMD) is tricky to setup - virtualized emulated IOMMU. We run automated tests (openQA) in such setup, but without passthrough of a real device. > 4. KVM using Qubes frameworks. @shawnanastasio > <https://github.com/shawnanastasio> has been working on > qubes-core-vchan-kvm[5-6] > and libkvmchan[7], which is a good starting point for adapting Qubes > packages to KVM. This would be a lot of work. I am familiar enough with > Qubes to take on some of the tasks if this ends up being the only viable > option. Yes, this is also an option. Some more context is here: https://github.com/QubesOS/qubes-issues/issues/4318 > Regards, > > Jason > > > [1] https://en.wikipedia.org/wiki/Das_U-Boot > [2] > http://xenbits.xenproject.org/docs/unstable-staging/features/dom0less.html > <http://xenbits.xenproject.org/docs/unstable-staging/features/dom0less.html> > [3] https://elinux.org/images/f/f7/Dom0less_-_ELC_2019.pdf > [4] https://www.denx.de/wiki/U-Boot/X86 > <https://www.denx.de/wiki/U-Boot/X86> > [5] > https://github.com/QubesOS/qubes-issues/issues/4318#issuecomment-630636383 > [6] https://github.com/shawnanastasio/qubes-core-vchan-kvm > [7] https://github.com/shawnanastasio/libkvmchan > - -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl8GgK8ACgkQ24/THMrX 1yzx0wf+PfOPm0orgXY9rJAn/1TUYwUb2/Eh4HFRZOL1S6BEICSRVnNo7ncSZFgu rpHZ85+MJKrYYkaPu0JiFSzG+/SxKtv0ylJjJnCUbNhucSHfkQy5wEvLeGo4NPxk ytcmzWvw6dMQARpsB4huO3XU2zbuNwIKTh9hOuITvHHe72vqKMDb3NeAQoACiHPU 4rMlOZBRALp1k44tS2KJIn/hSACiBBXUvXoA40ogtulN1q5UxDZ/nDm92k7vhOSA qUlT/btqgSxw5D7sjFmHTkYhUefyxl1zsrw9AQ435vqFlF/Tw0i9ijNummor8zMR aKcVtYYJa+k3EEcR8Y+CYjRn8PI+ZQ== =ZpDG -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "qubes-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/20200709022800.GH139534%40mail-itl.
