I have begun work on porting Qubes to work within a KVM host. I need a
development environment that can utilize the cuda cores on a secondary
Nvidia RTX GPU and also prefer to be able to utilize the graphics card.
For several weeks I attempted to get the GPU to successfully pass-though to
a Qubes virtual machine using the Nvidia drivers without success. I can
get it to work in dom0, but how could one think to do any work there.
I looked into KVM and discovered I can pass-though the GPU with no issues,
it just works.
So now I am in a dilemma. I love Qubes, I have been using Qubes since 2014
and was having a hard time coming to terms with having to move to a plain
KVM environment. I have come to the realization that moving to KVM is the
only option now. It was time to say goodbye to Qubes and was having a
difficult time dealing with that, so I again spent way too much time trying
to get the Nvidia GPU working in Qubes.
I then looked into alternatives to prevent my complete departure from
Qubes. Marek told me about DomB, which is now in its design stages. It
would allow me to statically partition my machine (like having 2 dom0 VMs -
remember the Nvidia GPU with nvidia drivers works in dom0), but there is no
experimental code ready yet. So, then I attempted to get Qubes to run
nested within KVM. I was having issues with the display and then decided
that instead of running Qubes nested, I should just get Qubes to run within
KVM. That's where we are today, no need to leave Qubes, invite Qubes over
to KVM!
*GOALS*
The final goals would be to support all Qubes features and apps.
*STAGE 1*
The initial goal is to get Qubes to be able to manage the virtual machines
(start, stop, etc) using 'qvm-*' tools and *Qubes Manager*. Seamless VM
video or audio will not be implemented in stage 1 so either a GPU will need
to be passed through to the VM (which will also be able to provide HDMI
audio), or access using spice or vnc. Stage 1 goals include the following:
- Use same template system Qubes currently uses including settings like
*qvm-prefs*, *features*, *tags*, etc.
- Obviously support PCI pass-through using Nvidia drivers for RTX GPU
- Support qrexec communication from host <-> vm
- Locking down KVM host
- Securing the network - look into ability to enabling *sys-net* and
*sys-firewall*
*FUTURE*
- Seamless windows
- Audio
- Encrypted memory within each VM (AMD processors)
*BUILD STATUS*
I have modified where necessary all Qubes source repos to allow building
for KVM within a Fedora-32 host and guest. All build modifications used
conditional tests based on the 'BACKEND_VMM' build configuration option
which is set to 'kvm'. When 'BACKEND_VMM' is set to 'xen', everything
builds as normal.
- *vmm-xen*: I still include this package to allow booting into KVM or
XEN. There is also one dependency on it I need to remove.
- *core-libvirt*: Configured to also compile the KVM modules and any
other modules provided within the Fedora 32 distribution packages.
- *core-vchan-xen*: Not required. Components that require it use the
'BACKEND_VMM' build variable. Nice forward thinking from the Qubes
developers!
- *core-vchan-libkvmchan*: Packaged *libkvmchan
<https://github.com/shawnanastasio/libkvmchan>* code based on the work
completed by @shawnanastasio <https://github.com/shawnanastasio>.
- *qubes-core-vchan-kvm*: Packaged *qubes-core-vchan-kvm
<https://github.com/shawnanastasio/qubes-core-vchan-kvm>* code based on
the work completed by @shawnanastasio <https://github.com/shawnanastasio>
.
- *linux-utils*: Removed *qmemman* for KVM build. Not sure is they can
be adapted for KVM. Will revisit KVM alternative later.
- *core-admin*:
- Added KVM *libvirt* template
- Added additional conditional 'BACKEND_VMM' for Xen specific build
and depends
- Still installs *qubes-qmemman.service* until files. Not sure if
they can be adapted to KVM.
- *qubes* python
- *other*: Minor changes here and there.
*INSTALL STATUS*
- *dom0*:
- All dom0 packages install without error (minus vmm-stubdom and iso
related packages)
- All Qubes services start successfully on boot
- *template*:
- qubes-template-fedora-32 installs within kvm host. A few manual
modification were made to qubes.xml to facilitate this.
*WIP*
- *core-admin*
- *qubes python package*
- Added a 'hypervisor' module to detect hypervisor type (xen, kvm,
etc) for cases like the following where it is expected that the
hypervisor
is Xen if 'xen.lowlevel' is able to be imported. In my case the Xen
module
is installed since I also have a Xen boot option:
- *qubes.app.VMMConnection.init_vmm_connection* change.
- old: if 'xen.lowlevel.{xs,xc}' in sys.modules:
- new: if hypervisor_type('xen'):
- There are a few dependencies on Xen such as:
-
*qubes.app.xs (xenstore) *
I was hoping that '*xenstore*' could be used as a standalone
application (without Xen being activated). I have not yet looked
at the
source code but tried starting the *xenstore* service which
failed since the '/proc/xen' directory does not exist. Wondering a
I
created an *procfs* entry for '/proc/xen' if the store would
run without Xen.
If *xenstore* won't work without X*en* then need to determine
the best alternative; convert *xenstore* to work without Xen or
some other solution?
- *qubes.ext.pci.attached_devices*:
- ls('', 'backend/pci'), ls('', 'backend/pci' + domid)
- read('', devpath + '/domain'), read('', devpath +
'/num_devs'), read('', devpath + '/dev-' + str(dev))
- *qubes.vm.qubesvm.stubdom_xid*: if xs is None: return -1
# No issue
- *qubes.vm.qubesvm.start_time*: read('',
'/vm/{}/start_time').format(self.uuid)
- *qubes.vm.qubesvm.create_qdb_entries*: set_permissions('',
'/local/domain/{}/memory'.format(self.xid, [{'dom': self.xid}])
- *qubes.vm.qubesvm.get_prefmem*: read('',
'/local/domain/{}/memory/meminfo').format(self.xid))
-
- *qubes.app.xc (xen connection)*
- *qubes.app.QubesHost.get_free_xen_memory*:
physinfo()['free_memory']
- *qubes.app.QubesHost.is_iommu_supported*:
physinfo()['virt_caps']
- *qubes.app.QubesHost.get_vm_stats*: domain_getinfo(int,
int)['domid', 'mem_kb', 'cpu_time', 'online_vpus']
- Added a 'hypervisor' script to '/usr/lib/qubes' for other
scripts like 'startup-misc.sh'
- if /usr/lib/qubes/hypervisor_type xen; then ...
*CURRENT ISSUES TO RESOLVE*
- *xenstore*: Will it work without Xen? If not convert it so it will
or provide another alternative?
- *qmemman*:
- Provide KVM alternative
- Should components like linux-utils that provide xen only utilities
have the xen utilities split into another repo like 'linux-utils-xen'?
Then when a KVM alternative can be provided it could be placed in
'linux-utils-kvm'?
- *Qubes python packages*:
- Not yet sure how much of it relies on any xen packages. Currently I
will continue using the hypervisor check and once all python packages are
functioning correctly with KVM we can look into better ways to handle xen
vs kvm or other hypervisors.
- *qubes-builder*:
- For some reason I can not build with 'DIST_BUILD_TOOLS=1" (standard
qubes xen components). I always get an error when building dom-fc32
of
"sudo: unrecognized option
'--resultdir=/home/user/qubes/chroot-dom0-fc32/home/user/qubes-src/vmm-xen/pkgs/dom0-fc32'.
Am I missing another config option?
- Libvirt often fails to compile using 32 cores giving some error
about some file that does not exist (when it does fail, it always
fails at
the same spot with same error message). It seems to be compiling too
fast
or maybe has something to do with using BTRFS filesystem. The rpm
spec for
libvirt uses the number of processors available (make -j32 V=1). It
will
build without errors if I generate a '.rpmmacros' file containing
'%_smp_mflags' -j10' to the 'chroot-dom0-fc32/home/user' directory.
Just
wondering is there is a way to set number of jobs per component, or
maybe
switching to using 'DIST_BUILD_TOOLS' will help.
Comments welcome,
Jason
--
You received this message because you are subscribed to the Google Groups
"qubes-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to qubes-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/qubes-devel/7850e47a-64df-4b2b-81b1-c29e0e22ccd4o%40googlegroups.com.
