On 06.04.2020 17:20, Tamas K Lengyel wrote: > The following series implements VM forking for Intel HVM guests to allow for > the fast creation of identical VMs without the assosciated high startup costs > of booting or restoring the VM from a savefile. > > JIRA issue: https://xenproject.atlassian.net/browse/XEN-89 > > The fork operation is implemented as part of the "xl fork-vm" command: > xl fork-vm -C <config> -Q <qemu-save-file> -m <max-vcpus> <parent_domid> > > By default a fully functional fork is created. The user is in charge however > to > create the appropriate config file for the fork and to generate the QEMU save > file before the fork-vm call is made. The config file needs to give the > fork a new name at minimum but other settings may also require changes. > Certain > settings in the config file of both the parent and the fork have to be set to > default. Details are documented. > > The interface also allows to split the forking into two steps: > xl fork-vm --launch-dm no \ > -m <max-vcpus> \ > -p <parent_domid> > xl fork-vm --launch-dm late \ > -C <config_file_for_fork> \ > -Q <qemu_save_file> \ > <fork_domid> > > The split creation model is useful when the VM needs to be created as fast as > possible. The forked VM can be unpaused without the device model being > launched > to be monitored and accessed via VMI. Note however that without its device > model running (depending on what is executing in the VM) it is bound to > misbehave or even crash when its trying to access devices that would be > emulated by QEMU. We anticipate that for certain use-cases this would be an > acceptable situation, in case for example when fuzzing is performed of code > segments that don't access such devices. > > Launching the device model requires the QEMU Xen savefile to be generated > manually from the parent VM. This can be accomplished simply by connecting to > its QMP socket and issuing the "xen-save-devices-state" command. For example > using the standard tool socat these commands can be used to generate the file: > socat - UNIX-CONNECT:/var/run/xen/qmp-libxl-<parent_domid> > { "execute": "qmp_capabilities" } > { "execute": "xen-save-devices-state", \ > "arguments": { "filename": "/path/to/save/qemu_state", \ > "live": false} } > > At runtime the forked VM starts running with an empty p2m which gets lazily > populated when the VM generates EPT faults, similar to how altp2m views are > populated. If the memory access is a read-only access, the p2m entry is > populated with a memory shared entry with its parent. For write memory > accesses > or in case memory sharing wasn't possible (for example in case a reference is > held by a third party), a new page is allocated and the page contents are > copied over from the parent VM. Forks can be further forked if needed, thus > allowing for further memory savings. > > A VM fork reset hypercall is also added that allows the fork to be reset to > the > state it was just after a fork, also accessible via xl: > xl fork-vm --fork-reset -p <fork_domid> > > This is an optimization for cases where the forks are very short-lived and run > without a device model, so resetting saves some time compared to creating a > brand new fork provided the fork has not aquired a lot of memory. If the fork > has a lot of memory deduplicated it is likely going to be faster to create a > new fork from scratch and asynchronously destroying the old one. > > The series has been tested with Windows VMs and functions as expected. Linux > VMs when forked from a running VM will have a frozen VNC screen. Linux VMs at > this time can only be forked with a working device model when the parent VM > was > restored from a snapshot using "xl restore -p". This is a known limitation. > Also note that PVHVM/PVH Linux guests have not been tested. Forking most > likely > works but PV devices and drivers would require additional wiring to set things > up properly since the guests are unaware of the forking taking place, unlike > the save/restore routine where the guest is made aware of the procedure. > > Forking time has been measured to be 0.0007s, device model launch to be around > 1s depending largely on the number of devices being emulated. Fork resets have > been measured to be 0.0001s under the optimal circumstances. > > New in v14: > minor adjustments > > Patch 1 implements the VM fork > Patch 2 implements fork reset operation > Patch 3 adds the toolstack-side code implementing VM forking and reset > > Tamas K Lengyel (3): > xen/mem_sharing: VM forking > x86/mem_sharing: reset a fork
I've applied these two, but ... > xen/tools: VM forking toolstack side ... since this one doesn't have any ack or alike I'll defer to the tool stack maintainers here. Jan
