Steve Sistare <steven.sist...@oracle.com> writes: > Signed-off-by: Steve Sistare <steven.sist...@oracle.com> > --- > docs/devel/migration/CPR.rst | 147 > ++++++++++++++++++++++++++++++++++++++ > docs/devel/migration/features.rst | 1 + > 2 files changed, 148 insertions(+) > create mode 100644 docs/devel/migration/CPR.rst > > diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst > new file mode 100644 > index 0000000..63c3647 > --- /dev/null > +++ b/docs/devel/migration/CPR.rst > @@ -0,0 +1,147 @@ > +CheckPoint and Restart (CPR) > +============================ > + > +CPR is the umbrella name for a set of migration modes in which the > +VM is migrated to a new QEMU instance on the same host. It is > +intended for use when the goal is to update host software components > +that run the VM, such as QEMU or even the host kernel. At this time, > +cpr-reboot is the only available mode. > + > +Because QEMU is restarted on the same host, with access to the same > +local devices, CPR is allowed in certain cases where normal migration > +would be blocked. However, the user must not modify the contents of > +guest block devices between quitting old QEMU and starting new QEMU. > + > +CPR unconditionally stops VM execution before memory is saved, and > +thus does not depend on any form of dirty page tracking. > + > +cpr-reboot mode > +--------------- > + > +In this mode, QEMU stops the VM, and writes VM state to the migration > +URI, which will typically be a file. After quitting QEMU, the user > +resumes by running QEMU with the ``-incoming`` option. Because the > +old and new QEMU instances are not active concurrently, the URI cannot > +be a type that streams data from one instance to the other. > + > +Guest RAM can be saved in place if backed by shared memory, or can be > +copied to a file. The former is more efficient and is therefore > +preferred. > + > +After state and memory are saved, the user may update userland host > +software before restarting QEMU and resuming the VM. Further, if > +the RAM is backed by persistent shared memory, such as a DAX device, > +then the user may reboot to a new host kernel before restarting QEMU. > + > +This mode supports VFIO devices provided the user first puts the > +guest in the suspended runstate, such as by issuing the > +``guest-suspend-ram`` command to the QEMU guest agent. The agent > +must be pre-installed in the guest, and the guest must support > +suspend to RAM. Beware that suspension can take a few seconds, so > +the user should poll to see the suspended state before proceeding > +with the CPR operation. > + > +Usage > +^^^^^ > + > +It is recommended that guest RAM be backed with some type of shared > +memory, such as ``memory-backend-file,share=on``, and that the > +``x-ignore-shared`` capability be set. This combination allows memory > +to be saved in place. Otherwise, after QEMU stops the VM, all guest > +RAM is copied to the migration URI. > + > +Outgoing: > + * Set the migration mode parameter to ``cpr-reboot``. > + * Set the ``x-ignore-shared`` capability if desired. > + * Issue the ``migrate`` command. It is recommended the the URI be a > + ``file`` type, but one can use other types such as ``exec``, > + provided the command captures all the data from the outgoing side, > + and provides all the data to the incoming side. > + * Quit when QEMU reaches the postmigrate state. > + > +Incoming: > + * Start QEMU with the ``-incoming defer`` option. > + * Set the migration mode parameter to ``cpr-reboot``. > + * Set the ``x-ignore-shared`` capability if desired. > + * Issue the ``migrate-incoming`` command. > + * If the VM was running when the outgoing ``migrate`` command was > + issued, then QEMU automatically resumes VM execution. > + > +Example 1 > +^^^^^^^^^ > +:: > + > + # qemu-kvm -monitor stdio > + -object > memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m > 4G > + ... > + > + (qemu) info status > + VM status: running > + (qemu) migrate_set_parameter mode cpr-reboot > + (qemu) migrate_set_capability x-ignore-shared on > + (qemu) migrate -d file:vm.state > + (qemu) info status > + VM status: paused (postmigrate) > + (qemu) quit > + > + ### optionally update kernel and reboot > + # systemctl kexec > + kexec_core: Starting new kernel > + ... > + > + # qemu-kvm ... -incoming defer > + (qemu) info status > + VM status: paused (inmigrate) > + (qemu) migrate_set_parameter mode cpr-reboot > + (qemu) migrate_set_capability x-ignore-shared on > + (qemu) migrate_incoming file:vm.state > + (qemu) info status > + VM status: running > + > +Example 2: VFIO > +^^^^^^^^^^^^^^^ > +:: > + > + # qemu-kvm -monitor stdio > + -object > memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m > 4G > + -device vfio-pci, ... > + -chardev socket,id=qga0,path=qga.sock,server=on,wait=off > + -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 > + ... > + > + (qemu) info status > + VM status: running > + > + # echo '{"execute":"guest-suspend-ram"}' | ncat --send-only -U qga.sock > + > + (qemu) info status > + VM status: paused (suspended) > + (qemu) migrate_set_parameter mode cpr-reboot > + (qemu) migrate_set_capability x-ignore-shared on > + (qemu) migrate -d file:vm.state > + (qemu) info status > + VM status: paused (postmigrate) > + (qemu) quit > + > + ### optionally update kernel and reboot > + # systemctl kexec > + kexec_core: Starting new kernel > + ... > + > + # qemu-kvm ... -incoming defer > + (qemu) info status > + VM status: paused (inmigrate) > + (qemu) migrate_set_parameter mode cpr-reboot > + (qemu) migrate_set_capability x-ignore-shared on > + (qemu) migrate_incoming file:vm.state > + (qemu) info status > + VM status: paused (suspended) > + (qemu) system_wakeup > + (qemu) info status > + VM status: running > + > +Caveats > +^^^^^^^ > + > +cpr-reboot mode may not be used with postcopy, background-snapshot, > +or COLO. > diff --git a/docs/devel/migration/features.rst > b/docs/devel/migration/features.rst > index 9d1abd2..d5ca7b8 100644 > --- a/docs/devel/migration/features.rst > +++ b/docs/devel/migration/features.rst > @@ -11,3 +11,4 @@ Migration has plenty of features to support different use > cases. > vfio > virtio > mapped-ram > + CPR
Reviewed-by: Fabiano Rosas <faro...@suse.de>