This patch series adds the live migration cpr-exec mode. In this mode, QEMU stops the VM, writes VM state to the migration URI, and directly exec's a new version of QEMU on the same host, replacing the original process while retaining its PID. Guest RAM is preserved in place, albeit with new virtual addresses. The user completes the migration by specifying the -incoming option, and by issuing the migrate-incoming command if necessary. This saves and restores VM state, with minimal guest pause time, so that QEMU may be updated to a new version in between.
The new interfaces are: * cpr-exec (MigMode migration parameter) * cpr-exec-args (migration parameter) * memfd-alloc=on (command-line option for -machine) * only-migratable-modes (command-line argument) The caller sets the mode parameter before invoking the migrate command. Arguments for the new QEMU process are taken from the cpr-exec-args parameter. The first argument should be the path of a new QEMU binary, or a prefix command that exec's the new QEMU binary, and the arguments should include the -incoming option. Memory backend objects must have the share=on attribute, and must be mmap'able in the new QEMU process. For example, memory-backend-file is acceptable, but memory-backend-ram is not. QEMU must be started with the '-machine memfd-alloc=on' option. This causes implicit RAM blocks (those not explicitly described by a memory-backend object) to be allocated by mmap'ing a memfd. Examples include VGA, ROM, and even guest RAM when it is specified without without reference to a memory-backend object. The memfds are kept open across exec, their values are saved in vmstate which is retrieved after exec, and they are re-mmap'd. The '-only-migratable-modes cpr-exec' option guarantees that the configuration supports cpr-exec. QEMU will exit at start time if not. Example: In this example, we simply restart the same version of QEMU, but in a real scenario one would set a new QEMU binary path in cpr-exec-args. # qemu-kvm -monitor stdio -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m 4G -machine memfd-alloc=on ... QEMU 9.1.50 monitor - type 'help' for more information (qemu) info status VM status: running (qemu) migrate_set_parameter mode cpr-exec (qemu) migrate_set_parameter cpr-exec-args qemu-kvm ... -incoming file:vm.state (qemu) migrate -d file:vm.state (qemu) QEMU 9.1.50 monitor - type 'help' for more information (qemu) info status VM status: running cpr-exec mode preserves attributes of outgoing devices that must be known before the device is created on the incoming side, such as the memfd descriptor number, but currently the migration stream is read after all devices are created. To solve this problem, I add two VMStateDescription options: precreate and factory. precreate objects are saved to their own migration stream, distinct from the main stream, and are read early by incoming QEMU, before devices are created. Factory objects are allocated on demand, without relying on a pre-registered object's opaque address, which is necessary because the devices to which the state will apply have not been created yet and hence have not registered an opaque address to receive the state. This patch series implements a minimal version of cpr-exec. Future series will add support for: * vfio * chardev's without loss of connectivity * vhost * fine-grained seccomp controls * hostmem-memfd * cpr-exec migration test Steve Sistare (26): oslib: qemu_clear_cloexec vl: helper to request re-exec migration: SAVEVM_FOREACH migration: delete unused parameter mis migration: precreate vmstate migration: precreate vmstate for exec migration: VMStateId migration: vmstate_info_void_ptr migration: vmstate_register_named migration: vmstate_unregister_named migration: vmstate_register at init time migration: vmstate factory object physmem: ram_block_create physmem: hoist guest_memfd creation physmem: hoist host memory allocation physmem: set ram block idstr earlier machine: memfd-alloc option migration: cpr-exec-args parameter physmem: preserve ram blocks for cpr migration: cpr-exec mode migration: migrate_add_blocker_mode migration: ram block cpr-exec blockers migration: misc cpr-exec blockers seccomp: cpr-exec blocker migration: fix mismatched GPAs during cpr-exec migration: only-migratable-modes accel/xen/xen-all.c | 5 + backends/hostmem-epc.c | 12 +- hmp-commands.hx | 2 +- hw/core/machine.c | 22 +++ hw/core/qdev.c | 1 + hw/intc/apic_common.c | 2 +- hw/vfio/migration.c | 3 +- include/exec/cpu-common.h | 3 +- include/exec/memory.h | 15 ++ include/exec/ramblock.h | 10 +- include/hw/boards.h | 1 + include/migration/blocker.h | 7 + include/migration/cpr.h | 14 ++ include/migration/misc.h | 11 ++ include/migration/vmstate.h | 133 +++++++++++++++- include/qemu/osdep.h | 9 ++ include/sysemu/runstate.h | 3 + include/sysemu/seccomp.h | 1 + include/sysemu/sysemu.h | 1 - migration/cpr.c | 131 ++++++++++++++++ migration/meson.build | 3 + migration/migration-hmp-cmds.c | 50 +++++- migration/migration.c | 48 +++++- migration/migration.h | 5 +- migration/options.c | 13 ++ migration/precreate.c | 139 +++++++++++++++++ migration/ram.c | 16 +- migration/savevm.c | 306 +++++++++++++++++++++++++++++------- migration/savevm.h | 3 + migration/trace-events | 7 + migration/vmstate-factory.c | 78 ++++++++++ migration/vmstate-types.c | 24 +++ migration/vmstate.c | 3 +- qapi/migration.json | 48 +++++- qemu-options.hx | 22 ++- replay/replay.c | 6 + stubs/migr-blocker.c | 5 + stubs/vmstate.c | 13 ++ system/globals.c | 1 - system/memory.c | 19 ++- system/physmem.c | 346 +++++++++++++++++++++++++++-------------- system/qemu-seccomp.c | 10 +- system/runstate.c | 29 ++++ system/trace-events | 4 + system/vl.c | 26 +++- target/s390x/cpu_models.c | 4 +- util/oslib-posix.c | 9 ++ util/oslib-win32.c | 4 + 48 files changed, 1417 insertions(+), 210 deletions(-) create mode 100644 include/migration/cpr.h create mode 100644 migration/cpr.c create mode 100644 migration/precreate.c create mode 100644 migration/vmstate-factory.c -- 1.8.3.1
