This series introduces cpr-luo, a CPR migration mode that uses the Linux Live Update Orchestrator (LUO) and Kexec Handover (KHO) to support KVM host kernel updates through kexec.
On the outgoing side, QEMU saves CPR state in a memfd and preserves that fd together with the RAMBlock backing fds in LUO. KHO carries the preserved LUO state into the new kernel. The incoming QEMU retrieves the CPR state and RAMBlock backing fds from LUO after kexec. The regular migration stream carries device vmstate. Although VFIO is currently blocked, I have an offline implementation that skips cpr-luo mode in cpr_is_incoming(), skips precopy, and carefully handles monitor_fd_param() although I never use it, so that VFIO is always reconstructed. It works with both iommufd and legacy for mlx5_vfio_pci by saving the VFIO and vendor specific state in vmstate as well. The result is that guest RAM remains available across a kexec-based host kernel update without copying its contents through the migration stream. Here is the main idea of the patchset: 1. Introduce the new cpr-luo mode. 2. Use a UUID to identify a LUO session. 3. Introduce APIs to create, preserve, and retrieve sessions and fds. 4. As LUO uses the name, id, and token instead of an fd, create an independent VMStateDescription rather than reuse the one for cpr-exec and cpr-transfer. 5. Compute the token when transforming entries from the hash table into the list. 6. Preserve fds on the source. Follow the approach used in cpr-exec and cpr-transfer. 7. Retrieve fds on the target. Follow the approach used in cpr-exec and cpr-transfer. The tricky part is where to set the cpr incoming mode. 8. There is no need to modify save_fd() or load_fd(), as we only need to migrate the corresponding name and token, not the fds. 9. Handle corner-case errors. 10. Block VFIO. I may need to rebase the current patchset on top of the patches below, from which I learned quite a lot in each iteration. While working on this patchset, I identified some cleanup and code issues in cpr-exec and cpr-transfer. I may send fixes for them separately. [PATCH v16 00/22] migration/cpr: support vhost-vsock devices https://lore.kernel.org/all/[email protected]/ Example: Add "kho=on liveupdate=on" to host kernel cmdline. Create QEMU instance with memfd. qemu-system-x86_64 \ -uuid 6f1b9f5a-6b92-4a5d-ae57-7a0ddabc1234 \ -machine q35,dump-guest-core=off,memory-backend=ram0,aux-ram-share=on \ -accel kvm -cpu host \ -name debug-threads=on \ -m 8G -smp 4 \ -object memory-backend-memfd,id=ram0,size=8G,share=on \ -device virtio-blk-pci,drive=drive0,id=virtblk0,num-queues=4 \ -drive file=boot.qcow2,if=none,id=drive0 \ -netdev user,id=net0,hostfwd=tcp::5028-:22 \ -device virtio-net-pci,netdev=net0 \ -vnc :8 -monitor stdio Don't terminate QEMU process after cpr-luo migration. (qemu) migrate_set_parameter mode cpr-luo (qemu) migrate -d file:cpr-luo.state (qemu) info status VM status: paused (postmigrate) Use kexec to reboot the KVM host. host# kexec -s -l /boot/vmlinuz-$(uname -r) --initrd=/boot/initramfs-$(uname -r).img --reuse-cmdline host# sync host# kexec -e Resume QEMU instance after kexec reboot. qemu-system-x86_64 \ -uuid 6f1b9f5a-6b92-4a5d-ae57-7a0ddabc1234 \ -machine q35,dump-guest-core=off,memory-backend=ram0,aux-ram-share=on \ -accel kvm -cpu host \ -name debug-threads=on \ -m 8G -smp 4 \ -object memory-backend-memfd,id=ram0,size=8G,share=on \ -device virtio-blk-pci,drive=drive0,id=virtblk0,num-queues=4 \ -drive file=boot.qcow2,if=none,id=drive0 \ -netdev user,id=net0,hostfwd=tcp::5028-:22 \ -device virtio-net-pci,netdev=net0 \ -vnc :8 -monitor stdio \ -incoming cpr-luo:defer (qemu) migrate_incoming file:cpr-luo.state (qemu) info status VM status: running Dongli Zhang (21): qapi/migration: add cpr-luo migration mode migration/cpr: move CprFd definition to header file scripts/update-linux-headers: Add liveupdate.h linux-headers: import liveupdate UAPI header migration/cpr-luo: require a non-null machine UUID migration/cpr-luo: add LUO session management migration/cpr-luo: add LUO fd management helpers migration/cpr-luo: track LUO fds migration/cpr-luo: add LUO fd VMStateDescription migration/cpr-luo: add tokens for LUO fds migration/cpr: move QEMUFile fd helpers to common code migration/cpr-luo: add state fd helpers migration/cpr: add more arguments for cpr_walk_fd() migration/cpr-luo: add helpers to preserve and retrieve all LUO fds migration/cpr-luo: enable cpr-luo mode for source side preservation migration/cpr-luo: enable cpr-luo mode for target side retrieval migration/cpr-luo: abort session when migration fails system/physmem: add cpr-luo blocker monitor/qmp: close LUO session when resuming source QEMU vfio: block cpr-luo migration migration: cpr-luo documentation backends/hostmem-memfd.c | 1 + backends/hostmem-shm.c | 1 + docs/devel/migration/CPR.rst | 64 +++++++ hw/vfio/cpr-iommufd.c | 17 +- hw/vfio/cpr-legacy.c | 8 + include/hw/vfio/vfio-cpr.h | 1 + include/migration/cpr.h | 37 +++- include/system/iommufd.h | 1 + include/system/system.h | 2 + linux-headers/linux/liveupdate.h | 239 +++++++++++++++++++++++++ migration/cpr-exec.c | 32 +--- migration/cpr-luo-stub.c | 78 ++++++++ migration/cpr-luo.c | 328 ++++++++++++++++++++++++++++++++++ migration/cpr.c | 126 +++++++++++-- migration/meson.build | 6 + migration/migration.c | 12 +- migration/options.c | 6 + migration/ram.c | 1 + monitor/qmp-cmds.c | 7 + qapi/migration.json | 11 +- scripts/update-linux-headers.sh | 2 +- system/globals.c | 5 + system/physmem.c | 9 +- system/vl.c | 9 + 24 files changed, 950 insertions(+), 53 deletions(-) base-commit: 3377670ee96b5a68361016680a95b9013709d4b7 Thank you very much! Dongli Zhang
