Dongli Zhang <[email protected]> writes: > I am rebasing the patchset on top of the most recent QEMU commit, especially > "migration/cpr: support vhost-vsock devices". > > Meanwhile, any feedback on the proposal is appreciated. >
Sorry for the delay, we're all swimming in slop these days. The only feedback I have at the moment is that this feature needs tests. What's your plan in this regard? Can this run with nested virtualization? Or perhaps is there a way to mock the kernel interfaces and ensure QEMU does the right thing even without them? We don't need much, but the code definitely needs to be exercised somehow. > So far, the main idea is to reuse as much code as possible from cpr-exec and > cpr-transfer. > > 1. Introduce the new cpr-luo mode. > 2. Use a UUID to identify a LUO session. > 3. Introduce APIs to create, preserve, and retrieve sessions and fds. > 4. As LUO uses the name, id, and token instead of an fd, create an independent > VMStateDescription rather than reuse the one for cpr-exec and cpr-transfer. > 5. Compute the token when transforming entries from the hash table into the > list. > 6. Preserve fds on the source. Follow the approach used in cpr-exec and > cpr-transfer. > 7. Retrieve fds on the target. Follow the approach used in cpr-exec and > cpr-transfer. The tricky part is where to set the cpr incoming mode. > 8. There is no need to modify save_fd() or load_fd(), as we only need to > migrate > the corresponding name and token, not the fds. > 9. Handle corner-case errors. > 10. Block VFIO. > > Thank you very much! > > Dongli Zhang > > On Mon, Oct 12, 2026 4:55:56PM -0700, Dongli Zhang wrote: >> This series introduces cpr-luo, a CPR migration mode that uses the Linux Live >> Update Orchestrator (LUO) and Kexec Handover (KHO) to support KVM host kernel >> updates through kexec. On the outgoing side, QEMU saves CPR state in a memfd >> and >> preserves >> >> This series introduces cpr-luo, a CPR migration mode that uses the Linux >> Live Update Orchestrator (LUO) and Kexec Handover (KHO) to support KVM host >> kernel updates through kexec. >> >> On the outgoing side, QEMU saves CPR state in a memfd and preserves that fd >> together with the RAMBlock backing fds in LUO. KHO carries the preserved LUO >> state into the new kernel. The incoming QEMU retrieves the CPR state and >> RAMBlock backing fds from LUO after kexec. >> >> The regular migration stream carries device vmstate. >> >> Although VFIO is currently blocked, I have an offline implementation that >> skips cpr-luo mode in cpr_is_incoming(), skips precopy, and carefully handles >> monitor_fd_param() although I never use it, so that VFIO is always >> reconstructed. It works with both iommufd and legacy for mlx5_vfio_pci by >> saving the VFIO and vendor specific state in vmstate as well. >> >> The result is that guest RAM remains available across a kexec-based host >> kernel update without copying its contents through the migration stream. >> >> Here is the main idea of the patchset: >> >> 1. Introduce the new cpr-luo mode. >> 2. Use a UUID to identify a LUO session. >> 3. Introduce APIs to create, preserve, and retrieve sessions and fds. >> 4. As LUO uses the name, id, and token instead of an fd, create an >> independent VMStateDescription rather than reuse the one for cpr-exec and >> cpr-transfer. >> 5. Compute the token when transforming entries from the hash table into the >> list. >> 6. Preserve fds on the source. Follow the approach used in cpr-exec and >> cpr-transfer. >> 7. Retrieve fds on the target. Follow the approach used in cpr-exec and >> cpr-transfer. The tricky part is where to set the cpr incoming mode. >> 8. There is no need to modify save_fd() or load_fd(), as we only need to >> migrate the corresponding name and token, not the fds. >> 9. Handle corner-case errors. >> 10. Block VFIO. >> >> I may need to rebase the current patchset on top of the patches below, >> from which I learned quite a lot in each iteration. While working on this >> patchset, I identified some cleanup and code issues in cpr-exec and >> cpr-transfer. I may send fixes for them separately. >> >> [PATCH v16 00/22] migration/cpr: support vhost-vsock devices >> https://urldefense.com/v3/__https://lore.kernel.org/all/20261012-up-vsock-vhost- >> [email protected]/__;!!ACWV5N9M2RV99hQ!N58C4d74- >> zpeHeXtxKWUAIRgf18-Xzk7_V7p49rzvh29deg5g2oBeAn0R5b89vJzLBPv6rkzjTo-8F0Iy16Z0oRy$ >> >> <https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]/__;!!ACWV5N9M2RV99hQ!N58C4d74-zpeHeXtxKWUAIRgf18-Xzk7_V7p49rzvh29deg5g2oBeAn0R5b89vJzLBPv6rkzjTo-8F0Iy16Z0oRy$> >> >> Example: >> >> Add "kho=on liveupdate=on" to host kernel cmdline. >> >> Create QEMU instance with memfd. >> >> qemu-system-x86_64 \ >> -uuid 6f1b9f5a-6b92-4a5d-ae57-7a0ddabc1234 \ >> -machine q35,dump-guest-core=off,memory-backend=ram0,aux-ram-share=on \ >> -accel kvm -cpu host \ >> -name debug-threads=on \ >> -m 8G -smp 4 \ >> -object memory-backend-memfd,id=ram0,size=8G,share=on \ >> -device virtio-blk-pci,drive=drive0,id=virtblk0,num-queues=4 \ >> -drive file=boot.qcow2,if=none,id=drive0 \ >> -netdev user,id=net0,hostfwd=tcp::5028-:22 \ >> -device virtio-net-pci,netdev=net0 \ >> -vnc :8 -monitor stdio >> >> Don't terminate QEMU process after cpr-luo migration. >> >> (qemu) migrate_set_parameter mode cpr-luo >> (qemu) migrate -d file:cpr-luo.state >> (qemu) info status >> VM status: paused (postmigrate) >> >> Use kexec to reboot the KVM host. >> >> host# kexec -s -l /boot/vmlinuz-$(uname -r) --initrd=/boot/initramfs-$(uname >> -r).img --reuse-cmdline >> host# sync >> host# kexec -e >> >> Resume QEMU instance after kexec reboot. >> >> qemu-system-x86_64 \ >> -uuid 6f1b9f5a-6b92-4a5d-ae57-7a0ddabc1234 \ >> -machine q35,dump-guest-core=off,memory-backend=ram0,aux-ram-share=on \ >> -accel kvm -cpu host \ >> -name debug-threads=on \ >> -m 8G -smp 4 \ >> -object memory-backend-memfd,id=ram0,size=8G,share=on \ >> -device virtio-blk-pci,drive=drive0,id=virtblk0,num-queues=4 \ >> -drive file=boot.qcow2,if=none,id=drive0 \ >> -netdev user,id=net0,hostfwd=tcp::5028-:22 \ >> -device virtio-net-pci,netdev=net0 \ >> -vnc :8 -monitor stdio \ >> -incoming cpr-luo:defer >> >> (qemu) migrate_incoming file:cpr-luo.state >> (qemu) info status >> VM status: running >> >> >> Dongli Zhang (21): >> qapi/migration: add cpr-luo migration mode >> migration/cpr: move CprFd definition to header file >> scripts/update-linux-headers: Add liveupdate.h >> linux-headers: import liveupdate UAPI header >> migration/cpr-luo: require a non-null machine UUID >> migration/cpr-luo: add LUO session management >> migration/cpr-luo: add LUO fd management helpers >> migration/cpr-luo: track LUO fds >> migration/cpr-luo: add LUO fd VMStateDescription >> migration/cpr-luo: add tokens for LUO fds >> migration/cpr: move QEMUFile fd helpers to common code >> migration/cpr-luo: add state fd helpers >> migration/cpr: add more arguments for cpr_walk_fd() >> migration/cpr-luo: add helpers to preserve and retrieve all LUO fds >> migration/cpr-luo: enable cpr-luo mode for source side preservation >> migration/cpr-luo: enable cpr-luo mode for target side retrieval >> migration/cpr-luo: abort session when migration fails >> system/physmem: add cpr-luo blocker >> monitor/qmp: close LUO session when resuming source QEMU >> vfio: block cpr-luo migration >> migration: cpr-luo documentation >> >> backends/hostmem-memfd.c | 1 + >> backends/hostmem-shm.c | 1 + >> docs/devel/migration/CPR.rst | 64 +++++++ >> hw/vfio/cpr-iommufd.c | 17 +- >> hw/vfio/cpr-legacy.c | 8 + >> include/hw/vfio/vfio-cpr.h | 1 + >> include/migration/cpr.h | 37 +++- >> include/system/iommufd.h | 1 + >> include/system/system.h | 2 + >> linux-headers/linux/liveupdate.h | 239 +++++++++++++++++++++++++ >> migration/cpr-exec.c | 32 +--- >> migration/cpr-luo-stub.c | 78 ++++++++ >> migration/cpr-luo.c | 328 ++++++++++++++++++++++++++++++++++ >> migration/cpr.c | 126 +++++++++++-- >> migration/meson.build | 6 + >> migration/migration.c | 12 +- >> migration/options.c | 6 + >> migration/ram.c | 1 + >> monitor/qmp-cmds.c | 7 + >> qapi/migration.json | 11 +- >> scripts/update-linux-headers.sh | 2 +- >> system/globals.c | 5 + >> system/physmem.c | 9 +- >> system/vl.c | 9 + >> 24 files changed, 950 insertions(+), 53 deletions(-) >> >> base-commit: 3377670ee96b5a68361016680a95b9013709d4b7 >> >> Thank you very much! >> >> Dongli Zhang >> >>
