Dongli Zhang <[email protected]> writes:

> I am rebasing the patchset on top of the most recent QEMU commit, especially
> "migration/cpr: support vhost-vsock devices".
>
> Meanwhile, any feedback on the proposal is appreciated.
>

Sorry for the delay, we're all swimming in slop these days. The only
feedback I have at the moment is that this feature needs tests. What's
your plan in this regard? Can this run with nested virtualization? Or
perhaps is there a way to mock the kernel interfaces and ensure QEMU
does the right thing even without them? We don't need much, but the code
definitely needs to be exercised somehow.

> So far, the main idea is to reuse as much code as possible from cpr-exec and
> cpr-transfer.
>
> 1. Introduce the new cpr-luo mode.
> 2. Use a UUID to identify a LUO session.
> 3. Introduce APIs to create, preserve, and retrieve sessions and fds.
> 4. As LUO uses the name, id, and token instead of an fd, create an independent
> VMStateDescription rather than reuse the one for cpr-exec and cpr-transfer.
> 5. Compute the token when transforming entries from the hash table into the 
> list.
> 6. Preserve fds on the source. Follow the approach used in cpr-exec and
> cpr-transfer.
> 7. Retrieve fds on the target. Follow the approach used in cpr-exec and
> cpr-transfer. The tricky part is where to set the cpr incoming mode.
> 8. There is no need to modify save_fd() or load_fd(), as we only need to 
> migrate
> the corresponding name and token, not the fds.
> 9. Handle corner-case errors.
> 10. Block VFIO.
>
> Thank you very much!
>
> Dongli Zhang
>
> On Mon, Oct 12, 2026 4:55:56PM -0700, Dongli Zhang wrote:
>> This series introduces cpr-luo, a CPR migration mode that uses the Linux Live
>> Update Orchestrator (LUO) and Kexec Handover (KHO) to support KVM host kernel
>> updates through kexec. On the outgoing side, QEMU saves CPR state in a memfd 
>> and
>> preserves
>> 
>> This series introduces cpr-luo, a CPR migration mode that uses the Linux
>> Live Update Orchestrator (LUO) and Kexec Handover (KHO) to support KVM host
>> kernel updates through kexec.
>> 
>> On the outgoing side, QEMU saves CPR state in a memfd and preserves that fd
>> together with the RAMBlock backing fds in LUO. KHO carries the preserved LUO
>> state into the new kernel. The incoming QEMU retrieves the CPR state and
>> RAMBlock backing fds from LUO after kexec.
>> 
>> The regular migration stream carries device vmstate.
>> 
>> Although VFIO is currently blocked, I have an offline implementation that
>> skips cpr-luo mode in cpr_is_incoming(), skips precopy, and carefully handles
>> monitor_fd_param() although I never use it, so that VFIO is always
>> reconstructed. It works with both iommufd and legacy for mlx5_vfio_pci by
>> saving the VFIO and vendor specific state in vmstate as well.
>> 
>> The result is that guest RAM remains available across a kexec-based host
>> kernel update without copying its contents through the migration stream.
>> 
>> Here is the main idea of the patchset:
>> 
>> 1. Introduce the new cpr-luo mode.
>> 2. Use a UUID to identify a LUO session.
>> 3. Introduce APIs to create, preserve, and retrieve sessions and fds.
>> 4. As LUO uses the name, id, and token instead of an fd, create an
>> independent VMStateDescription rather than reuse the one for cpr-exec and
>> cpr-transfer.
>> 5. Compute the token when transforming entries from the hash table into the
>> list.
>> 6. Preserve fds on the source. Follow the approach used in cpr-exec and
>> cpr-transfer.
>> 7. Retrieve fds on the target. Follow the approach used in cpr-exec and
>> cpr-transfer. The tricky part is where to set the cpr incoming mode.
>> 8. There is no need to modify save_fd() or load_fd(), as we only need to
>> migrate the corresponding name and token, not the fds.
>> 9. Handle corner-case errors.
>> 10. Block VFIO.
>> 
>> I may need to rebase the current patchset on top of the patches below,
>> from which I learned quite a lot in each iteration. While working on this
>> patchset, I identified some cleanup and code issues in cpr-exec and
>> cpr-transfer. I may send fixes for them separately.
>> 
>> [PATCH v16 00/22] migration/cpr: support vhost-vsock devices
>> https://urldefense.com/v3/__https://lore.kernel.org/all/20261012-up-vsock-vhost-
>> [email protected]/__;!!ACWV5N9M2RV99hQ!N58C4d74-
>> zpeHeXtxKWUAIRgf18-Xzk7_V7p49rzvh29deg5g2oBeAn0R5b89vJzLBPv6rkzjTo-8F0Iy16Z0oRy$
>>  
>> <https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]/__;!!ACWV5N9M2RV99hQ!N58C4d74-zpeHeXtxKWUAIRgf18-Xzk7_V7p49rzvh29deg5g2oBeAn0R5b89vJzLBPv6rkzjTo-8F0Iy16Z0oRy$>
>> 
>> Example:
>> 
>> Add "kho=on liveupdate=on" to host kernel cmdline.
>> 
>> Create QEMU instance with memfd.
>> 
>> qemu-system-x86_64 \
>> -uuid 6f1b9f5a-6b92-4a5d-ae57-7a0ddabc1234 \
>> -machine q35,dump-guest-core=off,memory-backend=ram0,aux-ram-share=on \
>> -accel kvm -cpu host \
>> -name debug-threads=on \
>> -m 8G -smp 4 \
>> -object memory-backend-memfd,id=ram0,size=8G,share=on \
>> -device virtio-blk-pci,drive=drive0,id=virtblk0,num-queues=4 \
>> -drive file=boot.qcow2,if=none,id=drive0 \
>> -netdev user,id=net0,hostfwd=tcp::5028-:22 \
>> -device virtio-net-pci,netdev=net0 \
>> -vnc :8 -monitor stdio
>> 
>> Don't terminate QEMU process after cpr-luo migration.
>> 
>> (qemu) migrate_set_parameter mode cpr-luo
>> (qemu) migrate -d file:cpr-luo.state
>> (qemu) info status
>> VM status: paused (postmigrate)
>> 
>> Use kexec to reboot the KVM host.
>> 
>> host# kexec -s -l /boot/vmlinuz-$(uname -r) --initrd=/boot/initramfs-$(uname 
>> -r).img --reuse-cmdline
>> host# sync
>> host# kexec -e
>> 
>> Resume QEMU instance after kexec reboot.
>> 
>> qemu-system-x86_64 \
>> -uuid 6f1b9f5a-6b92-4a5d-ae57-7a0ddabc1234 \
>> -machine q35,dump-guest-core=off,memory-backend=ram0,aux-ram-share=on \
>> -accel kvm -cpu host \
>> -name debug-threads=on \
>> -m 8G -smp 4 \
>> -object memory-backend-memfd,id=ram0,size=8G,share=on \
>> -device virtio-blk-pci,drive=drive0,id=virtblk0,num-queues=4 \
>> -drive file=boot.qcow2,if=none,id=drive0 \
>> -netdev user,id=net0,hostfwd=tcp::5028-:22 \
>> -device virtio-net-pci,netdev=net0 \
>> -vnc :8 -monitor stdio \
>> -incoming cpr-luo:defer
>> 
>> (qemu) migrate_incoming file:cpr-luo.state
>> (qemu) info status
>> VM status: running
>> 
>> 
>> Dongli Zhang (21):
>>   qapi/migration: add cpr-luo migration mode
>>   migration/cpr: move CprFd definition to header file
>>   scripts/update-linux-headers: Add liveupdate.h
>>   linux-headers: import liveupdate UAPI header
>>   migration/cpr-luo: require a non-null machine UUID
>>   migration/cpr-luo: add LUO session management
>>   migration/cpr-luo: add LUO fd management helpers
>>   migration/cpr-luo: track LUO fds
>>   migration/cpr-luo: add LUO fd VMStateDescription
>>   migration/cpr-luo: add tokens for LUO fds
>>   migration/cpr: move QEMUFile fd helpers to common code
>>   migration/cpr-luo: add state fd helpers
>>   migration/cpr: add more arguments for cpr_walk_fd()
>>   migration/cpr-luo: add helpers to preserve and retrieve all LUO fds
>>   migration/cpr-luo: enable cpr-luo mode for source side preservation
>>   migration/cpr-luo: enable cpr-luo mode for target side retrieval
>>   migration/cpr-luo: abort session when migration fails
>>   system/physmem: add cpr-luo blocker
>>   monitor/qmp: close LUO session when resuming source QEMU
>>   vfio: block cpr-luo migration
>>   migration: cpr-luo documentation
>> 
>>  backends/hostmem-memfd.c         |   1 +
>>  backends/hostmem-shm.c           |   1 +
>>  docs/devel/migration/CPR.rst     |  64 +++++++
>>  hw/vfio/cpr-iommufd.c            |  17 +-
>>  hw/vfio/cpr-legacy.c             |   8 +
>>  include/hw/vfio/vfio-cpr.h       |   1 +
>>  include/migration/cpr.h          |  37 +++-
>>  include/system/iommufd.h         |   1 +
>>  include/system/system.h          |   2 +
>>  linux-headers/linux/liveupdate.h | 239 +++++++++++++++++++++++++
>>  migration/cpr-exec.c             |  32 +---
>>  migration/cpr-luo-stub.c         |  78 ++++++++
>>  migration/cpr-luo.c              | 328 ++++++++++++++++++++++++++++++++++
>>  migration/cpr.c                  | 126 +++++++++++--
>>  migration/meson.build            |   6 +
>>  migration/migration.c            |  12 +-
>>  migration/options.c              |   6 +
>>  migration/ram.c                  |   1 +
>>  monitor/qmp-cmds.c               |   7 +
>>  qapi/migration.json              |  11 +-
>>  scripts/update-linux-headers.sh  |   2 +-
>>  system/globals.c                 |   5 +
>>  system/physmem.c                 |   9 +-
>>  system/vl.c                      |   9 +
>>  24 files changed, 950 insertions(+), 53 deletions(-)
>> 
>> base-commit: 3377670ee96b5a68361016680a95b9013709d4b7
>> 
>> Thank you very much!
>> 
>> Dongli Zhang
>> 
>> 

Reply via email to