On 7/9/2024 4:58 PM, Steve Sistare wrote:
Enable vfio-pci devices to be saved and restored across a cpr-exec of qemu.At vfio creation time, save the value of vfio container, group, and device descriptors in CPR state. In the container pre_save handler, suspend the use of virtual addresses in DMA mappings with VFIO_DMA_UNMAP_FLAG_VADDR, because guest ram will be remapped at a different VA after exec. DMA to already-mapped pages continues. Save the msi message area as part of vfio-pci vmstate, and save the interrupt and notifier eventfd's in vmstate. On qemu restart, vfio_realize() finds the saved descriptors, uses the descriptors, and notes that the device is being reused. Device and iommu state is already configured, so operations in vfio_realize that would modify the configuration are skipped for a reused device, including vfio ioctl's and writes to PCI configuration space. Vfio PCI device reset is also suppressed. The result is that vfio_realize constructs qemu data structures that reflect the current state of the device. However, the reconstruction is not complete until migrate_incoming is called. migrate_incoming loads the msi data, the vfio post_load handler finds eventfds in CPR state, rebuilds vector data structures, and attaches the interrupts to the new KVM instance. The container post_load handler then invokes the main vfio listener callback, which walks the flattened ranges of the vfio address space and calls VFIO_DMA_MAP_FLAG_VADDR to inform the kernel of the new VA's. Lastly, migration resumes the VM. This functionality is delivered by 3 patches for clarity. Part 1 handles device file descriptors and DMA. Part 2 adds eventfd and MSI/MSI-X vector support. Part 3 adds INTX support. [...] diff --git a/hw/vfio/cpr-legacy.c b/hw/vfio/cpr-legacy.c new file mode 100644 index 0000000..bc51ebe --- /dev/null +++ b/hw/vfio/cpr-legacy.c [...] + +bool vfio_legacy_cpr_register_container(VFIOContainerBase *bcontainer, + Error **errp) +{ + VFIOContainer *container = VFIO_CONTAINER(bcontainer); + + if (!vfio_can_cpr_exec(container, &bcontainer->cpr_blocker)) { + return migrate_add_blocker_modes(&bcontainer->cpr_blocker, errp, + MIG_MODE_CPR_EXEC, -1);
This is a bug. With the change in cpr_register return type to bool, this should be: return migrate_add_blocker_modes(...) == 0; - Steve
