On Tue, 9 Jul 2024 13:58:53 -0700 Steve Sistare <steven.sist...@oracle.com> wrote:
> Enable vfio-pci devices to be saved and restored across a cpr-exec of qemu. > > At vfio creation time, save the value of vfio container, group, and device > descriptors in CPR state. > > In the container pre_save handler, suspend the use of virtual addresses > in DMA mappings with VFIO_DMA_UNMAP_FLAG_VADDR, because guest ram will > be remapped at a different VA after exec. DMA to already-mapped pages > continues. Save the msi message area as part of vfio-pci vmstate, and > save the interrupt and notifier eventfd's in vmstate. > > On qemu restart, vfio_realize() finds the saved descriptors, uses the > descriptors, and notes that the device is being reused. Device and iommu > state is already configured, so operations in vfio_realize that would > modify the configuration are skipped for a reused device, including vfio > ioctl's and writes to PCI configuration space. Vfio PCI device reset > is also suppressed. The result is that vfio_realize constructs qemu > data structures that reflect the current state of the device. However, > the reconstruction is not complete until migrate_incoming is called. > migrate_incoming loads the msi data, the vfio post_load handler finds > eventfds in CPR state, rebuilds vector data structures, and attaches the > interrupts to the new KVM instance. The container post_load handler then > invokes the main vfio listener callback, which walks the flattened ranges > of the vfio address space and calls VFIO_DMA_MAP_FLAG_VADDR to inform the > kernel of the new VA's. Lastly, migration resumes the VM. Hi Steve, What's the iommufd plan for cpr? Thanks, Alex
