On Mon, Nov 02, 2020 at 11:00:12AM +0800, Jason Wang wrote: > > On 2020/10/30 下午7:13, Stefan Hajnoczi wrote: > > > I still don't get why it must be opaque. > > If the device state format needs to be in the VMM then each device > > needs explicit enablement in each VMM (QEMU, cloud-hypervisor, etc). > > > > Let's invert the question: why does the VMM need to understand the > > device state of a_passthrough_ device? > > > It's not a 100% passthrough device if you want to support live migration. > E.g the device state save and restore is not under the control of drivers in > the guest.
VFIO devices are already not pure passthrough (even without mdev) since the PCI bus is emulated and device-specific quirks may be implemented. Adding device state save/load does not change anything here. > And if I understand correctly, it usually requires device emulation or > mediation in either userspace or kernel to support e.g dirty page tracking > and other things. Breaking down the options further: 1. VFIO on physical PCI devices. Here I see two approaches: a. An mdev vendor driver implements the migration region described in Kirti's patch series. Individual device state fields are marshalled by the driver. b. The VFIO PCI core parses a PCI Capability that indicates migration support on the physical device and filters it out. The remainder of the device is passed through. The device state representation is saved/loaded by the physical hardware when the VFIO PCI core receives the ioctl and notifies the hardware. QEMU and host kernel code does not marshall individual device state fields. In the future it may be desirable to also expose the PCI Capability so that the guest is able to snapshot and restore the device. That could be useful for checkpointing AI/HPC workloads on GPUs, for example. I don't think there is a fundamental reason why device state save/load needs to be handled by the host except that VM live migration is supposed to be transparent to guests and cannot rely on guest cooperation. 2. vfio-user device backends. The device backend implements the save/load.
signature.asc
Description: PGP signature