Akihiko, On 7/27/26 07:39, Cédric Le Goater wrote:
Hello,Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a growing requirement, but real hardware with migration support is scarce and hard to debug. An emulated device provides a fully controlled testbed for developing and validating the entire software stack - vfio-pci variant drivers, VFIO core migration v2 framework, QEMU, libvirt - and for tuning complex migration policies such as downtime convergence. It also serves as an educational reference for understanding VFIO migration end-to-end, from device state serialization to dirty page tracking. This series adds an experimental VF live migration interface to the emulated igb (82576) device. It enables a vfio-pci variant driver (igb-vfio-pci) to migrate VFs using the standard VFIO migration v2 protocol with stop-copy and pre-copy support. The target scenario is nested virtualization: L0 QEMU (these patches) igb PF with x-vf-migration=on └── VFs with migration BAR + vendor cap L1 kernel igb-vfio-pci variant driver [1] translates VFIO migration v2 ioctls → BAR2 MMIO L1 QEMU (stock, unmodified) vfio-pci device model, standard migration fd L2 guest standard igbvf driver, unaware of migration The L1 QEMU is completely unmodified -- it sees a standard VFIO migratable device and uses the normal migration fd path. * Design The migration interface is exposed through a hidden 64KB PCI BAR (BAR2) on each VF, discovered via a vendor-specific PCI capability ("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY, RESUMING, PRE_COPY). Device state is serialized as a versioned blob of per-VF register (offset, value) pairs covering control, interrupt, RX/TX queue, receive address (RA/RA2), etc. plus TX context descriptors and VFRE/VFTE enable bits. The blob is transferred via DMA through the PF device, since VFIO owns the VF's IOMMU domain and the variant driver maps its DMA buffers through the PF. Dirty page tracking is implemented with per-range bitmaps maintained in IGBCore. All VF DMA paths in igb_core.c (TX data, RX data, descriptor writeback) are instrumented to record touched pages. The variant driver registers tracked IOVA ranges and queries dirty bitmaps through a DMA shared buffer, using a single MMIO doorbell (DIRTY_CTRL=QUERY) per iteration. * Caveats Patch 1 introduces PCI_BASE_ADDRESS_MEM_ALWAYS_ON, a QEMU-internal BAR flag that keeps the migration BAR mapped even after VFIO's Function Level Reset clears PCI_COMMAND_MEMORY. The x-vf-migration property is experimental (x- prefix, default off) and the migration BAR register interface may change. The dirty bitmaps are maintained inside the device, which is not realistic for discrete NICs without on-chip DRAM. * Testing The target scenario is nested virtualization: L0 runs QEMU with an igb PF (x-vf-migration=on), L1 runs the igb-vfio-pci variant driver and an unmodified QEMU, and L2 runs a standard igbvf driver. Migration under iperf3 load works correctly: dirty page tracking converges (from ~2000 pages per PRE_COPY iteration down to ~280 at STOP_COPY), and STOP_COPY stays under 250ms. * Todo/Ideas 1. Add migration blocker when x-vf-migration=on (no VMState yet) or add VMState support for L0 migration (dirty bitmaps, tracking engines, migration BAR registers, stats)2. Add PRE_COPY match data validation (magic, version, caps) 3. Support driver-provided DMA bitmaps per dirty range(DIRTY_RANGE_ADDR_LO/HI)4. Add QMP/HMP knobs for error injection and capability tuning: . Dirty rate throttling: artificial delay or rate limit on DMAdirty tracking to simulate different convergence scenarios . State blob size inflation: pad the migration blob to stress large-state transfers and test DATA_SIZE limits . Migration phase timing: expose per-VF counters for time spent in each state (PRE_COPY duration, STOP_COPY latency) . Hot page simulation: mark specific page ranges as always-dirty to test worst-case convergence . Error injection: force STATUS error codes, fail DMA transfers, corrupt state blobs, mask CAPS bits, etc.5. Add qtests for migration state machine transitions, dirty pagetracking ? * Credits Alex Williamson suggested the overall approach: a hidden migration BAR discovered via a vendor-specific PCI capability, the "vf-migration" device property to gate the feature. Thanks for the ever ongoing support and valuable discussions throughout these years. * AI disclaimer The lack of a migration-capable device has been a recurring pain point for VFIO development over the years, and we hope this proposal demonstrates the value of having one. Claude was used to analyze the IGB PF and VF internal state and identify the pain points of a working live migration of such devices. The generated code served as a starting point but *significant* time was then spent cleaning up, reworking, and shaping it into a clear, reviewable proposal. As QEMU does not yet accept AI-assisted contributions, this series is submitted as an RFC. Thanks, C. [1] https://github.com/legoater/vfio-pci-extras Cédric Le Goater (11): pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag igb: Add x-vf-migration property and vendor-specific capability for IGBVF igb: Add migration BAR with state machine igb: Add VF state serialization for live migration igb: Add VF post-load fixups for live migration igb: Add dirty page tracking for IGBVF migration igb: Quiesce VFs on STOP and include PF enable state in migration blob igb: Fix post-migration RX ring deadlock igb: Send RARP after VF migration to update bridge FDB docs: Add igb VF migration testing setup guide igb: Add migration statistics registers to VF migration BAR MAINTAINERS | 6 + docs/system/device-emulation.rst | 1 + docs/system/devices/igb-migration.rst | 313 +++++++ docs/system/devices/igb.rst | 6 + hw/net/igb_common.h | 11 + hw/net/igb_core.h | 8 + hw/net/igb_migration.h | 201 +++++ include/hw/pci/pci.h | 6 + hw/net/igb.c | 18 + hw/net/igb_core.c | 144 ++- hw/net/igb_migration.c | 1194 +++++++++++++++++++++++++ hw/net/igbvf.c | 29 +- hw/pci/pci.c | 6 +- hw/net/meson.build | 2 +- hw/net/trace-events | 20 + 15 files changed, 1939 insertions(+), 26 deletions(-) create mode 100644 docs/system/devices/igb-migration.rst create mode 100644 hw/net/igb_migration.h create mode 100644 hw/net/igb_migration.c
First thanks for the quick review ! I will recap here : * bisectability, will improve the first patches. * state serialization, the blob management is awful. I was expecting flames. Yes. it needs a rework. I lack a qbuf-style put/get API. The contents of the blob need a review. Seems feasible. * relocation between VFs, on my TODO. The code already checks the vfn but needs tightening and better support. * interrupts bits: will look into it. hopefully, the model "only" needs to be more precise. This was a difficult aspect of the igb to understand and I am still learning. * dirty tracking, I worked on 2 different interfaces and some of the code fell through the cracks. DMA failures, bitmap clearing, size validation, all need improvements. Mostly addressed already, remaining items are code reorg. should be fine. * quiesce: looks like a bug in the model. Will check. * RARP : I got inspiration from the tests. you are right saying it belongs to the management layer. Keeping it for now as it simplifies testing, but will move it out. * Stats: the addition is from yesterday. Needs fixes indeed. Thanks, C.
