On 2026/09/03 4:20, Cédric Le Goater wrote:
Hello,
Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
growing requirement, but real hardware with migration support is
scarce and hard to debug. An emulated device provides a fully
controlled testbed for developing and validating the entire software
stack - vfio-pci variant drivers, VFIO core migration v2 framework,
QEMU, libvirt - and for tuning complex migration policies such as
downtime convergence. It also serves as an educational reference for
understanding VFIO migration end-to-end, from device state
serialization to dirty page tracking.
This series adds an experimental VF live migration interface to the
emulated igb (82576) device. It enables a vfio-pci variant driver
(igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
protocol with stop-copy and pre-copy support.
The target scenario is nested virtualization:
L0 QEMU (these patches)
igb PF with x-vf-migration=on
└── VFs with migration DVSEC
L1 kernel
igb-vfio-pci variant driver [1]
translates VFIO migration v2 ioctls → DVSEC config writes
L1 QEMU (stock, unmodified)
vfio-pci device model, standard migration fd
L2 guest
standard igbvf driver, unaware of migration
The L1 QEMU is completely unmodified -- it sees a standard VFIO
migratable device and uses the normal migration fd path.
* Design
The migration interface is exposed through a DVSEC (Designated
Vendor-Specific Extended Capability, PCIe cap id 0x23) at offset
0x160 in VF extended config space. The DVSEC uses a command doorbell
model - all commands are synchronous via PCI config space writes.
Device state is serialized as a versioned blob of per-VF register
(offset, value) pairs covering control, interrupt, RX/TX queue,
receive address (RA/RA2), etc. plus TX context descriptors and
VFRE/VFTE enable bits. The buffer address is a guest physical
address (GPA) written by the driver via virt_to_phys; the device
accesses guest RAM directly through the system address space.
Dirty page tracking is implemented with per-range bitmaps maintained
in IGBCore. All VF DMA paths in igb_core.c (TX data, RX data,
descriptor writeback) are instrumented to record touched pages. The
variant driver registers tracked IOVA ranges and queries dirty bitmaps
through a shared buffer. Buffer structures include len, flags, and
reserved fields for future extensibility.
* Caveats
The x-vf-migration property is experimental (x- prefix, default off).
The dirty bitmaps are maintained inside the device, which is not
realistic for discrete NICs without on-chip DRAM.
* Testing
The target scenario is nested virtualization: L0 runs QEMU with an
igb PF (x-vf-migration=on), L1 runs the igb-vfio-pci variant driver
and an unmodified QEMU, and L2 runs a standard igbvf driver.
Migration under iperf3 load works correctly: dirty page tracking
converges (from ~2000 pages per PRE_COPY iteration down to ~280 at
STOP_COPY), and STOP_COPY stays under 250ms.
* Todo
1. Add migration blocker when x-vf-migration=on (no VMState yet) or
add VMState support for L0 migration (dirty bitmaps, tracking
engines, DVSEC registers, stats)
2. Add PRE_COPY state transfer to validate device INIT data (magic,
version, etc.)
3. Add qtests for migration state machine transitions, dirty page
tracking ?
* Ideas
1. RX bandwidth throttle (x-mig-rx-limit, uint32, default 0)
Return false from can_receive when the per-VF packet count in the
current tracking interval exceeds the limit. Reduces DMA writes
and dirty pages realistically.
2. Migration phase timing (GET_STATS extension)
Add per-VF timestamps: precopy_start_ns, stopcopy_start_ns,
precopy_duration_ns, stopcopy_duration_ns,
state_transition_count. Expose via GET_STATS.
3. Hot page simulation (x-mig-hot-pages, uint32, default 0)
Re-set the first N bitmap bits after each DIRTY_QUERY, simulating
workloads with hot pages that prevent convergence.
4. Error injection (x-mig-inject-error, uint32, default 0)
One-shot error code injection before command dispatch. A separate
x-mig-inject-dma-fail (bool) for persistent DMA failure testing.
* Credits
Alex Williamson suggested the overall approach of a variant driver
with the "x-vf-migration" device property to gate the feature. Thanks
for the ever ongoing support and valuable discussions throughout these
years.
* AI disclaimer
The lack of a migration-capable device has been a recurring pain point
for VFIO development over the years, and we hope this proposal
demonstrates the value of having one.
Claude was used to analyze the IGB PF and VF internal state and
identify the pain points of a working live migration of such devices.
The generated code served as a starting point but *significant* time
was then spent cleaning up, reworking, and shaping it into a clear,
reviewable IGB model extension.
As QEMU does not yet accept AI-assisted contributions, this series is
submitted as an RFC.
This largely repeats the concerns I raised previously [1]. Your
description of the *significant* time spent cleaning up and reworking
the generated code suggests that this series is intended for upstream
inclusion once the implementation direction is agreed. My concerns as a
maintainer are therefore correctness and maintainability.
The missing migration blocker is a concrete example: it is a basic
correctness issue that should be straightforward to address, yet it
remains a TODO. My comments on individual patches also raise the same
kinds of issues I pointed out in the previous version.
Many of the correctness problems found in v1 and v2 stem from the choice
to build on igb, which is more complex than virtio-net. That is why I
suggested using virtio-net as a simpler basis, unless there is a problem
that rules it out.
I would be willing to support allowing AI use in the project for work
like this. I would still need to see these correctness and
maintainability concerns addressed before I could support merging
the series.
Regards,
Akihiko Odaki
[1]
https://lore.kernel.org/qemu-devel/[email protected]/
