On 7/27/26 22:36, Alex Williamson wrote:
On Mon, 27 Jul 2026 07:39:24 +0200
Cédric Le Goater <[email protected]> wrote:
Hello,
Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
growing requirement, but real hardware with migration support is
scarce and hard to debug. An emulated device provides a fully
controlled testbed for developing and validating the entire software
stack - vfio-pci variant drivers, VFIO core migration v2 framework,
QEMU, libvirt - and for tuning complex migration policies such as
downtime convergence. It also serves as an educational reference for
understanding VFIO migration end-to-end, from device state
serialization to dirty page tracking.
This series adds an experimental VF live migration interface to the
emulated igb (82576) device. It enables a vfio-pci variant driver
(igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
protocol with stop-copy and pre-copy support.
The target scenario is nested virtualization:
L0 QEMU (these patches)
igb PF with x-vf-migration=on
└── VFs with migration BAR + vendor cap
L1 kernel
igb-vfio-pci variant driver [1]
translates VFIO migration v2 ioctls → BAR2 MMIO
L1 QEMU (stock, unmodified)
vfio-pci device model, standard migration fd
L2 guest
standard igbvf driver, unaware of migration
The L1 QEMU is completely unmodified -- it sees a standard VFIO
migratable device and uses the normal migration fd path.
* Design
The migration interface is exposed through a hidden 64KB PCI BAR
(BAR2) on each VF, discovered via a vendor-specific PCI capability
("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
RESUMING, PRE_COPY).
I think you're placing the migration BAR on the VF in order to
implement this in a small footprint, QEMU + vfio-pci variant driver,
without PF guest driver changes.
yes.
A model that better matches real
world hardware might be to put the migration BAR on the PF, segmented
per VF, and then have the PF driver vend those segments out to the VF
drivers.
True.
On the migration topic, I saw that the SR-IOV specs had "VF Migration
State Array" feature, which was deprecated.
That would remove the BAR always mapped problem,
That's the main problem today.
but expands
the footprint to include the PF driver. However, we're not exactly
clean with respect to the PF driver as implemented here when we're
going around the PF driver's back to setup DMA mappings.
That's not uncommon today. The PDS vfio-pci variant driver setups DMA
mappings in the PF to migrate the VF. But yes, given that the IGB PF
has no idea that a VF could be migrated, it's a bit of a resource hijack.
Can we take advantage of the fact that this is a virtual device to
avoid all these warts?
For example, do we really need MMIO BAR space for the register set
exposed or can we prune that down to some key registers and doorbells
and move the rest to memory? We can put the vendor capability in
extended config space to give ourselves more room to work with if
necessary.
There is plenty of space in the extended config space. The required
register set is relatively small.
We also don't really need to play by the physical rules for
access, the variant driver in the L1 kernel can allocate contiguous
ranges and write GPAs into config space registers.
yes.
L0 QEMU can just
write migration data and dirty bitmaps directly to those GPAs,
bypassing any pretense of DMA mapping.
yes that works.
A previous implementation of this proposal was allocating GPAs.
I then preferred a solution that was more PCI friendly. Anyhow,
it's not a problem to change it again, once we agree on the
HW interface.
There might be some tricks we can steal from virtio as it seems to
optionally honor things like vIOMMUs as well.
Yes. That's where the PCI DMAs mapping were interesting. I will
look at it.
Anyway, if we want to
confine the implementation to the virtual VF, avoiding dependencies on
the PF driver, both at the cross-driver API and device DMA state, I
think we can probably lean harder on QEMU being able to push data into
an arbitrary GPA regardless of the IO topology we're exposing.
So, the extra PCI BAR is indeed a problem and we should switch to
extended config space to avoid it. I have been exploring several
ideas and this proposal is an hybrid MMIO/shared buffer solution.
Here are my plans for the next.
Reduce the register set to a minimum and use a shared buffer for
all commands. The driver would allocate a single large enough buffer
for the largest command payload (DIRTY_QUERY with bitmap) and write
its GPA into BUF_ADDR once at init. Each operation is: fill buffer,
kick CTRL, poll STATUS.
Registers in extended config space :
0x00 Header Cap ID
0x04 CAPS Features, max_ranges, pgsizes
0x08 CTRL Doorbell (state transitions, dirty ops)
0x0C STATUS Completion + error code
0x10 BUF_ADDR_LO Shared buffer GPA low
0x14 BUF_ADDR_HI Shared buffer GPA high
Commands and payload :
1. SET_STATE
0x00 target_state driver Requested VFIO state
2. SAVE
0x00 data_size device Bytes written (32-bit)
0x04 reserved - May be more reserved bytes to align data
on 32bits
0x08 data[] device State blob
3. LOAD
0x00 data_size driver Bytes to load (32-bit)
0x04 reserved - May be more reserved
0x08 data[] driver State blob
4. DIRTY_ENABLE
0x00 iova driver Range start (64-bit)
0x08 size driver Range size (64-bit)
0x10 pgsize driver Page granularity
0x14 reserved -
5. DIRTY_DISABLE
6. DIRTY_QUERY
Request (driver):
0x00 iova driver Query range start (64-bit)
0x08 size driver Query range size (64-bit)
0x10 pgsize driver Page granularity (for queries)
0x14 reserved -
Response (device):
0x18 bitmap_size device Bytes in bitmap
0x1C dirty_pages device Set bits count
0x20 dma_writes device DMA writes since enable (64-bit)
0x28 reserved -
0x30 bitmap[] device Dirty page bitmap
7. GET_STATS
0x00 dma_writes device (32-bit)
0x04 dma_bytes device (64-bit)
0x0C dirty_pages_set device
0x10 dirty_pages_clr device
0x14 dirty_page_count device
0x18 dirty_query_cnt device
We could add a shared buffer header (not sure this is useful though)
0x00 command driver Operation to perform
0x04 status device Completion + error code
0x08 payload_size both Size of payload following header
0x0C reserved - Alignment
Do we want 'flags' anywhere ?
Thanks,
C.
