On 2026/07/27 14:39, Cédric Le Goater wrote:
Extend the migration BAR with dirty page tracking registers:
0x020 DIRTY_PGSIZE (RW) - dirty tracking page granularity
0x024 DIRTY_CTRL (WO) - 0=DISABLE, 1=ENABLE, 2=QUERY
0x028 DIRTY_RANGE_IOVA_LO (WO) - low 32 bits of tracked range start
0x02C DIRTY_RANGE_IOVA_HI (WO) - high 32 bits of tracked range start
0x030 DIRTY_RANGE_SIZE (WO) - tracked range size in bytes
0x034 DIRTY_BUF_ADDR_LO (WO) - low 32 bits of shared buffer address
0x038 DIRTY_BUF_ADDR_HI (WO) - high 32 bits of shared buffer address
0x03C DIRTY_STATUS (RO) - result of last DIRTY_CTRL
The CAPS register advertises the maximum number of ranges and
supported page sizes. The driver enables tracking on IOVA ranges
via DIRTY_CTRL and queries dirty bitmaps through a shared buffer.
As for state transfers, dirty buffer DMA is performed through the
PF device (pcie_sriov_get_pf).
Each tracked range maintains its own bitmap scoped to its boundaries.
DMA paths (TX/RX data, descriptor writeback) are instrumented to
record the touched pages. This enables the PRE_COPY state where the
driver iterates on dirty pages while the VM continues to run.
Assisted-by: Claude
Signed-off-by: Cédric Le Goater <[email protected]>
---
docs/system/devices/igb-migration.rst | 73 +++++
hw/net/igb_core.h | 2 +
hw/net/igb_migration.h | 82 ++++++
hw/net/igb_core.c | 48 ++--
hw/net/igb_migration.c | 367 +++++++++++++++++++++++++-
hw/net/igbvf.c | 4 +
hw/net/trace-events | 7 +
7 files changed, 562 insertions(+), 21 deletions(-)
diff --git a/docs/system/devices/igb-migration.rst
b/docs/system/devices/igb-migration.rst
index 04c44ef072af..4a3a93f01dfc 100644
--- a/docs/system/devices/igb-migration.rst
+++ b/docs/system/devices/igb-migration.rst
@@ -37,6 +37,14 @@ the following register layout::
0x014 DATA_XFER WO Trigger DMA save or DMA load
0x018 DATA_BUF_ADDR_LO WO Low 32 bits of state DMA buffer address
0x01C DATA_BUF_ADDR_HI WO High 32 bits of state DMA buffer address
+ 0x020 DIRTY_PGSIZE RW Dirty tracking page granularity
+ 0x024 DIRTY_CTRL WO 0=DISABLE, 1=ENABLE, 2=QUERY
+ 0x028 DIRTY_RANGE_IOVA_LO WO Low 32 bits of tracked range start
+ 0x02C DIRTY_RANGE_IOVA_HI WO High 32 bits of tracked range start
+ 0x030 DIRTY_RANGE_SIZE WO Tracked range size in bytes
+ 0x034 DIRTY_BUF_ADDR_LO WO Low 32 bits of shared buffer address
+ 0x038 DIRTY_BUF_ADDR_HI WO High 32 bits of shared buffer address
+ 0x03C DIRTY_STATUS RO Result of last DIRTY_CTRL (0=OK,
1-5=error)
State transitions follow the VFIO migration state machine: the driver
writes to ``DEVICE_STATE`` to move between states and reads ``STATUS``
@@ -61,3 +69,68 @@ code identifying the failure::
4 BAD_VFN VF number mismatch (source != destination)
5 DMA_FAILED DMA transfer to/from state buffer failed
6 NO_BUFFER DATA_XFER without buffer address set
+
+Dirty page tracking
+~~~~~~~~~~~~~~~~~~~
+
+The migration interface supports per-VF dirty page tracking, advertised
+by the ``F_DIRTY`` flag in ``CAPS``. This allows the variant driver to
+enter ``PRE_COPY`` state (``DEVICE_STATE`` = 5) while the VM continues
+to run, iterating on dirty pages to reduce the final stop-and-copy
+window.
+
+The device maintains one dirty tracking engine per range, each with its
+own bitmap scoped to the range boundaries. The ``CAPS`` register
+advertises the maximum number of ranges (``max_ranges`` in bits [11:8])
+and supported page sizes (bits [31:12]).
+
+Dirty tracking is controlled through the ``DIRTY_CTRL`` register:
+
+- **ENABLE** (1): the driver programs a tracked range via
+ ``DIRTY_RANGE_IOVA_LO/HI`` + ``DIRTY_RANGE_SIZE`` then writes
+ ``DIRTY_CTRL=ENABLE``. The device allocates a fixed-size bitmap for
+ the range and begins recording pages touched by DMA (TX data, RX
+ data, descriptor writeback). The page granularity is set by
+ ``DIRTY_PGSIZE`` (default 4096). After each ENABLE the driver reads
+ ``DIRTY_STATUS`` to check for errors.
+- **DISABLE** (0): tears down all ranges and stops tracking.
+- **QUERY** (2): the driver writes (IOVA, size, page_size) into a
+ shared buffer registered via ``DIRTY_BUF_ADDR_LO/HI`` (PF DMA
+ address, as for state transfers), then writes
+ ``DIRTY_CTRL=QUERY``. The device finds the matching range, copies
+ the dirty bitmap into the shared buffer, clears the tracked bits,
+ and sets the buffer's completion status.
+
+``DIRTY_STATUS`` values after each ``DIRTY_CTRL`` write::
+
+ 0 OK Success
+ 1 TOO_MANY_RANGES Exceeds max_ranges from CAPS
+ 2 BAD_RANGE Invalid range (zero size)
+ 3 BAD_PGSIZE Invalid or misaligned page size
+ 4 NOT_ENABLED Query without prior enable
+ 5 NO_BUFFER Query without shared buffer
+
+Dirty query shared buffer
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The shared buffer used for dirty queries is registered via
+``DIRTY_BUF_ADDR_LO/HI`` (PF DMA address). It is cache-line aligned
+(64 bytes) to separate driver-written request fields from
+device-written completion fields::
+
+ Offset Field Written by Description
+ 0x00 iova driver Query range start
+ 0x08 size driver Query range size
+ 0x10 page_size driver Page granularity
+ 0x14 flags driver Reserved (must be 0)
+ 0x18 reserved[10] - Pad to 64-byte cache line
+ 0x40 status device 0 = pending, 1 = complete
+ 0x44 bitmap_size device Bytes written to bitmap
+ 0x48 dirty_page_count device Number of set bits in bitmap
+ 0x4C reserved[12] - Pad to 64-byte cache line
+ 0x80 bitmap[] device Dirty page bitmap
+
+The driver fills the request fields, issues ``DIRTY_CTRL=QUERY``, and
+polls ``status`` for completion. The device reads the request, writes
+the dirty bitmap and completion fields via DMA, then sets
+``status = 1``.
diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
index 150567eb346d..50cee2c7683c 100644
--- a/hw/net/igb_core.h
+++ b/hw/net/igb_core.h
@@ -83,6 +83,8 @@ struct IGBCore {
struct NetTxPkt *tx_pkt;
} tx[IGB_NUM_QUEUES];
+ IGBVfDirtyState vf_dirty[IGB_MAX_VF_FUNCTIONS];
+
struct NetRxPkt *rx_pkt;
bool has_vnet;
diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h
index 739a189810b0..a2e7b363cb04 100644
--- a/hw/net/igb_migration.h
+++ b/hw/net/igb_migration.h
@@ -27,6 +27,12 @@
#define IGB_MIG_CAP_F_STATE (1u << 0) /* device state serialization */
#define IGB_MIG_CAP_F_DIRTY (1u << 1) /* dirty page tracking */
+/* MIG_CAPS register layout (read-only, offset 0x008) */
+#define IGB_MIG_CAPS_MAX_RANGES_SHIFT 8
+#define IGB_MIG_CAPS_MAX_RANGES_MASK (0xfu << 8) /* bits [11:8] */
+#define IGB_MIG_CAPS_MAX_RANGES 4
+#define IGB_MIG_CAPS_PGSIZES_MASK 0xfffff000u /* bits [31:12] */
+
#define IGB_MIG_CAP_SIZE 16
#define IGB_MIG_CAP_OFF_MAGIC 4 /* offset within cap for magic field */
#define IGB_MIG_CAP_OFF_BARID 8 /* offset within cap for BAR id */
@@ -49,6 +55,14 @@
#define IGB_MIG_DATA_XFER 0x014
#define IGB_MIG_DATA_BUF_ADDR_LO 0x018
#define IGB_MIG_DATA_BUF_ADDR_HI 0x01C
+#define IGB_MIG_DIRTY_PGSIZE 0x020
+#define IGB_MIG_DIRTY_CTRL 0x024
+#define IGB_MIG_DIRTY_RANGE_IOVA_LO 0x028
+#define IGB_MIG_DIRTY_RANGE_IOVA_HI 0x02C
+#define IGB_MIG_DIRTY_RANGE_SIZE 0x030
+#define IGB_MIG_DIRTY_BUF_ADDR_LO 0x034
+#define IGB_MIG_DIRTY_BUF_ADDR_HI 0x038
+#define IGB_MIG_DIRTY_STATUS 0x03C
/* DEVICE_STATE values - mirrors VFIO migration states */
#define IGB_MIG_STATE_ERROR 0
@@ -75,6 +89,50 @@
#define IGB_MIG_ERR_DMA_FAILED 5
#define IGB_MIG_ERR_NO_BUFFER 6
+/*
+ * DIRTY_CTRL register values. Write one of these to control
+ * the dirty page tracking state machine.
+ */
+#define IGB_MIG_DIRTY_CTRL_DISABLE 0
+#define IGB_MIG_DIRTY_CTRL_ENABLE 1
+#define IGB_MIG_DIRTY_CTRL_QUERY 2 /* query-and-clear */
+
+/* DIRTY_STATUS register values (read-only, cleared on next DIRTY_CTRL write)
*/
+#define IGB_MIG_DIRTY_STATUS_OK 0
+#define IGB_MIG_DIRTY_STATUS_TOO_MANY_RANGES 1
+#define IGB_MIG_DIRTY_STATUS_BAD_RANGE 2
+#define IGB_MIG_DIRTY_STATUS_BAD_PGSIZE 3
+#define IGB_MIG_DIRTY_STATUS_NOT_ENABLED 4
+#define IGB_MIG_DIRTY_STATUS_NO_BUFFER 5
+
+#define IGB_MIG_DIRTY_DEFAULT_PGSIZE 4096
+
+/*
+ * Dirty query shared buffer layout - DMA between driver and device.
+ * The driver writes request fields, kicks DIRTY_CTRL=QUERY, and the
+ * device reads the request, fills bitmap + completion via DMA. Each
+ * section is cache-line aligned (64 bytes).
+ */
+struct igb_mig_dirty_query {
+ /* Cache line 0: request (written by driver) */
+ uint64_t iova;
+ uint64_t size;
+ uint32_t page_size;
+ uint32_t flags;
+ uint32_t reserved0[10];
+
+ /* Cache line 1: completion (written by device) */
+ uint32_t status;
+ uint32_t bitmap_size;
+ uint32_t dirty_page_count;
+ uint32_t reserved1[12];
+
+ /* Cache line 2+: bitmap (written by device) */
+ uint8_t bitmap[];
The bitmap is at offset 0x7c, not the promised 0x80.
+};
+
+#define IGB_MIG_DIRTY_STATUS_COMPLETE 1
+
typedef struct IgbVfMigState {
bool migration_cap;
MemoryRegion mig_bar;
@@ -84,6 +142,12 @@ typedef struct IgbVfMigState {
uint8_t mig_data[IGB_VF_STATE_MAX_SIZE];
uint32_t mig_data_size;
uint64_t mig_data_buf_addr;
+
+ uint32_t mig_dirty_pgsize;
+ uint64_t mig_dirty_range_iova;
+ uint32_t mig_dirty_range_size;
+ uint64_t mig_dirty_buf_addr;
+ uint32_t mig_dirty_status;
} IgbVfMigState;
typedef struct IgbVfState IgbVfState;
@@ -92,4 +156,22 @@ bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp);
void igbvf_mig_bar_init(IgbVfState *s);
void igbvf_mig_state_reset(IgbVfState *s);
+typedef struct IGBVfDirtyRange {
+ uint64_t iova;
+ uint64_t size;
+ uint64_t page_size;
+ unsigned long *bitmap;
+ uint64_t nbits;
+} IGBVfDirtyRange;
+
+typedef struct IGBVfDirtyState {
+ uint32_t num_ranges;
+ IGBVfDirtyRange ranges[IGB_MIG_CAPS_MAX_RANGES];
+} IGBVfDirtyState;
+
+typedef struct IGBCore IGBCore;
+void igb_core_dirty_track_dma(IGBCore *core, int vfn,
+ dma_addr_t addr, dma_addr_t len);
+void igb_core_vf_dirty_disable(IgbVfState *s);
+
#endif
diff --git a/hw/net/igb_core.c b/hw/net/igb_core.c
index 2565dd7f96d3..03c349575c0a 100644
--- a/hw/net/igb_core.c
+++ b/hw/net/igb_core.c
@@ -824,6 +824,14 @@ igb_rx_ring_init(IGBCore *core, E1000E_RxRing *rxr, int
idx)
rxr->i = &i[idx];
}
+static inline void
+igb_pci_dma_write(IGBCore *core, PCIDevice *dev, int vfn,
+ dma_addr_t addr, const void *buf, dma_addr_t len)
+{
+ pci_dma_write(dev, addr, buf, len);
+ igb_core_dirty_track_dma(core, vfn, addr, len);
+}
+
static uint32_t
igb_txdesc_writeback(IGBCore *core, dma_addr_t base,
union e1000_adv_tx_desc *tx_desc,
@@ -847,13 +855,15 @@ igb_txdesc_writeback(IGBCore *core, dma_addr_t base,
if (tdwba & 1) {
uint32_t buffer = cpu_to_le32(core->mac[txi->dh]);
- pci_dma_write(d, tdwba & ~3, &buffer, sizeof(buffer));
+ igb_pci_dma_write(core, d, txi->idx % IGB_NUM_VM_POOLS,
+ tdwba & ~3, &buffer, sizeof(buffer));
} else {
uint32_t status = le32_to_cpu(tx_desc->wb.status) | E1000_TXD_STAT_DD;
tx_desc->wb.status = cpu_to_le32(status);
- pci_dma_write(d, base + offsetof(union e1000_adv_tx_desc, wb),
- &tx_desc->wb, sizeof(tx_desc->wb));
+ igb_pci_dma_write(core, d, txi->idx % IGB_NUM_VM_POOLS,
+ base + offsetof(union e1000_adv_tx_desc, wb),
+ &tx_desc->wb, sizeof(tx_desc->wb));
}
return igb_tx_wb_eic(core, txi->idx);
@@ -1264,6 +1274,7 @@ typedef struct IGBPacketRxDMAState {
size_t iov_ofs;
bool do_ps;
bool is_first;
+ int vfn;
IGBBAState bastate;
hwaddr ba[IGB_MAX_PS_BUFFERS];
IGBSplitDescriptorData ps_desc_data;
@@ -1590,7 +1601,8 @@ igb_write_rx_descr(IGBCore *core,
static inline void
igb_pci_dma_write_rx_desc(IGBCore *core, PCIDevice *dev, dma_addr_t addr,
- union e1000_rx_desc_union *desc, dma_addr_t len)
+ union e1000_rx_desc_union *desc, dma_addr_t len,
+ int vfn)
{
if (igb_rx_use_legacy_descriptor(core)) {
struct e1000_rx_desc *d = &desc->legacy;
@@ -1598,11 +1610,12 @@ igb_pci_dma_write_rx_desc(IGBCore *core, PCIDevice
*dev, dma_addr_t addr,
uint8_t status = d->status;
d->status &= ~E1000_RXD_STAT_DD;
- pci_dma_write(dev, addr, desc, len);
+ igb_pci_dma_write(core, dev, vfn, addr, desc, len);
if (status & E1000_RXD_STAT_DD) {
d->status = status;
- pci_dma_write(dev, addr + offset, &status, sizeof(status));
+ igb_pci_dma_write(core, dev, vfn,
+ addr + offset, &status, sizeof(status));
}
} else {
union e1000_adv_rx_desc *d = &desc->adv;
@@ -1611,11 +1624,12 @@ igb_pci_dma_write_rx_desc(IGBCore *core, PCIDevice
*dev, dma_addr_t addr,
uint32_t status = d->wb.upper.status_error;
d->wb.upper.status_error &= ~E1000_RXD_STAT_DD;
- pci_dma_write(dev, addr, desc, len);
+ igb_pci_dma_write(core, dev, vfn, addr, desc, len);
if (status & E1000_RXD_STAT_DD) {
d->wb.upper.status_error = status;
- pci_dma_write(dev, addr + offset, &status, sizeof(status));
+ igb_pci_dma_write(core, dev, vfn,
+ addr + offset, &status, sizeof(status));
}
}
}
@@ -1737,9 +1751,9 @@ igb_write_hdr_frag_to_rx_buffers(IGBCore *core,
{
assert(data_len <= pdma_st->rx_desc_header_buf_size -
pdma_st->bastate.written[0]);
- pci_dma_write(d,
- pdma_st->ba[0] + pdma_st->bastate.written[0],
- data, data_len);
+ igb_pci_dma_write(core, d, pdma_st->vfn,
+ pdma_st->ba[0] + pdma_st->bastate.written[0],
+ data, data_len);
pdma_st->bastate.written[0] += data_len;
pdma_st->bastate.cur_idx = 1;
}
@@ -1804,10 +1818,10 @@ igb_write_payload_frag_to_rx_buffers(IGBCore *core,
data,
bytes_to_write);
- pci_dma_write(d,
- pdma_st->ba[pdma_st->bastate.cur_idx] +
- pdma_st->bastate.written[pdma_st->bastate.cur_idx],
- data, bytes_to_write);
+ igb_pci_dma_write(core, d, pdma_st->vfn,
+ pdma_st->ba[pdma_st->bastate.cur_idx] +
+ pdma_st->bastate.written[pdma_st->bastate.cur_idx],
+ data, bytes_to_write);
pdma_st->bastate.written[pdma_st->bastate.cur_idx] += bytes_to_write;
data += bytes_to_write;
@@ -1908,6 +1922,7 @@ igb_write_packet_to_guest(IGBCore *core, struct NetRxPkt
*pkt,
rxi = rxr->i;
rx_desc_len = core->rx_desc_len;
+ pdma_st.vfn = rxi->idx % IGB_NUM_VM_POOLS;
pdma_st.rx_desc_packet_buf_size = igb_rxbufsize(core, rxi);
pdma_st.rx_desc_header_buf_size = igb_rxhdrbufsize(core, rxi);
pdma_st.iov = net_rx_pkt_get_iovec(pkt);
@@ -1944,7 +1959,8 @@ igb_write_packet_to_guest(IGBCore *core, struct NetRxPkt
*pkt,
etqf, ts,
&pdma_st,
rxi);
- igb_pci_dma_write_rx_desc(core, d, base, &desc, rx_desc_len);
+ igb_pci_dma_write_rx_desc(core, d, base, &desc, rx_desc_len,
+ rxi->idx % IGB_NUM_VM_POOLS);
igb_ring_advance(core, rxi, rx_desc_len / E1000_MIN_RX_DESC_LEN);
} while (pdma_st.desc_offset < pdma_st.total_size);
diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
index 4f123df6795d..9a9ce902bb4a 100644
--- a/hw/net/igb_migration.c
+++ b/hw/net/igb_migration.c
@@ -8,6 +8,8 @@
#include "qemu/osdep.h"
#include "qemu/log.h"
+#include "qemu/units.h"
+#include "qemu/bitmap.h"
#include "hw/pci/pci_device.h"
#include "hw/pci/pcie.h"
#include "net/eth.h"
@@ -69,7 +71,7 @@ bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp)
pci_set_long(dev->config + offset + IGB_MIG_CAP_OFF_BARID,
IGB_MIG_BAR_IDX);
pci_set_long(dev->config + offset + IGB_MIG_CAP_OFF_FLAGS,
- IGB_MIG_CAP_F_STATE);
+ IGB_MIG_CAP_F_STATE | IGB_MIG_CAP_F_DIRTY);
trace_igbvf_mig_cap_add(pcie_sriov_vf_number(dev), offset);
return true;
@@ -399,6 +401,161 @@ static int igbvf_mig_load(IgbVfState *s, const void *buf,
size_t size)
return 0;
}
+/*
+ * =====================================================================
+ * Per-VF dirty page tracking
+ * =====================================================================
+ *
+ * All VF DMA writes in igb_core.c go through igb_pci_dma_write(),
+ * which calls igb_core_dirty_track_dma() to mark the target page in a
+ * per-range bitmap before performing the actual DMA.
+ *
+ * The IGBCore::vf_dirty[] bitmaps live in IGBCore so they are easily
+ * accessible from the core TX and RX paths without reaching back into
+ * VF state.
+ */
+
+void igb_core_dirty_track_dma(IGBCore *core, int vfn,
+ dma_addr_t addr, dma_addr_t len)
+{
+ IGBVfDirtyState *ds = &core->vf_dirty[vfn];
+ bool matched = false;
+ uint32_t i;
+
+ if (!ds->num_ranges) {
+ return;
+ }
+
+ trace_igb_core_dirty_track_dma(vfn, addr, len);
+
+ for (i = 0; i < ds->num_ranges; i++) {
+ IGBVfDirtyRange *r = &ds->ranges[i];
+ uint64_t r_end = r->iova + r->size;
+ uint64_t dma_end = addr + len;
+ uint64_t start, end, start_page, end_page, page;
+
+ if (addr >= r_end || dma_end <= r->iova) {
+ continue;
+ }
+
+ matched = true;
+ start = MAX(addr, r->iova);
+ end = MIN(dma_end, r_end);
+
+ start_page = (start - r->iova) / r->page_size;
+ end_page = (end - 1 - r->iova) / r->page_size;
+
+ for (page = start_page; page <= end_page; page++) {
+ if (page < r->nbits) {
+ set_bit(page, r->bitmap);
+ }
+ }
+ }
+
+ if (!matched) {
+ trace_igb_core_dirty_track_dma_drop(vfn, addr, len);
+ }
+}
+
+static IGBVfDirtyState *igb_core_vf_dirty_state(IgbVfState *s)
+{
+ IGBCore *core = igbvf_get_core(s);
+ return &core->vf_dirty[s->vfn];
+}
+
+#define IGB_MIG_DIRTY_MAX_PAGES ((256 * GiB) / (4 * KiB))
+
+static uint32_t igb_core_vf_dirty_enable(IgbVfState *s, uint64_t pgsize,
+ uint64_t range_iova, uint64_t range_size)
+{
+ IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
+ IGBVfDirtyRange *r;
+
+ if (ds->num_ranges >= IGB_MIG_CAPS_MAX_RANGES) {
+ return IGB_MIG_DIRTY_STATUS_TOO_MANY_RANGES;
+ }
+
+ if (!range_size) {
+ return IGB_MIG_DIRTY_STATUS_BAD_RANGE;
+ }
+
+ if (!pgsize || (range_iova % pgsize) || (range_size % pgsize)) {
+ return IGB_MIG_DIRTY_STATUS_BAD_PGSIZE;
+ }
+
+ if (range_size / pgsize > IGB_MIG_DIRTY_MAX_PAGES) {
+ return IGB_MIG_DIRTY_STATUS_BAD_RANGE;
+ }
+
+ r = &ds->ranges[ds->num_ranges];
+ r->iova = range_iova;
+ r->size = range_size;
+ r->page_size = pgsize;
+ r->nbits = range_size / pgsize;
+ r->bitmap = bitmap_new(r->nbits);
+ ds->num_ranges++;
+ return IGB_MIG_DIRTY_STATUS_OK;
+}
+
+void igb_core_vf_dirty_disable(IgbVfState *s)
+{
+ IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
+ uint32_t i;
+
+ for (i = 0; i < ds->num_ranges; i++) {
+ IGBVfDirtyRange *r = &ds->ranges[i];
+
+ g_free(r->bitmap);
+ r->bitmap = NULL;
+ r->nbits = 0;
+ }
+ ds->num_ranges = 0;
+}
+
+static bool igb_core_vf_dirty_enabled(IgbVfState *s)
+{
+ IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
+ return !!ds->num_ranges;
+}
+
+static bool igb_core_vf_dirty_query(IgbVfState *s,
+ void *buf, size_t buf_size, size_t *out_size,
+ uint64_t range_iova, uint64_t range_size)
+{
+ IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
+ uint32_t i;
+
+ for (i = 0; i < ds->num_ranges; i++) {
+ IGBVfDirtyRange *r = &ds->ranges[i];
+ uint64_t start_page, range_pages, count;
+
+ if (range_iova < r->iova ||
+ range_iova + range_size > r->iova + r->size) {
+ continue;
+ }
+
+ start_page = (range_iova - r->iova) / r->page_size;
+ range_pages = (uint64_t)range_size / r->page_size;
+ count = MIN(range_pages, (uint64_t)buf_size * 8);
+
+ memset(buf, 0, buf_size);
+
+ if (start_page < r->nbits) {
+ uint64_t avail = r->nbits - start_page;
+ uint64_t n = MIN(count, avail);
+
+ bitmap_copy_with_src_offset(buf, r->bitmap, start_page, n);
+ bitmap_clear(r->bitmap, start_page, n);
+ }
+
+ *out_size = bitmap_empty(buf, count) ? 0 : DIV_ROUND_UP(count, 8);
+ return true;
+ }
+
+ *out_size = 0;
+ return false;
+}
+
/* ================================================================
* Migration BAR register read/write handlers
* ================================================================ */
I forgot to point out this, but this does not match with QEMU's comment
style:
> Multiline comment blocks should have a row of stars on the left,
> and the initial /``*`` and terminating ``*``/ both on their own lines:
>
> .. code-block:: c
>
> /*
> * like
> * this
> */
Please also avoid decoration with "=". It has little benefit while
updating the comment a bit troublesome.
@@ -410,25 +567,53 @@ static bool igbvf_mig_set_state(IgbVfState *s, uint32_t
new_state)
int ret;
switch (new_state) {
+ case IGB_MIG_STATE_PRE_COPY:
+ if (old != IGB_MIG_STATE_RUNNING) {
+ return false;
+ }
+ /*
+ * Take an initial snapshot so DATA_SIZE reflects the actual
+ * state size and DATA_AVAIL is set for the driver.
+ */
+ ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
+ if (ret < 0) {
+ ms->mig_error = -ret;
+ ms->mig_state = IGB_MIG_STATE_ERROR;
+ return false;
+ }
+ ms->mig_data_size = ret;
+ break;
+
case IGB_MIG_STATE_STOP:
if (old != IGB_MIG_STATE_RUNNING &&
old != IGB_MIG_STATE_STOP_COPY &&
+ old != IGB_MIG_STATE_PRE_COPY &&
old != IGB_MIG_STATE_RESUMING &&
old != IGB_MIG_STATE_ERROR) {
return false;
}
/* Restore DATA_SIZE to max, same as at reset */
ms->mig_data_size = igb_core_vf_max_data_size(s);
+ if (old == IGB_MIG_STATE_PRE_COPY ||
+ old == IGB_MIG_STATE_STOP_COPY ||
+ old == IGB_MIG_STATE_ERROR) {
+ igb_core_vf_dirty_disable(s);
+ }
break;
case IGB_MIG_STATE_RUNNING:
- if (old != IGB_MIG_STATE_STOP) {
+ if (old != IGB_MIG_STATE_STOP &&
+ old != IGB_MIG_STATE_PRE_COPY) {
return false;
}
+ if (old == IGB_MIG_STATE_PRE_COPY) {
+ igb_core_vf_dirty_disable(s);
+ }
break;
case IGB_MIG_STATE_STOP_COPY:
- if (old != IGB_MIG_STATE_STOP) {
+ if (old != IGB_MIG_STATE_STOP &&
+ old != IGB_MIG_STATE_PRE_COPY) {
return false;
}
ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
@@ -466,7 +651,8 @@ static uint32_t igbvf_mig_get_status(IgbVfState *s)
if (ms->mig_state == IGB_MIG_STATE_ERROR) {
status |= IGB_MIG_STATUS_ERR(ms->mig_error);
}
- if (ms->mig_state == IGB_MIG_STATE_STOP_COPY && ms->mig_data_size > 0) {
+ if ((ms->mig_state == IGB_MIG_STATE_STOP_COPY ||
+ ms->mig_state == IGB_MIG_STATE_PRE_COPY) && ms->mig_data_size > 0) {
status |= IGB_MIG_STATUS_DATA_AVAIL;
}
@@ -486,6 +672,30 @@ static void igbvf_mig_data_xfer(IgbVfState *s, uint32_t val)
}
switch (ms->mig_state) {
+ case IGB_MIG_STATE_PRE_COPY:
+ /*
+ * Re-snapshot device state so the driver can read a fresh
+ * copy on each pre-copy iteration.
+ */
+ ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
+ if (ret < 0) {
+ ms->mig_error = -ret;
+ ms->mig_state = IGB_MIG_STATE_ERROR;
+ return;
+ }
+ ms->mig_data_size = ret;
+ r = pci_dma_write(pcie_sriov_get_pf(PCI_DEVICE(s)),
+ ms->mig_data_buf_addr,
+ ms->mig_data, ms->mig_data_size);
+ if (r != MEMTX_OK) {
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "igbvf: VF%u state write failed at 0x%" PRIx64 "\n",
+ s->vfn, ms->mig_data_buf_addr);
+ ms->mig_error = IGB_MIG_ERR_DMA_FAILED;
+ ms->mig_state = IGB_MIG_STATE_ERROR;
+ }
+ break;
+
case IGB_MIG_STATE_STOP_COPY:
/* Save: DMA-write serialized state to driver buffer */
r = pci_dma_write(pcie_sriov_get_pf(PCI_DEVICE(s)),
@@ -547,7 +757,9 @@ static uint64_t igbvf_mig_read(void *opaque, hwaddr addr,
unsigned size)
val = igbvf_mig_get_status(s);
break;
case IGB_MIG_CAPS:
- val = IGB_MIG_CAP_F_STATE;
+ val = IGB_MIG_CAP_F_STATE | IGB_MIG_CAP_F_DIRTY |
+ (IGB_MIG_CAPS_MAX_RANGES << IGB_MIG_CAPS_MAX_RANGES_SHIFT) |
+ (1u << 12); /* 4K page size supported */
break;
case IGB_MIG_VERSION:
val = IGB_MIG_CAP_VERSION;
@@ -555,6 +767,19 @@ static uint64_t igbvf_mig_read(void *opaque, hwaddr addr,
unsigned size)
case IGB_MIG_DATA_SIZE:
val = ms->mig_data_size;
break;
+ case IGB_MIG_DIRTY_PGSIZE:
+ val = ms->mig_dirty_pgsize ? ms->mig_dirty_pgsize
+ : IGB_MIG_DIRTY_DEFAULT_PGSIZE;
+ break;
+ case IGB_MIG_DIRTY_BUF_ADDR_LO:
+ val = (uint32_t)ms->mig_dirty_buf_addr;
+ break;
+ case IGB_MIG_DIRTY_BUF_ADDR_HI:
+ val = (uint32_t)(ms->mig_dirty_buf_addr >> 32);
+ break;
+ case IGB_MIG_DIRTY_STATUS:
+ val = ms->mig_dirty_status;
+ break;
default:
qemu_log_mask(LOG_GUEST_ERROR,
"igbvf: VF%u bad migration BAR read at 0x%"
@@ -567,6 +792,106 @@ static uint64_t igbvf_mig_read(void *opaque, hwaddr addr,
unsigned size)
return val;
}
+static uint32_t igbvf_mig_dirty_count(const void *bitmap, size_t size)
+{
+ const unsigned long *p = bitmap;
+ size_t n = size / sizeof(unsigned long);
+ uint32_t count = 0;
+
+ for (size_t i = 0; i < n; i++) {
+ count += ctpopl(p[i]);
+ }
+ return count;
+}
+
+static void igbvf_mig_dirty_query(IgbVfState *s, uint64_t pgsize)
+{
+ IgbVfMigState *ms = &s->mig;
+ PCIDevice *dev = pcie_sriov_get_pf(PCI_DEVICE(s));
+ uint64_t buf_addr = ms->mig_dirty_buf_addr;
+ uint64_t range_iova = 0, range_size = 0;
+ uint32_t bmp_bytes;
+ size_t out_size;
+ g_autofree void *bitmap = NULL;
+ bool valid;
+
+ ldq_le_pci_dma(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query, iova),
+ &range_iova, MEMTXATTRS_UNSPECIFIED);
+ ldq_le_pci_dma(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query, size),
+ &range_size, MEMTXATTRS_UNSPECIFIED);
+
+ bmp_bytes = DIV_ROUND_UP(range_size / pgsize, 8);
+ bitmap = g_malloc0(bmp_bytes);
The bitmap is later used with QEMU's `unsigned long` bitmap API. A
one-page query allocates one byte but accesses an eight-byte word on a
64-bit host: guest-triggerable heap corruption.
The untrusted 64-bit query size is allocated before it is matched
against an enabled range. For example, a 64-TiB request with 4-KiB pages
asks g_malloc0() for 2 GiB, allowing L1 to abort QEMU through memory
exhaustion.
+
+ valid = igb_core_vf_dirty_query(s, bitmap, bmp_bytes, &out_size,
+ range_iova, range_size);
+
+ if (valid && out_size) {
+ if (pci_dma_write(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query, bitmap),
+ bitmap, out_size)) {
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "igbvf: VF%u dirty bitmap write failed at 0x%" PRIx64 "\n",
+ s->vfn, buf_addr);
+ valid = false;
+ }
+ }
Dirty bits are cleared before the bitmap is DMA-written. A DMA failure
permanently loses dirty information and can corrupt the migrated guest.
+
+ uint32_t dirty_pages = valid ? igbvf_mig_dirty_count(bitmap, out_size) : 0;
+
+ stl_le_pci_dma(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query,
bitmap_size),
+ valid ? out_size : 0, MEMTXATTRS_UNSPECIFIED);
+ stl_le_pci_dma(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query,
dirty_page_count),
+ dirty_pages, MEMTXATTRS_UNSPECIFIED);
+ stl_le_pci_dma(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query, status),
+ valid ? IGB_MIG_DIRTY_STATUS_COMPLETE : 0,
+ MEMTXATTRS_UNSPECIFIED);
+
+ trace_igbvf_mig_dirty_query(s->vfn, (uint64_t)out_size, dirty_pages);
+}
+
+static void igbvf_mig_dirty_ctrl(IgbVfState *s, uint32_t val)
+{
+ IgbVfMigState *ms = &s->mig;
+ uint64_t pgsize = ms->mig_dirty_pgsize
+ ? ms->mig_dirty_pgsize : IGB_MIG_DIRTY_DEFAULT_PGSIZE;
+
+ switch (val) {
+ case IGB_MIG_DIRTY_CTRL_ENABLE:
+ ms->mig_dirty_status = igb_core_vf_dirty_enable(s, pgsize,
+ ms->mig_dirty_range_iova,
+ ms->mig_dirty_range_size);
+ if (ms->mig_dirty_status) {
+ break;
+ }
+ trace_igbvf_mig_dirty_enable(s->vfn, pgsize,
+ ms->mig_dirty_range_size / pgsize);
+ break;
+ case IGB_MIG_DIRTY_CTRL_DISABLE:
+ igb_core_vf_dirty_disable(s);
+ ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
+ trace_igbvf_mig_dirty_disable(s->vfn);
+ break;
+ case IGB_MIG_DIRTY_CTRL_QUERY:
+ if (!igb_core_vf_dirty_enabled(s)) {
+ ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_NOT_ENABLED;
+ break;
+ }
+ if (!ms->mig_dirty_buf_addr) {
+ ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_NO_BUFFER;
+ break;
+ }
+ igbvf_mig_dirty_query(s, pgsize);
+ ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
+ break;
+ }
+}
+
static void igbvf_mig_write(void *opaque, hwaddr addr, uint64_t val,
unsigned size)
{
@@ -599,6 +924,31 @@ static void igbvf_mig_write(void *opaque, hwaddr addr,
uint64_t val,
ms->mig_data_buf_addr =
deposit64(ms->mig_data_buf_addr, 32, 32, val);
break;
+ case IGB_MIG_DIRTY_PGSIZE:
+ ms->mig_dirty_pgsize = (uint32_t)val;
+ break;
+ case IGB_MIG_DIRTY_CTRL:
+ igbvf_mig_dirty_ctrl(s, (uint32_t)val);
+ break;
+ case IGB_MIG_DIRTY_RANGE_IOVA_LO:
+ ms->mig_dirty_range_iova =
+ deposit64(ms->mig_dirty_range_iova, 0, 32, val);
+ break;
+ case IGB_MIG_DIRTY_RANGE_IOVA_HI:
+ ms->mig_dirty_range_iova =
+ deposit64(ms->mig_dirty_range_iova, 32, 32, val);
+ break;
+ case IGB_MIG_DIRTY_RANGE_SIZE:
+ ms->mig_dirty_range_size = (uint32_t)val;
DIRTY_RANGE_SIZE is only 32 bits. The companion driver likewise passes a
u64 size to a u32 MMIO helper at the range-programming call. Ranges of
4 GiB or larger silently truncate.
Regards,
Akihiko Odaki
+ break;
+ case IGB_MIG_DIRTY_BUF_ADDR_LO:
+ ms->mig_dirty_buf_addr =
+ deposit64(ms->mig_dirty_buf_addr, 0, 32, val);
+ break;
+ case IGB_MIG_DIRTY_BUF_ADDR_HI:
+ ms->mig_dirty_buf_addr =
+ deposit64(ms->mig_dirty_buf_addr, 32, 32, val);
+ break;
default:
qemu_log_mask(LOG_GUEST_ERROR,
"igbvf: VF%u bad migration BAR write at 0x%"
@@ -643,5 +993,12 @@ void igbvf_mig_state_reset(IgbVfState *s)
ms->mig_data_size = igb_core_vf_max_data_size(s);
ms->mig_data_buf_addr = 0;
memset(ms->mig_data, 0, sizeof(ms->mig_data));
+
+ igb_core_vf_dirty_disable(s);
+ ms->mig_dirty_pgsize = 0;
+ ms->mig_dirty_range_iova = 0;
+ ms->mig_dirty_range_size = 0;
+ ms->mig_dirty_buf_addr = 0;
+ ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
trace_igbvf_mig_reset(s->vfn);
}
diff --git a/hw/net/igbvf.c b/hw/net/igbvf.c
index e9f9fc3369d8..398306d3bfc6 100644
--- a/hw/net/igbvf.c
+++ b/hw/net/igbvf.c
@@ -306,6 +306,10 @@ static void igbvf_pci_uninit(PCIDevice *dev)
{
IgbVfState *s = IGBVF(dev);
+ if (s->mig.migration_cap) {
+ igb_core_vf_dirty_disable(s);
+ }
+
pcie_aer_exit(dev);
pcie_cap_exit(dev);
msix_unuse_all_vectors(dev);
diff --git a/hw/net/trace-events b/hw/net/trace-events
index 0b13a99b3f32..95fdea89d49c 100644
--- a/hw/net/trace-events
+++ b/hw/net/trace-events
@@ -303,6 +303,13 @@ igbvf_mig_set_state_err(uint16_t vfn, uint32_t old_state,
uint32_t new_state) "V
igbvf_mig_save_state(uint16_t vfn, uint32_t size) "VF%u: saved %u bytes of device
state"
igbvf_mig_load_state(uint16_t vfn, uint32_t size) "VF%u: loaded %u bytes of device
state"
igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset"
+igbvf_mig_dirty_enable(uint16_t vfn, uint64_t pgsize, uint64_t nbits) "VF%u: dirty tracking
enabled pgsize=%"PRIu64" nbits=%"PRIu64
+igbvf_mig_dirty_disable(uint16_t vfn) "VF%u: dirty tracking disabled"
+igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages) "VF%u: dirty query
returned %"PRIu64" bytes, %u dirty pages"
+
+# igb_core.c - VF migration diagnostics
+igb_core_dirty_track_dma(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA
addr=0x%"PRIx64" len=%"PRIu64
+igb_core_dirty_track_dma_drop(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA dropped
addr=0x%"PRIx64" len=%"PRIu64" no matching range"
# spapr_llan.c
spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_bufs) "pool=%d
count=%"PRId32" rxbufs=%"PRIu32
