This RFC explores using VMStateDescription to move virtio-net migration
work out of stop-and-copy. The goal is to reduce guest-visible downtime
by loading as much virtio-net state as possible during the setup phase,
then deciding at stop-and-copy whether the destination can finish with a
minimal resync or needs to fall back to the existing full reload path.
This is a continuation of the earlier RFC that prototyped the idea with
SaveVMHandlers [1]. This version reworks the approach around
VMStateDescription.
At a high level, the series does the following:
- adds an `early-mig` property for virtio-net and enables it by default
only for newer machine types, while older machine types keep the
legacy default through compat properties
- adds an early_setup-based VMSD so virtio-net setup work can be moved
before stop-and-copy
- snapshots state that was sent early, detects relevant mid-migration
deltas at stop-and-copy, and falls back to the existing full
virtio-net reload path if any tracked state changed
- if no tracked deltas are found, skips the full reload and only resends
the minimal state that still must be synchronized at stop-and-copy
- extends the same model to vhost-net by starting most vhost state early
and deferring backend binding/finalization until stop-and-copy
This RFC series works as follows:
- state loaded early is treated as provisional
- at stop-and-copy, tracked VirtIODevice/VirtIONet state is compared
against the early snapshot
- if anything relevant changed, migration falls back to the existing
full virtio-net reload path
- only the no-delta case keeps the shorter stop-and-copy path
For an actual patch series, the following should be implemented:
- handle deltas individually instead of forcing a full reload whenever
any tracked state changes
- include support for vhost-vDPA
The `early-mig` property is enabled by default only for machine types >=
10.2. Older machine types keep the legacy default off through compat
properties, so existing machine-version behavior is preserved unless the
user overrides the setting explicitly.
For vhost-net, failures while starting vhost early or during the
stop-and-copy quickstart path are treated as non-fatal to migration. In
those cases the destination continues on the userspace virtio-net
datapath, and the normal post-switchover vhost start path may retry
later. If a delta forces a full reload, any early-started vhost
instance is stopped before restart so notifier/backend state is torn
down safely.
Testing framework:
- Local live migration testing compared:
- baseline: the current VMSD migration flow
- early VMSD: this RFC series
- Setup:
- 30 measured runs after 3 warmup runs per mode (baseline vs early)
- virtio-net with 4 queue pairs
- guest UDP request/response probe sampled at 10 ms intervals
- no configuration deltas injected; only early VMSD path used
- Metrics:
- query-migrate downtime
- maximum observed probe gap as a guest-visible interruption proxy
Testing results:
- vhost=on (n=30):
- query-migrate downtime median: 69.0 ms baseline, 60.5 ms early
(12.3% lower)
- maximum probe gap median: 309.0 ms baseline, 282.5 ms early
(8.6% lower)
- p95: downtime 85 ms baseline vs 86 ms early; probe gap 338 ms in
both modes
- vhost=off (n=30):
- `query-migrate` downtime median: 55.0 ms baseline, 49.0 ms early
(10.9% lower)
- maximum probe gap median: 283.5 ms baseline, 252.5 ms early
(10.9% lower)
- p95: downtime 69 ms baseline vs 81 ms early; probe gap 310.0 ms
baseline vs 301.0 ms early
Median results improved relative to baseline in both the vhost=on and
vhost=off runs. Higher-percentile results were mixed, especially for
query-migrate downtime in the vhost=off case. That may partly reflect
the relatively small test setup and the fact that these are broader
migration-level or end-to-end measurements rather than explicit
virtio-net device downtime.
A non-RFC submission will have more targeted testing with larger
migration state and more explicit device-focused measurement.
Comments on the overall VMSD structure, migration compatibility
implications, and whether this is the right direction for early
virtio-net device migration would be appreciated.
[1]
https://lore.kernel.org/qemu-devel/[email protected]/
Jonah Palmer (14):
machine,virtio-net: add early-mig property
virtio,virtio-net: add initial early VMSD for setup-phase migration
virtio,virtio-net: virtio-delta VMSD - VQ state
virtio-net: detect VirtIODevice status mid-migration change
virtio-net: detect VirtIODevice config buffer mid-migration change
virtio-net: detect VirtIONet MAC addr mid-migration change
virtio-net: detect VirtIONet MAC table mid-migration changes
virtio-net: detect VirtIONet status mid-migration change
virtio-net: detect VirtIONet Rx filter mid-migration changes
virtio-net: detect VirtIONet VLAN filter table changes
virtio-net: detect VirtIONet guest offload & MQ mid-migration changes
virtio-net: detect RSS state mid-migration changes
virtio-net: detect pending Tx work for VQs mid-migration changes
virtio-net,vhost-net: early migration support for vhost-net
hw/core/machine.c | 5 +
hw/net/vhost_net.c | 183 ++++++++++++++
hw/net/virtio-net.c | 423 +++++++++++++++++++++++++++++++++
hw/virtio/virtio.c | 164 ++++++++++++-
include/hw/virtio/virtio-net.h | 50 ++++
include/hw/virtio/virtio.h | 18 ++
include/net/vhost_net.h | 9 +
7 files changed, 851 insertions(+), 1 deletion(-)
--
2.51.0
