On 6/3/2026 10:17 AM, Markus Armbruster wrote:
External email: Use caution opening links or attachments
Avihai Horon <[email protected]> writes:
Switchover-ack is a mechanism to synchronize between source and
destination QEMU during migration to prevent the source from switching
over prematurely.
VFIO uses switchover-ack to ensure switchover happens only after
destination side has loaded the precopy initial bytes. This is important
for VFIO, as otherwise downtime could be impacted and be higher.
In its current state, switchover-ack is a one-time mechanism, meaning
that switchover is acked only once and past that another ACK cannot be
requested again. This was sufficient until now, as VFIO precopy initial
bytes was defined to be monotonically decreasing. Thus, when precopy
initial bytes reached zero for all VFIO devices, a single ACK would be
sent and its validity would hold.
However, now the new VFIO_PRECOPY_INFO_REINIT feature allows precopy
initial bytes to be re-initialized during precopy. Specifically, it
means that initial bytes can grow after reaching zero, which would
invalidate a previously sent switchover ACK.
To solve this, make switchover-ack reusable and allow devices to request
switchover ACKs when needed via the save_query_pending SaveVMHandler.
Since now switchover ACK can be requested for a specific device and in
different times, make switchover ACK per-device (instead of a single ACK
for all devices) and let source side do the pending ACKs accounting.
Keep the legacy switchover-ack mechanism for backward compatibility and
turn it on by a compatibility property for older machines. Enable the
property until VFIO implements the new switchover-ack.
I figure this is about the transmission of ACKs via the return path. If
both source and destination understand the new per-device ACK, they use
it. Else, they use old one. Correct?
Correct.
This is not visible to the management application. It may use migration
capability @switchover-ack to enable just as before. Correct?
Correct.
Can you think of a reason why management applications might need to know
whether a certain qemu-system-FOO supports per-device ACKs?
Not that I can think of. It should be an implementation detail internal
to QEMU.
Signed-off-by: Avihai Horon <[email protected]>
---
qapi/migration.json | 16 ++++-----
include/migration/client-options.h | 1 +
include/migration/register.h | 2 ++
migration/migration.h | 32 ++++++++++++++++--
migration/savevm.h | 6 ++--
hw/core/machine.c | 1 +
migration/migration.c | 37 ++++++++++++++-------
migration/options.c | 10 ++++++
migration/savevm.c | 53 +++++++++++++++++++++++-------
migration/trace-events | 5 +--
10 files changed, 125 insertions(+), 38 deletions(-)
diff --git a/qapi/migration.json b/qapi/migration.json
index 27a7970556..550cb77762 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -507,15 +507,13 @@
# and should not affect the correctness of postcopy migration.
# (since 7.1)
#
-# @switchover-ack: If enabled, migration will not stop the source VM
-# and complete the migration until an ACK is received from the
-# destination that it's OK to do so. Exactly when this ACK is
-# sent depends on the migrated devices that use this feature. For
-# example, a device can use it to make sure some of its data is
-# sent and loaded in the destination before doing switchover.
-# This can reduce downtime if devices that support this capability
-# are present. 'return-path' capability must be enabled to use
-# it. (since 8.1)
+# @switchover-ack: If enabled, migration will rely on destination side
+# to acknowledge the source's switchover decision. The
+# acknowledgement may depend, for example, on some device's data
+# being loaded in the destination before doing switchover. This
+# can reduce downtime if devices that support this capability are
+# present. 'return-path' capability must be enabled to use it.
Please use the opportunity to fix markup: @return-path.
docs/devel/qapi-code-gen.rst:
Use @foo to reference a member description within the current
definition. This is an rST extension. It is currently rendered the
same way as ````foo````, but carries additional meaning.
Suggest "Capability @return-path must be ..."
Sure, will change it.
The old text is concrete: "will not stop the source VM ... until". The
new text is vague "will rely on destination side to acknowledge". What
does that mean exactly? How does it impact behavior the user /
management application cares about?
How about the suggestion below? It's both concrete and doesn't mention
the number of ACKs (as requested by Peter).
# @switchover-ack: If enabled, migration will not stop the source VM
# and complete the migration until the destination has acknowledged
# that switchover is safe. The acknowledgement may depend...
Thanks.
+# (since 8.1)
#
# @dirty-limit: If enabled, migration will throttle vCPUs as needed to
# keep their dirty page rate within @vcpu-dirty-limit. This can
[...]
