Avihai Horon <[email protected]> writes: > On 6/3/2026 10:17 AM, Markus Armbruster wrote: >> External email: Use caution opening links or attachments >> >> >> Avihai Horon <[email protected]> writes: >> >>> Switchover-ack is a mechanism to synchronize between source and >>> destination QEMU during migration to prevent the source from switching >>> over prematurely. >>> >>> VFIO uses switchover-ack to ensure switchover happens only after >>> destination side has loaded the precopy initial bytes. This is important >>> for VFIO, as otherwise downtime could be impacted and be higher. >>> >>> In its current state, switchover-ack is a one-time mechanism, meaning >>> that switchover is acked only once and past that another ACK cannot be >>> requested again. This was sufficient until now, as VFIO precopy initial >>> bytes was defined to be monotonically decreasing. Thus, when precopy >>> initial bytes reached zero for all VFIO devices, a single ACK would be >>> sent and its validity would hold. >>> >>> However, now the new VFIO_PRECOPY_INFO_REINIT feature allows precopy >>> initial bytes to be re-initialized during precopy. Specifically, it >>> means that initial bytes can grow after reaching zero, which would >>> invalidate a previously sent switchover ACK. >>> >>> To solve this, make switchover-ack reusable and allow devices to request >>> switchover ACKs when needed via the save_query_pending SaveVMHandler. >>> >>> Since now switchover ACK can be requested for a specific device and in >>> different times, make switchover ACK per-device (instead of a single ACK >>> for all devices) and let source side do the pending ACKs accounting. >>> >>> Keep the legacy switchover-ack mechanism for backward compatibility and >>> turn it on by a compatibility property for older machines. Enable the >>> property until VFIO implements the new switchover-ack. >> >> I figure this is about the transmission of ACKs via the return path. If >> both source and destination understand the new per-device ACK, they use >> it. Else, they use old one. Correct? > > Correct. > >> >> This is not visible to the management application. It may use migration >> capability @switchover-ack to enable just as before. Correct? > > Correct. > >> >> Can you think of a reason why management applications might need to know >> whether a certain qemu-system-FOO supports per-device ACKs? > > Not that I can think of. It should be an implementation detail internal to > QEMU.
Thank you. >>> Signed-off-by: Avihai Horon <[email protected]> >>> --- >>> qapi/migration.json | 16 ++++----- >>> include/migration/client-options.h | 1 + >>> include/migration/register.h | 2 ++ >>> migration/migration.h | 32 ++++++++++++++++-- >>> migration/savevm.h | 6 ++-- >>> hw/core/machine.c | 1 + >>> migration/migration.c | 37 ++++++++++++++------- >>> migration/options.c | 10 ++++++ >>> migration/savevm.c | 53 +++++++++++++++++++++++------- >>> migration/trace-events | 5 +-- >>> 10 files changed, 125 insertions(+), 38 deletions(-) >>> >>> diff --git a/qapi/migration.json b/qapi/migration.json >>> index 27a7970556..550cb77762 100644 >>> --- a/qapi/migration.json >>> +++ b/qapi/migration.json >>> @@ -507,15 +507,13 @@ >>> # and should not affect the correctness of postcopy migration. >>> # (since 7.1) >>> # >>> -# @switchover-ack: If enabled, migration will not stop the source VM >>> -# and complete the migration until an ACK is received from the >>> -# destination that it's OK to do so. Exactly when this ACK is >>> -# sent depends on the migrated devices that use this feature. For >>> -# example, a device can use it to make sure some of its data is >>> -# sent and loaded in the destination before doing switchover. >>> -# This can reduce downtime if devices that support this capability >>> -# are present. 'return-path' capability must be enabled to use >>> -# it. (since 8.1) >>> +# @switchover-ack: If enabled, migration will rely on destination side >>> +# to acknowledge the source's switchover decision. The >>> +# acknowledgement may depend, for example, on some device's data >>> +# being loaded in the destination before doing switchover. This >>> +# can reduce downtime if devices that support this capability are >>> +# present. 'return-path' capability must be enabled to use it. >> >> Please use the opportunity to fix markup: @return-path. >> docs/devel/qapi-code-gen.rst: >> >> Use @foo to reference a member description within the current >> definition. This is an rST extension. It is currently rendered the >> same way as ````foo````, but carries additional meaning. >> >> Suggest "Capability @return-path must be ..." > > Sure, will change it. > >> >> The old text is concrete: "will not stop the source VM ... until". The >> new text is vague "will rely on destination side to acknowledge". What >> does that mean exactly? How does it impact behavior the user / >> management application cares about? > > How about the suggestion below? It's both concrete and doesn't mention the > number of ACKs (as requested by Peter). > > # @switchover-ack: If enabled, migration will not stop the source VM > # and complete the migration until the destination has acknowledged > # that switchover is safe. The acknowledgement may depend... > > Thanks. Better! With these improvements, QAPI schema Acked-by: Markus Armbruster <[email protected]> >> >>> +# (since 8.1) >>> # >>> # @dirty-limit: If enabled, migration will throttle vCPUs as needed to >>> # keep their dirty page rate within @vcpu-dirty-limit. This can >> [...] >>
