On 9/11/2025 11:09 AM, Markus Armbruster wrote:
Steve Sistare <[email protected]> writes:
Add the cpr-exec migration mode. Usage:
qemu-system-$arch -machine aux-ram-share=on ...
migrate_set_parameter mode cpr-exec
migrate_set_parameter cpr-exec-command \
<arg1> <arg2> ... -incoming <uri-1> \
migrate -d <uri-1>
The migrate command stops the VM, saves state to uri-1,
directly exec's a new version of QEMU on the same host,
replacing the original process while retaining its PID, and
loads state from uri-1. Guest RAM is preserved in place,
albeit with new virtual addresses.
The new QEMU process is started by exec'ing the command
specified by the @cpr-exec-command parameter. The first word of
the command is the binary, and the remaining words are its
arguments. The command may be a direct invocation of new QEMU,
or may be a non-QEMU command that exec's the new QEMU binary.
This mode creates a second migration channel that is not visible
to the user. At the start of migration, old QEMU saves CPR state
to the second channel, and at the end of migration, it tells the
main loop to call cpr_exec. New QEMU loads CPR state early, before
objects are created.
Because old QEMU terminates when new QEMU starts, one cannot
stream data between the two, so uri-1 must be a type,
such as a file, that accepts all data before old QEMU exits.
Otherwise, old QEMU may quietly block writing to the channel.
Memory-backend objects must have the share=on attribute, but
memory-backend-epc is not supported. The VM must be started with
the '-machine aux-ram-share=on' option, which allows anonymous
memory to be transferred in place to the new process. The memfds
are kept open across exec by clearing the close-on-exec flag, their
values are saved in CPR state, and they are mmap'd in new QEMU.
Signed-off-by: Steve Sistare <[email protected]>
---
qapi/migration.json | 25 +++++++++++++++-
include/migration/cpr.h | 1 +
migration/cpr-exec.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++
migration/cpr.c | 26 ++++++++++++++++-
migration/migration.c | 10 ++++++-
migration/ram.c | 1 +
migration/vmstate-types.c | 8 +++++
migration/trace-events | 1 +
8 files changed, 143 insertions(+), 3 deletions(-)
diff --git a/qapi/migration.json b/qapi/migration.json
index ea410fd..cbc90e8 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -694,9 +694,32 @@
# until you issue the `migrate-incoming` command.
#
# (since 10.0)
+#
+# @cpr-exec: The migrate command stops the VM, saves state to the
+# migration channel, directly exec's a new version of QEMU on the
+# same host, replacing the original process while retaining its
+# PID, and loads state from the channel. Guest RAM is preserved
+# in place. Devices and their pinned pages are also preserved for
+# VFIO and IOMMUFD.
+#
+# Old QEMU starts new QEMU by exec'ing the command specified by
+# the @cpr-exec-command parameter. The command may be a direct
+# invocation of new QEMU, or may be a non-QEMU command that exec's
+# the new QEMU binary.
Not sure we need the last sentence.
If we keep it, maybe say "a wrapper script" instead of "a non-QEMU
command".
I prefer to keep it because the point is not obvious, and I had some
discussions about it in previous versions of the series. I will
rewrite as:
or may be a wrapper that exec's the new QEMU binary.
+#
+# Because old QEMU terminates when new QEMU starts, one cannot
+# stream data between the two, so the channel must be a type,
+# such as a file, that accepts all data before old QEMU exits.
+# Otherwise, old QEMU may quietly block writing to the channel.
+#
+# Memory-backend objects must have the share=on attribute, but
+# memory-backend-epc is not supported. The VM must be started
+# with the '-machine aux-ram-share=on' option.
I assume violations of this constraint fail cleanly.
Yes. Migration blockers are added, and print a clear message.
+#
+# (since 10.2)
##
{ 'enum': 'MigMode',
- 'data': [ 'normal', 'cpr-reboot', 'cpr-transfer' ] }
+ 'data': [ 'normal', 'cpr-reboot', 'cpr-transfer', 'cpr-exec' ] }
##
# @ZeroPageDetection:
Acked-by: Markus Armbruster <[email protected]>
Thanks! - steve
