Currently, mgmt can only query for remaining RAM using QMP command
"query-migrate" and monitor the "ram" section. There's no way to report
system-wide remaining data including VFIO devices. It was not a problem
before, because for a very long time RAM was the only part that matters.
After VFIO migrations landed upstream, it may not be enough. There can be
GPU devices that contain GBs of device states. Mgmt may want to know how
much remaining for special devices like VFIO, because all of them will be
accounted as VM data to migrate and will contribute to downtime in the
switchover phase.
Add a new "remaining" field in query-migrate results on the top level,
reflecting system-wide remaining data, which will include everything like
VFIO devices.
This information will be useful for mgmt to implement generic way of stall
detection that covers all system resources. For example, when system-wide
remaining data (especially, if sampled right after each migration
iteration) does not decrease anymore for a relatively long period of time,
then it may imply there is a challenge of converging, mgmt can react based
on how this value changes over time.
Before this patch, "expected_downtime" almost played this role. For
example, by monitoring "expected_downtime" at the beginning of each
iteration can in most cases also reflect the progress of migration
system-wide.
Said that, "expected_downtime" was always calculated based on a bandwidth
value that can fluctuate if avail-switchover-bandwidth is not used. This
new "remaining" field will remove that part of uncertainty for mgmt no
matter if avail-switchover-bandwidth is used by the mgmt.
With the new field, HMP "info migrate" now reports this:
(qemu) info migrate
Status: active
Time (ms): total=12080, setup=14, exp_down=300
Remaining: 1.36 GiB <--- this is the new line
RAM info:
Throughput (Mbps): 840.50
Sizes: pagesize=4 KiB, total=4.02 GiB
Transfers: transferred=1.18 GiB, remain=1.36 GiB
Channels: precopy=1.18 GiB, multifd=0 B, postcopy=0 B
Page Types: normal=307923, zero=388148
Page Rates (pps): transfer=25660
Others: dirty_syncs=1
When VFIO is not involved, the value reported in the new field should be
approximately the same as reported in the "remaining" field of the RAM
section. It is only approximately because the system-wide remaining data
is a cached value, which gets frequently updated by migration core. OTOH,
the RAM's remaining data is accurate.
When VFIO is involved, the new value reported should normally be larger,
because it will include the size of VFIO remaining data too.
Cc: Aseef Imran <[email protected]>
Reviewed-by: Juraj Marcin <[email protected]>
Reviewed-by: Dr. David Alan Gilbert <[email protected]>
Acked-by: Markus Armbruster <[email protected]> # QAPI schema
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Peter Xu <[email protected]>
---
qapi/migration.json | 4 ++++
migration/migration-hmp-cmds.c | 5 +++++
migration/migration.c | 7 +++++++
3 files changed, 16 insertions(+)
diff --git a/qapi/migration.json b/qapi/migration.json
index ed475e4261..9a4817ec73 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -300,6 +300,9 @@
# average memory load of the virtual CPU indirectly. Note that
# zero means guest doesn't dirty memory. (Since 8.1)
#
+# @remaining: amount of bytes remaining to be migrated system-wide,
+# includes both RAM and all devices (like VFIO). (Since 11.1)
+#
# Features:
#
# @unstable: Members @postcopy-latency, @postcopy-vcpu-latency,
@@ -310,6 +313,7 @@
##
{ 'struct': 'MigrationInfo',
'data': {'*status': 'MigrationStatus', '*ram': 'MigrationRAMStats',
+ '*remaining': 'size',
'*vfio': 'VfioStats',
'*xbzrle-cache': 'XBZRLECacheStats',
'*total-time': 'int',
diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c
index 4f6c1dbf89..2703448c52 100644
--- a/migration/migration-hmp-cmds.c
+++ b/migration/migration-hmp-cmds.c
@@ -178,6 +178,11 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict)
}
}
+ if (info->has_remaining) {
+ g_autofree char *remaining = size_to_str(info->remaining);
+ monitor_printf(mon, "Remaining: \t\t%s\n", remaining);
+ }
+
if (info->has_socket_address) {
SocketAddressList *addr;
diff --git a/migration/migration.c b/migration/migration.c
index ab09dcbcf4..ecc69dc4d2 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1076,6 +1076,12 @@ static void populate_time_info(MigrationInfo *info,
MigrationState *s)
}
}
+static void populate_global_info(MigrationInfo *info, MigrationState *s)
+{
+ info->has_remaining = true;
+ info->remaining = qatomic_read(&mig_stats.dirty_bytes_total);
+}
+
static void populate_ram_info(MigrationInfo *info, MigrationState *s)
{
size_t page_size = qemu_target_page_size();
@@ -1177,6 +1183,7 @@ static void fill_source_migration_info(MigrationInfo
*info)
/* TODO add some postcopy stats */
populate_time_info(info, s);
populate_ram_info(info, s);
+ populate_global_info(info, s);
migration_populate_vfio_info(info);
break;
case MIGRATION_STATUS_COLO:
--
2.53.0
