On Sat, Apr 25, 2026 at 07:46:45AM +0200, Markus Armbruster wrote: > Peter Xu <[email protected]> writes: > > > On Fri, Apr 24, 2026 at 09:17:21AM +0200, Markus Armbruster wrote: > >> Peter Xu <[email protected]> writes: > >> > >> > Currently, mgmt can only query for remaining RAM, > >> > >> Remind me: how? > > > > It is the same command, as mentioned in [1] below. I'll enrich the commit > > message here to explain. > > > >> > >> > not system-wise > >> > remaining > >> > data. It was not a problem before, because for a very long time RAM was > >> > the only part that matters. > >> > > >> > After VFIO migrations landed upstream, it may not be true anymore > >> > especially considering that there can be GPU devices that contain GBs of > >> > device states. > >> > > >> > Add a new "remaining" field in query-migrate results, reflecting > >> > system-wise remaining data, which will include everything (e.g. VFIO). > >> > >> "system-wise"? Do you mean "system-wide"? Maybe "total"? > > > > Since "total" has been used elsewhere, I'll use "system-wide", hoping > > that's easier to digest. > > Which "total" do you mean? Perhaps MigrationStats member > > # @total: total amount of bytes involved in the migration process > > What does this @total count? RAM only? If yes, the description is > misleading and needs fixing. Separate patch, followup fine.
I'll follow up. > > >> > This information will be useful for mgmt to implement generic way of > >> > stall > >> > detection that covers all system resources. Say, when system remaining > >> > data does not decrease anymore for a relatively long period of time, then > >> > it may mean that there is a challenge of converging, so mgmt can act > >> > based > >> > on how this value changes over time (especially if sampled after each > >> > migration iteration). > >> > > >> > Before this patch, "expected_downtime" almost played this role. For > >> > example, by monitoring "expected_downtime" at the beginning of each > >> > iteration can in most cases also reflect the progress of migration > >> > system-wise. Said that, "expected_downtime" was always calculated based > >> > on > >> > a bandwidth value that can fluctuate a lot if avail-switchover-bandwidth > >> > is > >> > not used. This new "remaining" field will remove that part of uncertainty > >> > for mgmt. > >> > > >> > With the new field, HMP "info migrate" now reports this: > >> > > >> > (qemu) info migrate > >> > Status: active > >> > Time (ms): total=12080, setup=14, exp_down=300 > > "exp_down" isn't nice for humans. I *guess* it's for "expected > downtime". Could use "expected_downtime=300" instead. Not this patch's > problem, of course. Will follow up too. > > >> > Remaining: 1.36 GiB <------------------- newline > >> > >> "Newline" is ASCI character '\n'. I guess you mean "this is the new > >> line". > > > > Yes. I'll remove this "<----..." if it causes any confusion. > > Annotating output like you did feels just fine, only the word you chose > makes it mildly confusing. Perhaps > > Remaining: 1.36 GiB <--- this is the new line > > would be clearer. Sure. > > >> > RAM info: > >> > Throughput (Mbps): 840.50 > >> > Sizes: pagesize=4 KiB, total=4.02 GiB > >> > Transfers: transferred=1.18 GiB, remain=1.36 GiB > >> > Channels: precopy=1.18 GiB, multifd=0 B, postcopy=0 B > >> > Page Types: normal=307923, zero=388148 > >> > Page Rates (pps): transfer=25660 > >> > Others: dirty_syncs=1 > >> > > >> > It should be the same value as RAM's remaining report when VFIO is not > >> > involved, and it should report more than that when VFIO is involved. > >> > >> "RAM's remaining report" is the "remain=1.36 GiB" part, isn't it? > > > > [1] > > > > Correct. > > Thanks. Could be a bit more explicit. Up to you. I'll attach a new version at the end. > > >> > Cc: Markus Armbruster <[email protected]> > >> > Reviewed-by: Juraj Marcin <[email protected]> > >> > Reviewed-by: Dr. David Alan Gilbert <[email protected]> > >> > Signed-off-by: Peter Xu <[email protected]> > >> > --- > >> > qapi/migration.json | 4 ++++ > >> > migration/migration-hmp-cmds.c | 5 +++++ > >> > migration/migration.c | 7 +++++++ > >> > 3 files changed, 16 insertions(+) > >> > > >> > diff --git a/qapi/migration.json b/qapi/migration.json > >> > index e3ad3f0604..a6e24b5685 100644 > >> > --- a/qapi/migration.json > >> > +++ b/qapi/migration.json > >> > @@ -300,6 +300,9 @@ > >> > # average memory load of the virtual CPU indirectly. Note that > >> > # zero means guest doesn't dirty memory. (Since 8.1) > >> > # > >> > +# @remaining: amount of bytes remaining to be migrated system-wise, > >> > +# includes both RAM and all devices (like VFIO). (Since 11.1) > >> > +# > >> > # Features: > >> > # > >> > # @unstable: Members @postcopy-latency, @postcopy-vcpu-latency, > >> > @@ -310,6 +313,7 @@ > >> > ## > >> > { 'struct': 'MigrationInfo', > >> > 'data': {'*status': 'MigrationStatus', '*ram': 'MigrationRAMStats', > >> > + '*remaining': 'uint64', > >> > >> It's a byte count, so let's make it 'size'. > > > > Will do. > > > > Since this will be the last functional change so far on the whole series, > > and the update seems to be pretty under control (say, qapi schema.py > > generates same c code for both "size" and "uint64"), > > Yes, 'size' is almost exactly the same as 'uint64'. If I remember > correctly, the one difference is the use of visit_type_size() instead of > visit_type_uint64(). visit_type_size() recognizes additional syntax > with "human" visitors: qobject keyval, string input, and opts visitor. > > > could I request an ACK > > on this one with a short diff below, instead of reposting the whole series? > > > > The diff attached here (I'll also fix the commit messages on > > e.g. system-wide wordings if I'll not repost): > > > > diff --git a/qapi/migration.json b/qapi/migration.json > > index b7518b29c6..c701ef1cf5 100644 > > --- a/qapi/migration.json > > +++ b/qapi/migration.json > > @@ -300,7 +300,7 @@ > > # average memory load of the virtual CPU indirectly. Note that > > # zero means guest doesn't dirty memory. (Since 8.1) > > # > > -# @remaining: amount of bytes remaining to be migrated system-wise, > > +# @remaining: amount of bytes remaining to be migrated system-wide, > > # includes both RAM and all devices (like VFIO). (Since 11.1) > > # > > # Features: > > @@ -313,7 +313,7 @@ > > ## > > { 'struct': 'MigrationInfo', > > 'data': {'*status': 'MigrationStatus', '*ram': 'MigrationRAMStats', > > - '*remaining': 'uint64', > > + '*remaining': 'size', > > '*vfio': 'VfioStats', > > '*xbzrle-cache': 'XBZRLECacheStats', > > '*total-time': 'int', > > > > ===8<==== > > > > The complete new version of patch is here (I updated quite a few places on > > the commit message): > > > > https://gitlab.com/peterx/qemu/-/commit/86d973360890cecc564a4a5bcf9a01b9efde368a > > > > Thanks, > > I read the commit message. No surprises except > > It should be the same value as RAM's remaining report when VFIO is not > involved, and it should report more than that when VFIO is involved. > > One note is that this field will be an estimate and may not be sampled the > exact same time versus the RAM remaining section. So it may report > slightly different values even if only RAM is involved. The difference > shouldn't matter though to mgmt to make correct decisions. > > The second paragraph is new. The first paragraph says they "should be > the same", the second that they "may [be] slightly different". > Suboptimal. Yes, it is misleading, I overlooked that. :( > > Here's my try: > > It should be approximately the same value ... > > Only approximately, because this field will be ... > > It's just a commit message, though. Up to you. New version: When VFIO is not involved, the value reported in the new field should be approximately the same as reported in the "remaining" field of the RAM section. It is only an approximate value because the system-wide remaining data is a cached value, which gets frequently updated by migration core. OTOH, the RAM's remaining data is accurate. When VFIO is involved, the new value reported should normally be larger, because it will include the size of VFIO remaining data too. > > QAPI schema > Acked-by: Markus Armbruster <[email protected]> Thanks, I will at least wait for 1-2 days if it still needs update. -- Peter Xu
