On 10/1/26 21:46, Peter Xu wrote: > On Fri, Sep 11, 2026 at 04:23:43PM +0200, Denis V. Lunev wrote: >> Restoring a big Windows guest spends most of its destination-side time in >> post_load hooks which do nothing but move memory regions around. Each one >> ends a memory transaction, and a transaction commit re-renders every >> flatview it touches at a cost which grows with the number of regions in >> the machine. A hook which runs once per vCPU therefore pays that render >> once per vCPU, and the machine gets slower to migrate the bigger it is. >> >> The Hyper-V SynIC is the case that hurts: restoring the synthetic >> interrupt controller maps a message page and an event page per vCPU, so a >> 64-vCPU guest forces 128 remaps, each with its own rebuild, while the rest >> of the stream is still being read. > This is partly a known issue, not from Hyper-V, but from virtio mmio > regions.. please see: > > https://wiki.qemu.org/ToDo/LiveMigration#Optimize_memory_updates_for_non-iterative_vmstates > https://lore.kernel.org/r/[email protected] > > I believe we also thought about do MR update per-device, so batching but > smaller scale, easier to make sure no illegal access to a stale flatview. > > So in general, I agree this approach might be the right way to do, which is > to shrink the transaction to be smaller than "batch everything".. as what > Chuang used to do. I still have some pure questions inline. > >> Patch 1 adds post_load_deferrable. A vmsd which sets it has its hook > Nit, IMHO if so it needs to be called "deferred", as "deferrable" implies > the defer is optional. > >> queued during the load and run once the stream has been consumed, in the >> order the hooks would have fired, with the whole drain sharing one memory >> transaction. Patch 2 sets it on the SynIC subsection. >> >> It is opt-in rather than automatic, and the three preconditions are >> spelled out on the field: the hook must not fail, nothing later in the > Actually, I _think_ maybe it can still fail.. IIUC source QEMU only dies if > it receives shut from migrate_send_rp_shut(), which is after the deferred > loads at least with the current change. Worth check.. > >> load may depend on what it does, and it must not read guest memory or >> resolve an address space. > Yes, but I think this is partial of the whole picture: IIUC if this > deferred hook may inject some MRs that may be accessed by other VMSD > loaders, or anything (including hard-coded loading process), I think it's > an issue too. > > So personally I don't like this API very much yet on how it was defined; > it'll be very hard to be used right unless we fully understand what will > happen.. > > I wonder if there's better way to define the API to be clearer. > > Since all the known issues about this is about MR updates: virtio MMIO > regions, hyper-v, pci bar/bridge (mentioned below), I wonder if this can be > something dedicated to MR updates, and maybe it doesn't need to be > "deferred", just grouped together properly into one transaction, which can > happen in the middle too or maybe it doesn't matter much. Then it applies > some form of limitation to what can be split out from normal VMSD flow. The > current API relies on allowing to defer anything, which is fine but very > hard to control, and we may face tricky bugs if users grows but when > they're not used right.. > >> A hook which breaks the first is fatal rather >> than silently reported, because by drain time the source may already have >> been told the migration succeeded. >> >> Deferring is not free in general, which is the other reason it is opt-in. >> Deferring the APIC post_load, whose cost is a synchronous run_on_cpu per >> vCPU rather than a memory remap, moves 3 ms out of the section walk and >> pays about 9 ms of drain for it. Deferral helps a hook which repeats > I'm just curious: why something will take 9ms if deferred, even if it used > to take 3ms? I think I misread something, but I can't tell myself. > >> topology work; it makes a hook which does cross-thread work worse. >> >> Measurements >> ------------ >> >> Destination-side non-iterable load, ie. the sum of vmstate_downtime_load >> over non-iterable sections, on a guest which has actually programmed its >> Hyper-V state. Five interleaved rounds per point on an otherwise idle >> host, twice; medians, with the spread across all ten rounds. >> >> upstream 377 ms (363-400) >> + pci mapping transactions 247 ms (242-255) >> + this series 96 ms (86-97) > Definitely a great improvement. I think we need this, just one way or > another. I may have some other trivial comments later in the patch. Thanks > for working on it.
Right now I have series staying at 32 ms with boost to ordinary QEMU startup, that is why I have said that I'll send follow up. This patch will be included along as flatview rebuild on startup. Anyway, right now I need to understand and eat your notes to send v2. This just intersects with upcoming release of downstream, but this change is in my priority list and I hope to finish v2 early next week. Thank you for you time, Den
