On 2026/8/21 03:59, Peter Xu wrote:
On Thu, Aug 20, 2026 at 08:58:31PM +0800, Yanfei Xu wrote:
My understanding is that within each iteration round, the HVAs carried by
different WRs never overlap, so serializing WRs within the same chunk isn't
necessary. This patchset builds on that assumption — please correct me if
I've missed or misunderstood anything.
This series improves RDMA migration throughput during the iteration phase
and iterable stop-copy phase by allowing multiple writes to the same
registration chunk.
RDMA tracks RAM in registration chunks. During the final iterable
transfer, workloads always leave many scattered 4 KiB dirty pages, with
several pages falling into the same chunk. The current code waits for
an earlier write to a chunk to complete before posting the next one.
These serial completion waits can leave send queue capacity unused and
limit RDMA bandwidth, increasing downtime.
The first patch replaces the per-chunk transit bitmap with reference
counts. The second patch removes the per-chunk wait.
rdma_registration_start/stop functions called in each round synchronously
drain all in-flight WRs with cq, which can serve as a barrier between
rounds.
The test configuration is:
RDMA chunk size: 32 MiB
Guest: 32 vCPUs, 128 GiB RAM
Test runs: 10
Max HCA bandwidth: 100 Gbs
Workload: idle
Average results: Before After
pin-all is true
Downtime 396.7 ms 345.7 ms ~12.9% improve
Final iterable bandwidth 9952.0 MiB/s 11639.10 MiB/s ~17.0% improve
Average results: Before After
pin-all is false
Downtime 248.7 ms 191.1 ms ~23.2% improve
Final iterable bandwidth 5605.1 MiB/s 9826.0 MiB/s ~75.3% improve
Note: Final iterable bandwidth actually means the bandwidth during
qemu_savevm_state_complete_precopy_iterable(). non-iterable data doesn't
use RDMA Write.
So I left some comment in patch 2, but maybe I should just comment directly
here.. please check that first.
If you would agree with what I said there (and you should have noticed
another patch I sent days ago removing UNREGISTER path), I think what we
really need might be:
Completely agree, very good suggestions. Will rework the patchset based on
these points (I might get to it in a few days).
Thanks for your detailed explanation!
Regards,
Yanfei
- Move RDMA draining from ram_save_iterate()/complete() into each time we
finish scanning a round
- Remove transit_bitmap completely (if you see after applying your this
series, the UNREGISTER should be the only user..), because with the
correct draining IIUC we don't need this anymore
Thanks,
