My understanding is that within each iteration round, the HVAs carried by different WRs never overlap, so serializing WRs within the same chunk isn't necessary. This patchset builds on that assumption — please correct me if I've missed or misunderstood anything.
This series improves RDMA migration throughput during the iteration phase and iterable stop-copy phase by allowing multiple writes to the same registration chunk. RDMA tracks RAM in registration chunks. During the final iterable transfer, workloads always leave many scattered 4 KiB dirty pages, with several pages falling into the same chunk. The current code waits for an earlier write to a chunk to complete before posting the next one. These serial completion waits can leave send queue capacity unused and limit RDMA bandwidth, increasing downtime. The first patch replaces the per-chunk transit bitmap with reference counts. The second patch removes the per-chunk wait. rdma_registration_start/stop functions called in each round synchronously drain all in-flight WRs with cq, which can serve as a barrier between rounds. The test configuration is: RDMA chunk size: 32 MiB Guest: 32 vCPUs, 128 GiB RAM Test runs: 10 Max HCA bandwidth: 100 Gbs Workload: idle Average results: Before After pin-all is true Downtime 396.7 ms 345.7 ms ~12.9% improve Final iterable bandwidth 9952.0 MiB/s 11639.10 MiB/s ~17.0% improve Average results: Before After pin-all is false Downtime 248.7 ms 191.1 ms ~23.2% improve Final iterable bandwidth 5605.1 MiB/s 9826.0 MiB/s ~75.3% improve Note: Final iterable bandwidth actually means the bandwidth during qemu_savevm_state_complete_precopy_iterable(). non-iterable data doesn't use RDMA Write. Yanfei Xu (2): migration/rdma: Track in-flight writes with refcounts migration/rdma: Allow multiple in-flight writes per chunk migration/rdma.c | 65 +++++++++++++++++++++++------------------- migration/trace-events | 5 ++-- 2 files changed, 37 insertions(+), 33 deletions(-) -- 2.20.1
