On Thu, Aug 20, 2026 at 08:58:31PM +0800, Yanfei Xu wrote: > My understanding is that within each iteration round, the HVAs carried by > different WRs never overlap, so serializing WRs within the same chunk isn't > necessary. This patchset builds on that assumption — please correct me if > I've missed or misunderstood anything. > > This series improves RDMA migration throughput during the iteration phase > and iterable stop-copy phase by allowing multiple writes to the same > registration chunk. > > RDMA tracks RAM in registration chunks. During the final iterable > transfer, workloads always leave many scattered 4 KiB dirty pages, with > several pages falling into the same chunk. The current code waits for > an earlier write to a chunk to complete before posting the next one. > These serial completion waits can leave send queue capacity unused and > limit RDMA bandwidth, increasing downtime. > > The first patch replaces the per-chunk transit bitmap with reference > counts. The second patch removes the per-chunk wait. > > rdma_registration_start/stop functions called in each round synchronously > drain all in-flight WRs with cq, which can serve as a barrier between > rounds. > > The test configuration is: > > RDMA chunk size: 32 MiB > Guest: 32 vCPUs, 128 GiB RAM > Test runs: 10 > Max HCA bandwidth: 100 Gbs > Workload: idle > > Average results: Before After > pin-all is true > Downtime 396.7 ms 345.7 ms ~12.9% improve > Final iterable bandwidth 9952.0 MiB/s 11639.10 MiB/s ~17.0% improve > > > Average results: Before After > pin-all is false > Downtime 248.7 ms 191.1 ms ~23.2% improve > Final iterable bandwidth 5605.1 MiB/s 9826.0 MiB/s ~75.3% improve > > Note: Final iterable bandwidth actually means the bandwidth during > qemu_savevm_state_complete_precopy_iterable(). non-iterable data doesn't > use RDMA Write.
So I left some comment in patch 2, but maybe I should just comment directly here.. please check that first. If you would agree with what I said there (and you should have noticed another patch I sent days ago removing UNREGISTER path), I think what we really need might be: - Move RDMA draining from ram_save_iterate()/complete() into each time we finish scanning a round - Remove transit_bitmap completely (if you see after applying your this series, the UNREGISTER should be the only user..), because with the correct draining IIUC we don't need this anymore Thanks, -- Peter Xu
