From: Jack Wang <[email protected]> Control-channel sends (registration requests, ram block replies, etc.) carry only a few dozen bytes, but qemu_rdma_post_send_control() still always memcpy()s them into a pre-registered scratch buffer and lets the HCA fetch them with a separate local memory read before sending.
This series asks the QP for a little inline send space at creation time (patch 1) and then, once that plumbing exists, drops the mandatory memcpy() by pointing the SGEs directly at the header/payload for any message that fits inline (patch 2). Messages too large to inline keep using the old copy-into-registered-buffer path, so there is no wire protocol change. In testing, this doesn't show an obvious improvement in total migration time or downtime -- the control channel isn't the bottleneck there. Still, it seems like the right thing to do: inline sends are supported by modern HCAs and reduce per-message latency on the control channel by skipping the extra local memory read, so it should help control-plane responsiveness (e.g. registration round trips) even where it doesn't move the overall migration numbers. Jack Wang (2): migration/rdma: send small control messages inline migration/rdma: avoid memcpy for inline control sends migration/rdma.c | 82 +++++++++++++++++++++++++++++++++++++----------- 1 file changed, 63 insertions(+), 19 deletions(-) -- 2.43.0
