Hi Yanfei, You're right on both points, and I traced this through rdma-core to confirm.
On the QP-allocation failure: in the mlx5 provider, mlx5_calc_send_wqe() computes the WQE size from the requested max_inline_data and returns -EINVAL if it exceeds the device's max SQ descriptor size. I also checked whether we could query the device's max inline capability up front instead of retrying blind — we can't.That's because the limit depends on the whole combination of requested QP caps, not a fixed device constant — so there's nothing to query in advance. On the SQ footprint point: agreed, it's a real per-QP memory cost regardless of whether 512B is ever used. Given all that, the fix has to be create-and-retry: request our size, and on failure retry with a reduced value (or 0) until ibv_create_qp() succeeds. I'll send a patch for that, along with shrinking the initial request from 512 down to something closer to our real payload sizes (max 40B). Regards, Jack On Sat, Sep 5, 2026 at 8:11 AM Yanfei Xu <[email protected]> wrote: > > > On 2026/9/1 23:51, Jack Wang wrote: > > From: Jack Wang <[email protected]> > > > > Control-channel sends (registration requests, ram block replies, etc.) > > are tiny, but the HCA still does a separate memory read to fetch them > > before sending. Ask the QP for a little inline send space at creation > > time, and use it whenever a message is small enough to fit, so the > > HCA can just copy the bytes straight out of the work request instead. > > > > Falls back to the old path if the provider grants less inline room > > than we asked for, or none at all. No wire protocol change. > > > > Signed-off-by: Jack Wang <[email protected]> > > --- > > migration/rdma.c | 22 ++++++++++++++++++++++ > > 1 file changed, 22 insertions(+) > > > > diff --git a/migration/rdma.c b/migration/rdma.c > > index e976739fad3c..08f3b901be4a 100644 > > --- a/migration/rdma.c > > +++ b/migration/rdma.c > > @@ -66,6 +66,14 @@ static inline uint64_t rdma_merge_max(void) > > #define RDMA_CONTROL_MAX_BUFFER (512 * 1024) > > #define RDMA_CONTROL_MAX_COMMANDS_PER_MESSAGE 4096 > > > > +/* > > + * Requested max_inline_data for the QP: enough for a control header > > + * plus the largest fixed-size control payload, so small control > > + * messages can be sent inline instead of via a separate HCA-side > > + * memory read. > > + */ > > +#define RDMA_CONTROL_MAX_INLINE_DATA 512 > > + > > #define RDMA_CONTROL_VERSION_CURRENT 1 > > /* > > * Capabilities for negotiation. > > @@ -329,6 +337,7 @@ typedef struct RDMAContext { > > struct ibv_context *verbs; > > struct rdma_event_channel *channel; > > struct ibv_qp *qp; /* queue pair */ > > + uint32_t max_inline_data; /* max size for inline sends */ > > struct ibv_comp_channel *recv_comp_channel; /* recv completion > > channel */ > > struct ibv_comp_channel *send_comp_channel; /* send completion > > channel */ > > struct ibv_pd *pd; /* protection domain */ > > @@ -936,6 +945,14 @@ static int qemu_rdma_alloc_qp(RDMAContext *rdma) > > attr.cap.max_recv_wr = 3; > > attr.cap.max_send_sge = 1; > > attr.cap.max_recv_sge = 1; > > + /* > > + * Ask for enough inline data to cover a control header plus the > > + * largest fixed-size control payload (RDMARegister/RDMACompress), > > + * so those sends can skip a local memory read on the HCA. The > > + * provider may grant less (or none); qemu_rdma_post_send_control() > > Seems it's actually opposite? From my understanding, QP allocation will > failed if > the required size is greater than provider's max inline cap. > > Then just sharing some my findings after learning about the inline > feature: I > found that requesting 512 bytes of inline data isn't free. Providers > generally > size the SQ for the worst-case WQE, so allowing a 512-byte inline > payload may > significantly increase both the maximum WQE size and the total SQ memory > footprint. > > Regards, > Yanfei > > > + * checks the actual granted size before using IBV_SEND_INLINE. > > + */ > > + attr.cap.max_inline_data = RDMA_CONTROL_MAX_INLINE_DATA; > > attr.send_cq = rdma->send_cq; > > attr.recv_cq = rdma->recv_cq; > > attr.qp_type = IBV_QPT_RC; > > @@ -945,6 +962,7 @@ static int qemu_rdma_alloc_qp(RDMAContext *rdma) > > } > > > > rdma->qp = rdma->cm_id->qp; > > + rdma->max_inline_data = attr.cap.max_inline_data; > > return 0; > > } > > > > @@ -1447,6 +1465,10 @@ static int qemu_rdma_post_send_control(RDMAContext > > *rdma, uint8_t *buf, > > .num_sge = 1, > > }; > > > > + if (sge.length <= rdma->max_inline_data) { > > + send_wr.send_flags |= IBV_SEND_INLINE; > > + } > > + > > trace_rdma_post_send_control(control_desc(head->type)); > > > > /*
