Hi Marcel
On 22/08/2021 16:39, Marcel Apfelbaum wrote:
> Hi,
>
> On Sat, Jul 31, 2021 at 5:03 PM Li Zhijian <lizhij...@cn.fujitsu.com> wrote:
>> The responder mr registering with ODP will sent RNR NAK back to
>> the requester in the face of the page fault.
>> ---------
>> ibv_poll_cq wc.status=13 RNR retry counter exceeded!
>> ibv_poll_cq wrid=WRITE RDMA!
>> ---------
>> ibv_advise_mr(3) helps to make pages present before the actual IO is
>> conducted so that the responder does page fault as little as possible.
>>
>> Signed-off-by: Li Zhijian <lizhij...@cn.fujitsu.com>
>> ---
>> migration/rdma.c | 40 ++++++++++++++++++++++++++++++++++++++++
>> migration/trace-events | 1 +
>> 2 files changed, 41 insertions(+)
>>
>> diff --git a/migration/rdma.c b/migration/rdma.c
>> index 8784b5f22a6..a2ad00d665f 100644
>> --- a/migration/rdma.c
>> +++ b/migration/rdma.c
>> @@ -1117,6 +1117,30 @@ static int qemu_rdma_alloc_qp(RDMAContext *rdma)
>> return 0;
>> }
>>
>> +/*
>> + * ibv_advise_mr to avoid RNR NAK error as far as possible.
>> + * The responder mr registering with ODP will sent RNR NAK back to
>> + * the requester in the face of the page fault.
>> + */
>> +static void qemu_rdma_advise_prefetch_write_mr(struct ibv_pd *pd, uint64_t
>> addr,
>> + uint32_t len, uint32_t lkey,
>> + const char *name, bool wr)
>> +{
>> + int ret;
>> + int advice = wr ? IBV_ADVISE_MR_ADVICE_PREFETCH_WRITE :
>> + IBV_ADVISE_MR_ADVICE_PREFETCH;
>> + struct ibv_sge sg_list = {.lkey = lkey, .addr = addr, .length = len};
>> +
>> + ret = ibv_advise_mr(pd, advice,
>> + IB_UVERBS_ADVISE_MR_FLAG_FLUSH, &sg_list, 1);
>> + /* ignore the error */
> Following
> https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/man/ibv_advise_mr.3.md
> it looks like it is a best-effort optimization,
> I don't see any down-sides to it.
> However it seems like it is recommended to use
> IBV_ADVISE_MR_FLAG_FLUSH in order to
> increase the optimization chances.
Good catch, i will update it soon.
Thanks
>
> Anyway
>
> Reviewed-by: Marcel Apfelbaum <marcel.apfelb...@gmail.com>
>
> Thanks,
> Marcel
>
>
