Hi all,
I’m testing the latency time of an RDMA system (2 card and 1 switch).
The connection server/client is established and the code is just as
mirror on both sides.
The RDMA action scheme is below:
1- client sends a buffer;
2- server receives a buffer;
3- server sends a buffer;
4- client receives a buffer.
The scheme is in a for loop, so after step 4 client repeats step 1.
On client side I put a clock: t_start is before loop and t_end is after
loop.
I consider the round trip time RTT as (t_end-t_start)/NTX where NTX is
number of transfer (number of loop). The latency time is half RTT, if
transfer time is negligible, i.e. for small (8 Bytes – 4 Kbytes) buffer
size and high NTX value (> 10000) is true.
The RDMA communication stops casually with error status 5. It means that
the send or receive operation on one side (server or client) casually is
not in time. I summarize post_send and post_receive operation are
quasi-synchronous. I would like that RDMA operation are completely
synchronous. I use the completion queue and the acknowledgement event.
Do you see the same character in other RDMA use? How can I synchronize
all RDMA operation?
Could you please give me some suggestions?
Thank you very much,
Andrea
Example of structure code:
Ibv_recv_wr rrq(mr), *bad_rrq;
assert(!conn->post_recv(&rrq, &bad_rrq, 0));
assert(!conn->get_completion(&compl_queue));
//printf("ok %p %s
", compl_queue, strerror(errno));
assert(1 == compl_queue->poll(1, &wc));
//printf("status %i
", (int)wc.status);
if (wc.status !=0){
printf("RDMA server closes.
");
break;
}
-----------------
Ibv_send_wr srq((void *)rpd->address, rpd->rkey), *bad_srq;
srq.add_sge(mr);
assert(!conn.post_send(&srq, &bad_srq, 0));
assert(!conn.get_completion(&compl_queue));
assert(1 == compl_queue->poll(1, &wc));
//printf("ok %p %s status %i
", compl_queue, strerror(errno), wc.status);
//printf("loop %i
", i);
if (wc.status != 0){
//printf("KO %p %s status %i
", compl_queue, strerror(errno), wc.status);
printf("RDMA data sending problem in loop %i.
", i);
break;
}
-----------
Andrea Gozzelino
INFN - Laboratori Nazionali di Legnaro (LNL)
Viale dell'Universita' 2 -I-35020 - Legnaro (PD)- ITALIA
Office: E-101
Tel: +39 049 8068346
Fax: +39 049 641925
Mail: andrea.gozzel...@lnl.infn.it
Cell: +39 3488245552
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
