Hi all, I’m testing the latency time of an RDMA system (2 card and 1 switch).
The connection server/client is established and the code is just as mirror on both sides. The RDMA action scheme is below: 1- client sends a buffer; 2- server receives a buffer; 3- server sends a buffer; 4- client receives a buffer. The scheme is in a for loop, so after step 4 client repeats step 1. On client side I put a clock: t_start is before loop and t_end is after loop. I consider the round trip time RTT as (t_end-t_start)/NTX where NTX is number of transfer (number of loop). The latency time is half RTT, if transfer time is negligible, i.e. for small (8 Bytes – 4 Kbytes) buffer size and high NTX value (> 10000) is true. The RDMA communication stops casually with error status 5. It means that the send or receive operation on one side (server or client) casually is not in time. I summarize post_send and post_receive operation are quasi-synchronous. I would like that RDMA operation are completely synchronous. I use the completion queue and the acknowledgement event. Do you see the same character in other RDMA use? How can I synchronize all RDMA operation? Could you please give me some suggestions? Thank you very much, Andrea Example of structure code: Ibv_recv_wr rrq(mr), *bad_rrq; assert(!conn->post_recv(&rrq, &bad_rrq, 0)); assert(!conn->get_completion(&compl_queue)); //printf("ok %p %s ", compl_queue, strerror(errno)); assert(1 == compl_queue->poll(1, &wc)); //printf("status %i ", (int)wc.status); if (wc.status !=0){ printf("RDMA server closes. "); break; } ----------------- Ibv_send_wr srq((void *)rpd->address, rpd->rkey), *bad_srq; srq.add_sge(mr); assert(!conn.post_send(&srq, &bad_srq, 0)); assert(!conn.get_completion(&compl_queue)); assert(1 == compl_queue->poll(1, &wc)); //printf("ok %p %s status %i ", compl_queue, strerror(errno), wc.status); //printf("loop %i ", i); if (wc.status != 0){ //printf("KO %p %s status %i ", compl_queue, strerror(errno), wc.status); printf("RDMA data sending problem in loop %i. ", i); break; } ----------- Andrea Gozzelino INFN - Laboratori Nazionali di Legnaro (LNL) Viale dell'Universita' 2 -I-35020 - Legnaro (PD)- ITALIA Office: E-101 Tel: +39 049 8068346 Fax: +39 049 641925 Mail: andrea.gozzel...@lnl.infn.it Cell: +39 3488245552 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html