RDMA synchronous operations
Hi all,
I’m testing the latency time of an RDMA system (2 card and 1 switch).
The connection server/client is established and the code is just as
mirror on both sides.
The RDMA action scheme is below:
1- client sends a buffer;
2- server receives a buffer;
3- server sends a buffer;
4- client receives a buffer.
The scheme is in a for loop, so after step 4 client repeats step 1.
On client side I put a clock: t_start is before loop and t_end is after
loop.
I consider the round trip time RTT as (t_end-t_start)/NTX where NTX is
number of transfer (number of loop). The latency time is half RTT, if
transfer time is negligible, i.e. for small (8 Bytes – 4 Kbytes) buffer
size and high NTX value ( 1) is true.
The RDMA communication stops casually with error status 5. It means that
the send or receive operation on one side (server or client) casually is
not in time. I summarize post_send and post_receive operation are
quasi-synchronous. I would like that RDMA operation are completely
synchronous. I use the completion queue and the acknowledgement event.
Do you see the same character in other RDMA use? How can I synchronize
all RDMA operation?
Could you please give me some suggestions?
Thank you very much,
Andrea
Example of structure code:
Ibv_recv_wr rrq(mr), *bad_rrq;
assert(!conn-post_recv(rrq, bad_rrq, 0));
assert(!conn-get_completion(compl_queue));
//printf(ok %p %s
, compl_queue, strerror(errno));
assert(1 == compl_queue-poll(1, wc));
//printf(status %i
, (int)wc.status);
if (wc.status !=0){
printf(RDMA server closes.
);
break;
}
-
Ibv_send_wr srq((void *)rpd-address, rpd-rkey), *bad_srq;
srq.add_sge(mr);
assert(!conn.post_send(srq, bad_srq, 0));
assert(!conn.get_completion(compl_queue));
assert(1 == compl_queue-poll(1, wc));
//printf(ok %p %s status %i
, compl_queue, strerror(errno), wc.status);
//printf(loop %i
, i);
if (wc.status != 0){
//printf(KO %p %s status %i
, compl_queue, strerror(errno), wc.status);
printf(RDMA data sending problem in loop %i.
, i);
break;
}
---
Andrea Gozzelino
INFN - Laboratori Nazionali di Legnaro (LNL)
Viale dell'Universita' 2 -I-35020 - Legnaro (PD)- ITALIA
Office: E-101
Tel: +39 049 8068346
Fax: +39 049 641925
Mail: andrea.gozzel...@lnl.infn.it
Cell: +39 3488245552
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RDMA synchronous operations
Hi all,
I’m testing the latency time of an RDMA system (2 card and 1 switch).
The connection server/client is established and the code is just as
mirror on both sides.
The RDMA action scheme is below:
1- client sends a buffer;
2- server receives a buffer;
3- server sends a buffer;
4- client receives a buffer.
The scheme is in a for loop, so after step 4 client repeats step 1.
On client side I put a clock: t_start is before loop and t_end is after
loop.
I consider the round trip time RTT as (t_end-t_start)/NTX where NTX is
number of transfer (number of loop). The latency time is half RTT, if
transfer time is negligible, i.e. for small (8 Bytes – 4 Kbytes) buffer
size and high NTX value ( 1) is true.
The RDMA communication stops casually with error status 5. It means that
the send or receive operation on one side (server or client) casually is
not in time. I summarize post_send and post_receive operation are
quasi-synchronous. I would like that RDMA operation are completely
synchronous. I use the completion queue and the acknowledgement event.
Do you see the same character in other RDMA use? How can I synchronize
all RDMA operation?
Could you please give me some suggestions?
Thank you very much,
Andrea
Example of structure code:
Ibv_recv_wr rrq(mr), *bad_rrq;
assert(!conn-post_recv(rrq, bad_rrq, 0));
assert(!conn-get_completion(compl_queue));
//printf(ok %p %s
, compl_queue, strerror(errno));
assert(1 == compl_queue-poll(1, wc));
//printf(status %i
, (int)wc.status);
if (wc.status !=0){
printf(RDMA server closes.
);
break;
}
-
Ibv_send_wr srq((void *)rpd-address, rpd-rkey), *bad_srq;
srq.add_sge(mr);
assert(!conn.post_send(srq, bad_srq, 0));
assert(!conn.get_completion(compl_queue));
assert(1 == compl_queue-poll(1, wc));
//printf(ok %p %s status %i
, compl_queue, strerror(errno), wc.status);
//printf(loop %i
, i);
if (wc.status != 0){
//printf(KO %p %s status %i
, compl_queue, strerror(errno), wc.status);
printf(RDMA data sending problem in loop %i.
, i);
break;
}
---
Andrea Gozzelino
INFN - Laboratori Nazionali di Legnaro (LNL)
Viale dell'Universita' 2 -I-35020 - Legnaro (PD)- ITALIA
Office: E-101
Tel: +39 049 8068346
Fax: +39 049 641925
Mail: andrea.gozzel...@lnl.infn.it
Cell: +39 3488245552
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RDMA synchronous operations
Hi all,
I’m testing the latency time of an RDMA system (2 card and 1 switch).
The connection server/client is established and the code is just as
mirror on both sides.
The RDMA action scheme is below:
1- client sends a buffer;
2- server receives a buffer;
3- server sends a buffer;
4- client receives a buffer.
The scheme is in a for loop, so after step 4 client repeats step 1.
On client side I put a clock: t_start is before loop and t_end is after
loop.
I consider the round trip time RTT as (t_end-t_start)/NTX where NTX is
number of transfer (number of loop). The latency time is half RTT, if
transfer time is negligible, i.e. for small (8 Bytes – 4 Kbytes) buffer
size and high NTX value ( 1) is true.
The RDMA communication stops casually with error status 5. It means that
the send or receive operation on one side (server or client) casually is
not in time. I summarize post_send and post_receive operation are
quasi-synchronous. I would like that RDMA operation are completely
synchronous. I use the completion queue and the acknowledgement event.
Do you see the same character in other RDMA use? How can I synchronize
all RDMA operation?
Could you please give me some suggestions?
Thank you very much,
Andrea
Example of structure code:
Ibv_recv_wr rrq(mr), *bad_rrq;
assert(!conn-post_recv(rrq, bad_rrq, 0));
assert(!conn-get_completion(compl_queue));
//printf(ok %p %s
, compl_queue, strerror(errno));
assert(1 == compl_queue-poll(1, wc));
//printf(status %i
, (int)wc.status);
if (wc.status !=0){
printf(RDMA server closes.
);
break;
}
-
Ibv_send_wr srq((void *)rpd-address, rpd-rkey), *bad_srq;
srq.add_sge(mr);
assert(!conn.post_send(srq, bad_srq, 0));
assert(!conn.get_completion(compl_queue));
assert(1 == compl_queue-poll(1, wc));
//printf(ok %p %s status %i
, compl_queue, strerror(errno), wc.status);
//printf(loop %i
, i);
if (wc.status != 0){
//printf(KO %p %s status %i
, compl_queue, strerror(errno), wc.status);
printf(RDMA data sending problem in loop %i.
, i);
break;
}
---
Andrea Gozzelino
INFN - Laboratori Nazionali di Legnaro (LNL)
Viale dell'Universita' 2 -I-35020 - Legnaro (PD)- ITALIA
Office: E-101
Tel: +39 049 8068346
Fax: +39 049 641925
Mail: andrea.gozzel...@lnl.infn.it
Cell: +39 3488245552
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RDMA synchronous operations
Hi all,
I’m testing the latency time of an RDMA system (2 card and 1 switch).
The connection server/client is established and the code is just as
mirror on both sides.
The RDMA action scheme is below:
1- client sends a buffer;
2- server receives a buffer;
3- server sends a buffer;
4- client receives a buffer.
The scheme is in a for loop, so after step 4 client repeats step 1.
On client side I put a clock: t_start is before loop and t_end is after
loop.
I consider the round trip time RTT as (t_end-t_start)/NTX where NTX is
number of transfer (number of loop). The latency time is half RTT, if
transfer time is negligible, i.e. for small (8 Bytes – 4 Kbytes) buffer
size and high NTX value ( 1) is true.
The RDMA communication stops casually with error status 5. It means that
the send or receive operation on one side (server or client) casually is
not in time. I summarize post_send and post_receive operation are
quasi-synchronous. I would like that RDMA operation are completely
synchronous. I use the completion queue and the acknowledgement event.
Do you see the same character in other RDMA use? How can I synchronize
all RDMA operation?
Could you please give me some suggestions?
Thank you very much,
Andrea
Example of structure code:
Ibv_recv_wr rrq(mr), *bad_rrq;
assert(!conn-post_recv(rrq, bad_rrq, 0));
assert(!conn-get_completion(compl_queue));
//printf(ok %p %s
, compl_queue, strerror(errno));
assert(1 == compl_queue-poll(1, wc));
//printf(status %i
, (int)wc.status);
if (wc.status !=0){
printf(RDMA server closes.
);
break;
}
-
Ibv_send_wr srq((void *)rpd-address, rpd-rkey), *bad_srq;
srq.add_sge(mr);
assert(!conn.post_send(srq, bad_srq, 0));
assert(!conn.get_completion(compl_queue));
assert(1 == compl_queue-poll(1, wc));
//printf(ok %p %s status %i
, compl_queue, strerror(errno), wc.status);
//printf(loop %i
, i);
if (wc.status != 0){
//printf(KO %p %s status %i
, compl_queue, strerror(errno), wc.status);
printf(RDMA data sending problem in loop %i.
, i);
break;
}
---
Andrea Gozzelino
INFN - Laboratori Nazionali di Legnaro (LNL)
Viale dell'Universita' 2 -I-35020 - Legnaro (PD)- ITALIA
Office: E-101
Tel: +39 049 8068346
Fax: +39 049 641925
Mail: andrea.gozzel...@lnl.infn.it
Cell: +39 3488245552
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RDMA synchronous operations
Hi all,
I’m testing the latency time of an RDMA system (2 card and 1 switch).
The connection server/client is established and the code is just as
mirror on both sides.
The RDMA action scheme is below:
1- client sends a buffer;
2- server receives a buffer;
3- server sends a buffer;
4- client receives a buffer.
The scheme is in a for loop, so after step 4 client repeats step 1.
On client side I put a clock: t_start is before loop and t_end is after
loop.
I consider the round trip time RTT as (t_end-t_start)/NTX where NTX is
number of transfer (number of loop). The latency time is half RTT, if
transfer time is negligible, i.e. for small (8 Bytes – 4 Kbytes) buffer
size and high NTX value ( 1) is true.
The RDMA communication stops casually with error status 5. It means that
the send or receive operation on one side (server or client) casually is
not in time. I summarize post_send and post_receive operation are
quasi-synchronous. I would like that RDMA operation are completely
synchronous. I use the completion queue and the acknowledgement event.
Do you see the same character in other RDMA use? How can I synchronize
all RDMA operation?
Could you please give me some suggestions?
Thank you very much,
Andrea
Example of structure code:
Ibv_recv_wr rrq(mr), *bad_rrq;
assert(!conn-post_recv(rrq, bad_rrq, 0));
assert(!conn-get_completion(compl_queue));
//printf(ok %p %s\n, compl_queue, strerror(errno));
assert(1 == compl_queue-poll(1, wc));
//printf(status %i\n, (int)wc.status);
if (wc.status !=0){
printf(RDMA server closes. \n);
break;
--
Ibv_send_wr srq((void *)rpd-address, rpd-rkey), *bad_srq;
srq.add_sge(mr);
assert(!conn.post_send(srq, bad_srq, 0));
assert(!conn.get_completion(compl_queue));
assert(1 == compl_queue-poll(1, wc));
//printf(ok %p %s status %i\n, compl_queue, strerror(errno),
wc.status);
//printf(loop %i\n, i);
if (wc.status != 0){
//printf(KO %p %s status %i\n, compl_queue,
strerror(errno),
wc.status);
printf(RDMA data sending problem in loop %i.\n, i);
break;
}
-
Andrea Gozzelino
INFN - Laboratori Nazionali di Legnaro (LNL)
Viale dell'Universita' 2 -I-35020 - Legnaro (PD)- ITALIA
Office: E-101
Tel: +39 049 8068346
Fax: +39 049 641925
Mail: andrea.gozzel...@lnl.infn.it
Cell: +39 3488245552
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
