On Jul 09, 2010 07:13 PM, Hefty, Sean sean.he...@intel.com wrote:
Is librdmacm version 1.0.12 stable?
Yes - but you are using APIs that are new to this release. But those
calls are wrappers around existing libibverbs calls.
The flag IBV_SEND_INLINE supports buffer size until 64 bytes (cards'
feature).
I know that RDMA is a protocol connected with 10 Gigabit/s Ethernet.
How can I see this trasfer speed? Can you suggest the path?
I attached the starting latency_RDMA.cpp code, which works with
librdmacm version 1.0.12 with BUF_SIZE = 64 bytes (thanks to
Chien).
I would like to explore transfer speed in range 1 bytes BUF_SIZE
4
Mbytes.
The use of INLINE allows you to post a send message without the buffer
being registered. As soon as you try to post a send with a buffer
larger than what can be supported as inline, the buffer must be
registered, and the memory region should be passed into the send
command. Add something like this to both the client and server:
struct ibv_mr *send_mr;
...
send_mr = rdma_reg_msgs(id, send_msg, BUF_SIZE);
// add matching rdma_dereg_mr(send_mr); at end
and change your rdma_post_send calls to:
ret = rdma_post_send(id, NULL, send_msg, BUF_SIZE, send_mr,
IBV_SEND_INLINE);
- Sean
Hi Sean, Hi Chien
the code now works fine. Thank you very much, Sean.
In attachment you find:
-- code and script client;
-- txt file with data recap;
-- xls file with graph.
Note that in txt file the round trip time is wrongly called latency.
So in the xls file I correct the formula. I measure the trip time.
latency = RTT/2;
speed in txt file is reduced by 2.
In xls file I put also the speed in Gbits/s.
In this week I will recap code, test data and information about RDMA
test on NE020 cards.
Thank you again for contribution.
Regards,
Andrea
Andrea Gozzelino
INFN - Laboratori Nazionali di Legnaro (LNL)
Viale dell'Universita' 2 -I-35020 - Legnaro (PD)- ITALIA
Office: E-101
Tel: +39 049 8068346
Fax: +39 049 641925
Mail: andrea.gozzel...@lnl.infn.it
Cell: +39 3488245552
#! /bin/sh
# Program: Script bash to run REDIGO (RDMA) performance test
# Authors: Damiano Bortolato - Andrea Gozzelino (INFN - LNL)
# Date: July 2010
# Array buffer size
# Bandwidth test -- Latency measurement too
array_BUFFER_SIZE=( 1 4 16 32 64 128 256 512 1024 8192 65536 131072 200704
262144 393216 524288 1048576 16777216 )
# default number of transfer is 10^6
# Constant that sets date formatting for appending to filenames
# Date format is DDMMYYhhmmss or day,month,year,hour,minute,second.
Date=`date +%d%m%y%H%M%S`
for i in ${array_buffer_si...@]}
do
./RDMA_Speed -b $i -n 100 Test_$i.log
mv Test_Speed.txt Test_Speed_$i.txt
done
# Ending operations
mkdir REDIGO_Speed_$Date
mv *.txt ./REDIGO_Speed_$Date
mv *.log ./REDIGO_Speed_$Date
cd REDIGO_Speed_$Date
cat *.txt REDIGO_Data.txt
cat *.log REDIGO_Log.log
cd ..
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100| 1024| 31| 3.30323e+07| 3.1e-05|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100| 1048576| 2014| 5.20643e+08| 0.002014|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100| 128| 27| 4.74074e+06| 2.7e-05|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100|131072| 253| 5.18071e+08| 0.000253|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100|16| 23|695652| 2.3e-05|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100| 1| 22| 45454.5| 2.2e-05|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100|200704| 409| 4.90719e+08| 0.000409|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100| 256| 27| 9.48148e+06| 2.7e-05|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100|262144| 563|4.6562e+08| 0.000563|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100|32| 23|1.3913e+06| 2.3e-05|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100|393216| 701| 5.60936e+08| 0.000701|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100| 4| 23|173913| 2.3e-05|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100| 512| 28| 1.82857e+07| 2.8e-05|
#transfer|buffer_size(bytes)|total_time(s)|speed(bytes/s)|latency(s)|
100|524288| 927| 5.65575e+08| 0.000927|
