Thank you. This did help. With the concurrency set to 16, I was able to get
a max write speed of 1138 MB/s. Any ideas on how we can make that faster,
though? Ideally, we'd like to get to 1.5 GB/s.
Carl
From: Liang Zhen [mailto:li...@whamcloud.com]
Sent: Thursday, February 16, 2012 1:45 AM
Am I reading your earlier post correctly that you have a single server
acting as the MDS and OSS? Have you changed your peer_credits and credits
for ko2iblnd kernel module on the server and client? You also mentioned
changing osc.*.max_dirty_mb, you probably need to adjust
osc.*.max_rpcs_in_fligh
Apparently I didn't read your email very carefully since you were actually
only wondering about LNET performance and not client performance. You can
ignore the max_rpcs_in_flight for lnet. There used to be a
srpc_peer_credits setting in lnet_selftest, not sure if it exists in your
version. Should
While it's possible the default credits (8 as I recall) is not enough for peak
performance, it seems to me that something else is wrong:
Each 1MB RPC should take ~300uS (based on MPI/IB xfer rates of 3.2+ GB/s), so
that means there is another 400uS overhead per RPC that is not masked with 8
conc
It does seem extreme for data center IB latency but it may not be in the
data center. The LNet write should take 2 RTT latencies, and 3 for reads
so you could double/triple those times plus any overhead.
Carl can you clarify if you are using QDR IB and/or any campus or wide area
IB extenders?
Je
-discuss@lists.Lustre.org; isaac_hu...@xyratex.com
Subject: Re: [Lustre-discuss] EXTERNAL: Re: LNET Performance Issue
It does seem extreme for data center IB latency but it may not be in the data
center. The LNet write should take 2 RTT latencies, and 3 for reads so you
could double/triple those
Monday, February 20, 2012 8:37 PM
> To: Kevin Van Maren
> Cc: Barberi, Carl E; lustre-discuss@lists.Lustre.org; isaac_hu...@xyratex.com
> Subject: Re: [Lustre-discuss] EXTERNAL: Re: LNET Performance Issue
>
> It does seem extreme for data center IB latency but it may not be in the
