thanks guys for all your help. looks like the issue is fundamentally poor performance across 100GbE, where I'm only getting ~50Gb/s using iperf. I believe the MTU is set correctly across all my systems Using connectx-4 in 100GbE mode.
thanks again, -john On 11/28/17 9:03 PM, Colin Faber wrote:
Are peer credits set appropriately across your fabric? On Nov 28, 2017 8:40 PM, "john casu" <[email protected] <mailto:[email protected]>> wrote: Thanks, just about to try that MTU setting. It's a small lustre system... 2 routers, MDS/MGS pair, OSS pair, JBOD pair (112 drives for OST) and yes, routing between EDR & 100GbE -john On 11/28/17 7:28 PM, Raj wrote: John, increasing MTU size on Ethernet side should increase the b/w. I also have a feeling that some lnet routers and/or intermediate switches/routers does not have jumbo frame turned on (some switches needs to be set at 9212 bytes ). How many LNet routers are you using? I believe you are routing between EDR IB and 100GbE. On Tue, Nov 28, 2017 at 7:21 PM John Casu <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote: just built a system w. lnet routers that bridge Infiniband & 100GbE, using Centos built in Infiniband support servers are Infiniband, clients are 100GbE (connectx-4 cards) my direct write performance from clients over Infiniband is around 15GB/s When I introduced the lnet routers, performance dropped to 10GB/s Thought the problem was an MTU of 1500, but when I changed the MTUs to 9000 performance dropped to 3GB/s. When I tuned according to John Fragella's LUG slides, things went even slower (1.5GB/s write) does anyone have any ideas on what I'm doing wrong?? thanks, -john c. _______________________________________________ lustre-discuss mailing list [email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org <http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org> _______________________________________________ lustre-discuss mailing list [email protected] <mailto:[email protected]> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org <http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org>
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
