Hi Lustre Community , Could you help me solve my luster perfomance problem. ? I have just set up several Lustre gateways between two infiniband networks A and B.
Node A <--> Getway <--> Node B lustre 2.12.2 Lustre 2.13.3 Lustre 2.12.3 I am using ConnectX 5 Mellanox cards on servers with AMD Epyc processors. I have tested and validated all speeds at infiniband (ib_read_bw and ib_write_bw) and tcp / ip (iperf3) level. >From a lustre point of view (lnet selfttest the bit rates are correct between >the nodes of A and the gateways. On the other hand, the performance of the B nodes and the gateways are very very low: [LNet Rates of servers] [R] Avg: 26 RPC/s Min: 26 RPC/s Max: 26 RPC/s [W] Avg: 36 RPC/s Min: 36 RPC/s Max: 36 RPC/s [LNet Bandwidth of servers] [R] Avg: 8.61 MiB/s Min: 8.61 MiB/s Max: 8.61 MiB/s [W] Avg: 9.59 MiB/s Min: 9.59 MiB/s Max: 9.59 MiB/s [LNet Rates of servers] [R] Avg: 33 RPC/s Min: 33 RPC/s Max: 33 RPC/s [W] Avg: 42 RPC/s Min: 42 RPC/s Max: 42 RPC/s [LNet Bandwidth of servers] [R] Avg: 8.62 MiB/s Min: 8.62 MiB/s Max: 8.62 MiB/s [W] Avg: 8.81 MiB/s Min: 8.81 MiB/s Max: 8.81 MiB/s Do you have any idea on the causes of these ridiculous flow rates? Regards,
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
