I would suggest you a few other tests to help isolate where the issue might be.
1. What is the single thread "DD" write speed? 2. Lnet_selfttest: Please see " Chapter 28. Testing Lustre Network Performance (LNet Self-Test)" in the Lustre manual if this is a new test for you. This will help show how much Lnet bandwith you have from your single client. There are tunable in the lnet later that can affect things. Which QRD HCA are you using? 3. OBDFilter_survey : Please see " 29.3. Testing OST Performance (obdfilter-survey)" in the Lustre manual. This test will help demonstrate what the backed NVMe/ZFS setup can do at the OBD layer in Lustre. Thanks, Keith -----Original Message----- From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Riccardo Veraldi Sent: Thursday, August 17, 2017 10:48 PM To: Dennis Nelson <dnel...@ddn.com>; lustre-discuss@lists.lustre.org Subject: Re: [lustre-discuss] Lustre poor performance this is my lustre.conf [drp-tst-ffb01:~]$ cat /etc/modprobe.d/lustre.conf options lnet networks=o2ib5(ib0),tcp5(enp1s0f0) data transfer is over infiniband ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 65520 inet 172.21.52.83 netmask 255.255.252.0 broadcast 172.21.55.255 On 8/17/17 10:45 PM, Riccardo Veraldi wrote: > On 8/17/17 9:22 PM, Dennis Nelson wrote: >> It appears that you are running iozone on a single client? What kind of >> network is tcp5? Have you looked at the network to make sure it is not the >> bottleneck? >> > yes the data transfer is on ib0 interface and I did a memory to memory > test through InfiniBand QDR resulting in 3.7GB/sec. > tcp is used to connect to the MDS. It is tcp5 to differentiate it from > my other many Lustre clusters. I could have called it tcp but it does > not make any difference performance wise. > I ran the test from one single node yes, I ran the same test also > locally on a zpool identical to the one on the Lustre OSS. > Ihave 4 identical servers each of them with the aame nvme disks: > > server1: OSS - OST1 Lustre/ZFS raidz1 > > server2: OSS - OST2 Lustre/ZFS raidz1 > > server3: local ZFS raidz1 > > server4: Lustre client > > > > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
