Re: [Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

[Barberi, Carl E]

I am using QDR IB and I am not using any IB extenders.  This is a 
self-contained network, with no access outside of the lab it is currently in.

Carl

From: Jeremy Filizetti [mailto:jeremy.filize...@gmail.com]
Sent: Monday, February 20, 2012 8:37 PM
To: Kevin Van Maren
Cc: Barberi, Carl E; lustre-discuss@lists.Lustre.org; isaac_hu...@xyratex.com
Subject: Re: [Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

It does seem extreme for data center IB latency but it may not be in the data 
center.  The LNet write should take 2 RTT latencies, and 3 for reads so you 
could double/triple those times plus any overhead.

Carl can you clarify if you are using QDR IB and/or any campus or wide area IB 
extenders?

Jeremy
On Mon, Feb 20, 2012 at 8:14 PM, Kevin Van Maren 
mailto:kvanma...@fusionio.com>> wrote:
While it's possible the default credits (8 as I recall) is not enough for peak 
performance, it seems to me that something else is wrong:
Each 1MB RPC should take ~300uS (based on MPI/IB xfer rates of 3.2+ GB/s), so 
that means there is another 400uS overhead per RPC that is not masked with 8 
concurrent RPCs, in addition to the overhead masked when he increased 
concurrency.  This is crazy, with a 1uS network latency.

Unless the RPCs are are being broken into tiny chunks or something -- does lnet 
do single-page xfers and not use a rendezvous protocol for full-sized RPCs?  It 
definitely seems that something is broken when o2iblnd gets ~1/3 of the MPI BW, 
given that lnd was designed for high-speed xfers.

The max_rpcs_in_flight normally needs tweaking to improve disk concurrency, 
where a single client needs to drive a high queue depth. Still finding it hard 
to believe 8 1MB concurrent RPCs can't handle the network.

Kevin


On Feb 20, 2012, at 5:44 PM, "Jeremy Filizetti" 
mailto:jeremy.filize...@gmail.com>> wrote:
Am I reading your earlier post correctly that you have a single server acting 
as the MDS and OSS?  Have you changed your peer_credits and credits for 
ko2iblnd kernel module on the server and client?  You also mentioned changing 
osc.*.max_dirty_mb, you probably need to adjust osc.*.max_rpcs_in_flight as 
well.  Can you post your rpc stats "lctl get_param osc.*.rpc_stats"?  I would 
guess they are bunching up around 7-8 if your running with the default 
max_rpcs_in_flight=8.

Jeremy

On Mon, Feb 20, 2012 at 4:59 PM, Barberi, Carl E 
mailto:carl.e.barb...@lmco.com>> wrote:
Thank you.   This did help.  With the concurrency set to 16, I was able to get 
a max write speed of 1138 MB/s.  Any ideas on how we can make that faster, 
though?  Ideally, we'd like to get to 1.5 GB/s.

Carl

From: Liang Zhen [mailto:li...@whamcloud.com<mailto:li...@whamcloud.com>]
Sent: Thursday, February 16, 2012 1:45 AM
To: Barberi, Carl E
Cc: 'lustre-discuss@lists.Lustre.org<mailto:lustre-discuss@lists.Lustre.org>'
Subject: EXTERNAL: Re: [Lustre-discuss] LNET Performance Issue

Hi, I assume you are using "size=1M" for brw test right? performance could 
increase if you set "concurrency" while adding brw test, i.e: --concurrency=16

Liang

On Feb 16, 2012, at 3:30 AM, Barberi, Carl E wrote:

We are having issues with LNET performance over Infiniband.  We have a 
configuration with a single MDT and six (6) OSTs.  The Lustre client I am using 
to test is configured to use 6 stripes (lfs setstripe -c  6 /mnt/lustre).  When 
I perform a test using the following command:

dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000

I typically get a write rate of about 815 MB/s, and we never exceed 848 MB/s.  
When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but when 
I run a series of lnet-selftests, the read and write rates range from 850MB/s - 
875MB/s max.  I have performed the following optimizations to increase the data 
rate:

On the Client:
lctl set_param osc.*.checksums=0
lctl set_param osc.*.max_dirty_mb=256

On the OSTs
lctl set_param obdfilter.*.writethrough_cache_enable=0
lctl set_param obdfilter.*.read_cache_enable=0

echo 4096 > /sys/block//queue/nr_requests

I have also loaded the ib_sdp module, which also brought an increase in speed.  
However, we need to be able to record at no less than 1GB/s, which we cannot 
achieve right now.  Any thoughts on how I can optimize LNET, which clearly 
seems to be the bottleneck?

Thank you for any help you can provide,
Carl Barberi
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org<mailto:Lustre-discuss@lists.lustre.org>
http://lists.lustre.org/mailman/listinfo/lustre-discuss


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org<mailto:Lustre-discuss@lists.lustre.org>
http://lists.lustre.org/mailman/listinfo/lustre-discuss

___
Lustre-discuss mailing list
Lustre-discuss@li

Re: [Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

[Barberi, Carl E]

Thank you.   This did help.  With the concurrency set to 16, I was able to get 
a max write speed of 1138 MB/s.  Any ideas on how we can make that faster, 
though?  Ideally, we'd like to get to 1.5 GB/s.

Carl

From: Liang Zhen [mailto:li...@whamcloud.com]
Sent: Thursday, February 16, 2012 1:45 AM
To: Barberi, Carl E
Cc: 'lustre-discuss@lists.Lustre.org'
Subject: EXTERNAL: Re: [Lustre-discuss] LNET Performance Issue

Hi, I assume you are using "size=1M" for brw test right? performance could 
increase if you set "concurrency" while adding brw test, i.e: --concurrency=16

Liang

On Feb 16, 2012, at 3:30 AM, Barberi, Carl E wrote:


We are having issues with LNET performance over Infiniband.  We have a 
configuration with a single MDT and six (6) OSTs.  The Lustre client I am using 
to test is configured to use 6 stripes (lfs setstripe -c  6 /mnt/lustre).  When 
I perform a test using the following command:

dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000

I typically get a write rate of about 815 MB/s, and we never exceed 848 MB/s.  
When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but when 
I run a series of lnet-selftests, the read and write rates range from 850MB/s - 
875MB/s max.  I have performed the following optimizations to increase the data 
rate:

On the Client:
lctl set_param osc.*.checksums=0
lctl set_param osc.*.max_dirty_mb=256

On the OSTs
lctl set_param obdfilter.*.writethrough_cache_enable=0
lctl set_param obdfilter.*.read_cache_enable=0

echo 4096 > /sys/block//queue/nr_requests

I have also loaded the ib_sdp module, which also brought an increase in speed.  
However, we need to be able to record at no less than 1GB/s, which we cannot 
achieve right now.  Any thoughts on how I can optimize LNET, which clearly 
seems to be the bottleneck?

Thank you for any help you can provide,
Carl Barberi
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org<mailto:Lustre-discuss@lists.lustre.org>
http://lists.lustre.org/mailman/listinfo/lustre-discuss

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] LNET Performance Issue

[Barberi, Carl E]

Sure.

lctl list_nids output for test client:

172.25.103.23@o2ib<mailto:172.25.103.23@o2ib>
172.25.104.23@o2ib1<mailto:172.25.104.23@o2ib1>

lctl list_nids output for MDS:

172.25.103.1@o2ib<mailto:172.25.103.1@o2ib>
172.25.103.1@o2ib1<mailto:172.25.103.1@o2ib1>

lctl list_nids output for each OSS:
172.25.103.x@o2ib<mailto:172.25.103.x@o2ib>
172.25.104.x@o2ib1<mailto:172.25.104.x@o2ib1>

modprobe.d/lustre.conf output:
MDS: options lnet networks=o2ib0(ib0),o2ib1(ib0)
OSS: options lnet networks=o2ib0(ib0),o2ib1(ib2)
Client: options lnet networks=o2ib0(ib0),o2ib1(ib1)

Carl


From: Kevin Van Maren [mailto:kvanma...@fusionio.com]
Sent: Wednesday, February 15, 2012 3:25 PM
To: Barberi, Carl E
Cc: lustre-discuss@lists.Lustre.org
Subject: EXTERNAL: Re: [Lustre-discuss] LNET Performance Issue

Perhaps someone else here has a thought, but it does not make sense to me that 
loading SDP (which accelerates TCP traffic by by-passing the TCP stack) makes 
lnet faster if you are using @o2ib, and _not_ @tcp0 for your NIDs.

Any chance you've configured both TCP and O2IB nids on the machine, and it is 
somehow picking the TCP nids to use?

Can you confirm the "lctl list_nids" output, and your lustre/lnet sections of 
your modprobe.conf?

Kevin


On Feb 15, 2012, at 12:30 PM, Barberi, Carl E wrote:


We are having issues with LNET performance over Infiniband.  We have a 
configuration with a single MDT and six (6) OSTs.  The Lustre client I am using 
to test is configured to use 6 stripes (lfs setstripe -c  6 /mnt/lustre).  When 
I perform a test using the following command:

dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000

I typically get a write rate of about 815 MB/s, and we never exceed 848 MB/s.  
When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but when 
I run a series of lnet-selftests, the read and write rates range from 850MB/s - 
875MB/s max.  I have performed the following optimizations to increase the data 
rate:

On the Client:
lctl set_param osc.*.checksums=0
lctl set_param osc.*.max_dirty_mb=256

On the OSTs
lctl set_param obdfilter.*.writethrough_cache_enable=0
lctl set_param obdfilter.*.read_cache_enable=0

echo 4096 > /sys/block//queue/nr_requests

I have also loaded the ib_sdp module, which also brought an increase in speed.  
However, we need to be able to record at no less than 1GB/s, which we cannot 
achieve right now.  Any thoughts on how I can optimize LNET, which clearly 
seems to be the bottleneck?

Thank you for any help you can provide,
Carl Barberi




Confidentiality Notice: This e-mail message, its contents and any attachments 
to it are confidential to the intended recipient, and may contain information 
that is privileged and/or exempt from disclosure under applicable law. If you 
are not the intended recipient, please immediately notify the sender and 
destroy the original e-mail message and any attachments (and any copies that 
may have been made) from your system or otherwise. Any unauthorized use, 
copying, disclosure or distribution of this information is strictly prohibited. 
Email addresses that end with a "-c" identify the sender as a Fusion-io 
contractor.
  
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] LNET Performance Issue

[Barberi, Carl E]

We are having issues with LNET performance over Infiniband.  We have a 
configuration with a single MDT and six (6) OSTs.  The Lustre client I am using 
to test is configured to use 6 stripes (lfs setstripe -c  6 /mnt/lustre).  When 
I perform a test using the following command:

dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000

I typically get a write rate of about 815 MB/s, and we never exceed 848 MB/s.  
When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but when 
I run a series of lnet-selftests, the read and write rates range from 850MB/s - 
875MB/s max.  I have performed the following optimizations to increase the data 
rate:

On the Client:
lctl set_param osc.*.checksums=0
lctl set_param osc.*.max_dirty_mb=256

On the OSTs
lctl set_param obdfilter.*.writethrough_cache_enable=0
lctl set_param obdfilter.*.read_cache_enable=0

echo 4096 > /sys/block//queue/nr_requests

I have also loaded the ib_sdp module, which also brought an increase in speed.  
However, we need to be able to record at no less than 1GB/s, which we cannot 
achieve right now.  Any thoughts on how I can optimize LNET, which clearly 
seems to be the bottleneck?

Thank you for any help you can provide,
Carl Barberi
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] EXTERNAL: Re: Unable to write to the Lustre File System as any user except root

[Barberi, Carl E]

We found the issue.  The ability to write to the Lustre FS was dependent on the 
synchronization of the user IDs between the MDS and the clients, which I assume 
LDAP/NIS would take care of (as you guys mentioned).  Thank you for your help.  
I appreciate it.

Carl

From: Cliff White [mailto:cli...@whamcloud.com]
Sent: Wednesday, October 12, 2011 11:48 AM
To: Barberi, Carl E
Cc: Kilian CAVALOTTI; lustre-discuss@lists.lustre.org
Subject: Re: [Lustre-discuss] EXTERNAL: Re: Unable to write to the Lustre File 
System as any user except root

If you are not using LDAP, etc, then the user's information must be in the 
MDS's password files.
Users must be known to the MDS.
cliffw

On Wed, Oct 12, 2011 at 6:37 AM, Barberi, Carl E 
mailto:carl.e.barb...@lmco.com>> wrote:
Thank you Kilian.  However, I was just able to perform Lustre operations as 
another user.  So now it seems that only one of my users exhibits the problem I 
saw.  I don't have NIS or LDAP configured on this MDS, but I have another 
Lustre FS setup elsewhere that does not exhibit any of the problems I've seen.  
Any thoughts as to why now only one user cannot access the Lustre FS?

Thanks,
Carl

-Original Message-
From: Kilian CAVALOTTI 
[mailto:kilian.cavalotti.w...@gmail.com<mailto:kilian.cavalotti.w...@gmail.com>]
Sent: Wednesday, October 12, 2011 2:24 AM
To: Barberi, Carl E
Cc: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
Subject: EXTERNAL: Re: [Lustre-discuss] Unable to write to the Lustre File 
System as any user except root

Hi Carl,

On Tue, Oct 11, 2011 at 9:07 PM, Barberi, Carl E
mailto:carl.e.barb...@lmco.com>> wrote:
> " LustreError: 11-0: an error occurred while communicating with
> 192.168.10.2@o2ib.  The mds_getxattr operation failed with -13."

You likely miss authentication information on your MDS about the user
you're trying to write as.
Just configure NIS, LDAP or whatever you're using on your MDS, and you
should be good to go.

Cheers,
--
Kilian
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org<mailto:Lustre-discuss@lists.lustre.org>
http://lists.lustre.org/mailman/listinfo/lustre-discuss



--
cliffw
Support Guy
WhamCloud, Inc.
www.whamcloud.com<http://www.whamcloud.com>


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] EXTERNAL: Re: Unable to write to the Lustre File System as any user except root

[Barberi, Carl E]

Thank you Kilian.  However, I was just able to perform Lustre operations as 
another user.  So now it seems that only one of my users exhibits the problem I 
saw.  I don't have NIS or LDAP configured on this MDS, but I have another 
Lustre FS setup elsewhere that does not exhibit any of the problems I've seen.  
Any thoughts as to why now only one user cannot access the Lustre FS?

Thanks,
Carl

-Original Message-
From: Kilian CAVALOTTI [mailto:kilian.cavalotti.w...@gmail.com] 
Sent: Wednesday, October 12, 2011 2:24 AM
To: Barberi, Carl E
Cc: lustre-discuss@lists.lustre.org
Subject: EXTERNAL: Re: [Lustre-discuss] Unable to write to the Lustre File 
System as any user except root

Hi Carl,

On Tue, Oct 11, 2011 at 9:07 PM, Barberi, Carl E
 wrote:
> “ LustreError: 11-0: an error occurred while communicating with
> 192.168.10.2@o2ib.  The mds_getxattr operation failed with -13.”

You likely miss authentication information on your MDS about the user
you're trying to write as.
Just configure NIS, LDAP or whatever you're using on your MDS, and you
should be good to go.

Cheers,
-- 
Kilian
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] Unable to write to the Lustre File System as any user except root

[Barberi, Carl E]

I have a File Server set up running Lustre 2.1.  I have one MDS and 3 OSS with 
Infiniband interconnect.  The MDS and 3 OSS mount fine and I am able to mount a 
client.  If I perform any basic operation (such as "ll", "cp", "mv", etc.) as 
root, it appears to work.  However, when I run as any other user, I get the 
following error in my syslog:

" LustreError: 11-0: an error occurred while communicating with 
192.168.10.2@o2ib.  The mds_getxattr operation failed 
with -13."

It tells me that I don't have permission to read from or write to the file 
system.  Why am I seeing this?

Thank you,
Carl
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss