Re: [Lustre-discuss] Using NFS to mount lustre

[Parinay Kondekar]

Lustre files can be re-exported using NFS .

>From the presentation by Oleg in Lug-2008 (
http://wiki.lustre.org/images/d/da/LUG08-Lustre-NFS.pdf )

Lustre is a normal local fs from NFS perspective Just add an entry to
/etc/export *on lustre client*/NFS server

give it try.

Hope that helps.
parinay


On 21 June 2013 09:27, Teik Hooi Beh  wrote:

> Hi,
>
> I have just recently started to test Lustre and even with a small 2 nodes
> deployment, I find the performance is outstanding on 1Gb networking.
>
> I was trying to search through the list for anything that could help me to
> mount using NFS protocol but could not find anything.
>
> Is that possible or have anyone done that before? I am in particular
> interested if anyone have use Lustre with VMware. I am currently testing on
> 2.1.5.
>
> Thanks in advance.
>
> Regards
> Beh
>
> ___
> Lustre-discuss mailing list
> Lustre-discuss@lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Using NFS to mount lustre

[Teik Hooi Beh]

Thanks for the pointer.

Just a quick question, will the performance suffer since we have to add
another NFS layer to the lustre client?

Regards
Beh


On Fri, Jun 21, 2013 at 4:07 PM, Parinay Kondekar <
parinay_konde...@xyratex.com> wrote:

> Lustre files can be re-exported using NFS .
>
> From the presentation by Oleg in Lug-2008 (
> http://wiki.lustre.org/images/d/da/LUG08-Lustre-NFS.pdf )
>
> Lustre is a normal local fs from NFS perspective Just add an entry to
> /etc/export *on lustre client*/NFS server
>
> give it try.
>
> Hope that helps.
> parinay
>
>
> On 21 June 2013 09:27, Teik Hooi Beh  wrote:
>
>> Hi,
>>
>> I have just recently started to test Lustre and even with a small 2 nodes
>> deployment, I find the performance is outstanding on 1Gb networking.
>>
>> I was trying to search through the list for anything that could help me
>> to mount using NFS protocol but could not find anything.
>>
>> Is that possible or have anyone done that before? I am in particular
>> interested if anyone have use Lustre with VMware. I am currently testing on
>> 2.1.5.
>>
>> Thanks in advance.
>>
>> Regards
>> Beh
>>
>> ___
>> Lustre-discuss mailing list
>> Lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>>
>
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] Poor Direct-IO Performance with Lustre-2.1.5

[Chan Ching Yu, Patrick]

Hi,

I am experiencing poor direct-IO performance using Lustre 2.1.5 (latest stable) 
on CentOS 6.3.

Two OSS servers connect to the same MD3200 (daisy chained by 4 MD1200).
5 disks (from each MD) form a RAID-5 virtual disk as an OST.
8 OSTs are created in the file system.

RAID segment size is 256K, stripe size is 1MB.

4 clients connect to the OSS servers by 10GigEthernet.
Network performace between servers and clients is normal. 1GB/s throughput is 
obtained in netperf and lnet self test.

4 clients are running iozone to write 4G files.

prompt$  RSH=ssh iozone -i 0 -I -M -C -w -r 1m -t 4 -s 4g -+m 
/root/iozone_clients

O_DIRECT feature enabled

Machine = Linux cluster.iseis.cuhk.edu.hk 2.6.32-279.14.1.el6.x86_64 #1 
SMP Tu  Setting no_unlink
Setting no_unlink
Record Size 1024 KB
File size set to 4194304 KB
Network distribution mode enabled.
Command line used: iozone -i 0 -I -M -C -w -r 1m -t 4 -s 4g -+m 
/root/iozone_clients
Output is in Kbytes/sec
Time Resolution = 0.01 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 4 processes
Each process writes a 4194304 Kbyte file in 1024 Kbyte records

Test running:
Children see throughput for  4 initial writers  =  238195.08 KB/sec
Min throughput per process  =   58905.94 KB/sec
Max throughput per process  =   60983.77 KB/sec
Avg throughput per process  =   59548.77 KB/sec
Min xfer= 4051968.00 KB
Child[0] xfer count = 4194304.00 KB, Throughput =   60983.77 KB/sec
Child[1] xfer count = 4066304.00 KB, Throughput =   59111.19 KB/sec
Child[2] xfer count = 4071424.00 KB, Throughput =   59194.18 KB/sec
Child[3] xfer count = 4051968.00 KB, Throughput =   58905.94 KB/sec

Aggregate throughout 238 MB/s is obtained. 

There is only about 30MB/s throughput (238 / 8) for each OST. (seen at Dell 
Storage Maanger Performance Monitor)
I think it is considered poor, as one OST has 4 effective disks within a RAID-5 
volume.

Why the direct io performance is so slow? Thanks in advance.


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] Fwd: Re: Using NFS to mount lustre

[parinay_konde...@xyratex.com]

FYISent from Samsung mobile 

 Original message 
Subject: Re: [Lustre-discuss] Using NFS to mount lustre
From: Michael Watters 
To: parinay_konde...@xyratex.com
CC: 

One thing to note is that you need to make sure that your UIDs match on the client/server.  I had a really hard time getting NFS to work because of mismatching UIDs.
On Fri, Jun 21, 2013 at 4:07 AM, Parinay Kondekar  wrote:
Lustre files can be re-exported using NFS .From the presentation by Oleg in Lug-2008 ( http://wiki.lustre.org/images/d/da/LUG08-Lustre-NFS.pdf )

Lustre is a normal local fs from NFS perspective Just add an entry to /etc/export on lustre client/NFS servergive it try.Hope that helps.
parinay
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Using NFS to mount lustre

[White, Cliff]

From: Teik Hooi Beh mailto:th...@thbeh.com>>
Date: Friday, June 21, 2013 1:29 AM
To: Parinay Kondekar 
mailto:parinay_konde...@xyratex.com>>
Cc: "lustre-discuss@lists.lustre.org" 
mailto:lustre-discuss@lists.lustre.org>>
Subject: Re: [Lustre-discuss] Using NFS to mount lustre

Thanks for the pointer.

Just a quick question, will the performance suffer since we have to add another 
NFS layer to the lustre client?

To be clear, you cannot export Lustre via NFS from Lustre servers. You 
re-export the Lustre filesystem from a Lustre client node.
Essentially, you turn one client into an NFS server.
So, the bottleneck in performance is that all NFS traffic must come from that 
node, and you are limited to the single-node network bandwidth, instead of 
streaming parallel IO from all servers with the native Lustre client.  In a 
small setup such as you describe you will likely not see any performance impact.
Cliffw


Regards
Beh


On Fri, Jun 21, 2013 at 4:07 PM, Parinay Kondekar 
mailto:parinay_konde...@xyratex.com>> wrote:
Lustre files can be re-exported using NFS .

>From the presentation by Oleg in Lug-2008 ( 
>http://wiki.lustre.org/images/d/da/LUG08-Lustre-NFS.pdf )

Lustre is a normal local fs from NFS perspective Just add an entry to 
/etc/export on lustre client/NFS server

give it try.

Hope that helps.
parinay


On 21 June 2013 09:27, Teik Hooi Beh mailto:th...@thbeh.com>> 
wrote:
Hi,

I have just recently started to test Lustre and even with a small 2 nodes 
deployment, I find the performance is outstanding on 1Gb networking.

I was trying to search through the list for anything that could help me to 
mount using NFS protocol but could not find anything.

Is that possible or have anyone done that before? I am in particular interested 
if anyone have use Lustre with VMware. I am currently testing on 2.1.5.

Thanks in advance.

Regards
Beh

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss





___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?

[Andrew Mast]

Hello, I am new to Lustre and wanted to run a small simple small copy test
between 2 virtual machines from MDT/OST server to client's local disk.

I realize small file performance is never fast, but this seems particularly
slow considering the data is all buffered in memory with little to no disk
activity.

Setup Info
Version is 2.4.50
Average file size is small. < 10KB
The amount of data being copied is about 250MB.
The VMs are on separate hosts.

Performance
7 minutes over a gigabit network.
NFS takes only 3 minutes.

Observations
iostat on the OST/MDT is usually 0% during the copy. Assuming all buffered.
Additional network traffic is minimal.
CPU load on the VMs is 15-20% during copy.

RPC stats on the client shows only 1 RPC in flight at a time. max inflight
is set to 64. Is that expected behavior for a copy?

Here is a snapshot of rpc_stats early during the copy:

   read write pages per rpc rpcs % cum % | rpcs % cum % 1: 1653 90 90 | 0 0
0 2: 164 8 98 | 0 0 0 4: 7 0 99 | 0 0 0 8: 3 0 99 | 0 0 0 16: 3 0 99 | 0 0
0 32: 5 0 99 | 0 0 0 64: 0 0 99 | 0 0 0 128: 1 0 100 | 0 0 0 read write
rpcs in flight rpcs % cum % | rpcs % cum % 0: 0 0 0 | 0 0 0 1: 1836 100 100
| 0 0 0 read write offset rpcs % cum % | rpcs % cum % 0: 1836 100 100 | 0 0
0

As I am new, any suggestions for what to look for or improve would be
greatly appreciated.
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?

[Lee, Brett]

"test between 2 virtual machines from MDT/OST server to client's local disk."

Andrew,

I'm confused by the description of your test.  Can you clarify?

--
Brett Lee
Sr. Systems Engineer
Intel High Performance Data Division

From: lustre-discuss-boun...@lists.lustre.org 
[mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Andrew Mast
Sent: Friday, June 21, 2013 3:42 PM
To: lustre-discuss@lists.lustre.org
Subject: [Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?

Hello, I am new to Lustre and wanted to run a small simple small copy test 
between 2 virtual machines from MDT/OST server to client's local disk.

I realize small file performance is never fast, but this seems particularly 
slow considering the data is all buffered in memory with little to no disk 
activity.

Setup Info
Version is 2.4.50
Average file size is small. < 10KB
The amount of data being copied is about 250MB.
The VMs are on separate hosts.

Performance
7 minutes over a gigabit network.
NFS takes only 3 minutes.

Observations
iostat on the OST/MDT is usually 0% during the copy. Assuming all buffered.
Additional network traffic is minimal.
CPU load on the VMs is 15-20% during copy.

RPC stats on the client shows only 1 RPC in flight at a time. max inflight is 
set to 64. Is that expected behavior for a copy?

Here is a snapshot of rpc_stats early during the copy:

   readwrite
pages per rpc rpcs   % cum % |   rpcs   % cum %
1: 1653  90  90   |  0   0   0
2:  164   8  98   |  0   0   0
4:7   0  99   |  0   0   0
8:3   0  99   |  0   0   0
16:  3   0  99   |  0   0   0
32:  5   0  99   |  0   0   0
64:  0   0  99   |  0   0   0
128:1   0 100   |  0   0   0

  read write
rpcs in flightrpcs   % cum % |   rpcs   % cum %
0:0   0   0   |  0   0   0
1: 1836 100 100   |  0   0   0

  read write
offsetrpcs   % cum % |   rpcs   % cum %
0: 1836 100 100   |  0   0   0

As I am new, any suggestions for what to look for or improve would be greatly 
appreciated.
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?

[Drokin, Oleg]

Hello!

On Jun 21, 2013, at 5:42 PM, Andrew Mast wrote:

> Hello, I am new to Lustre and wanted to run a small simple small copy test 
> between 2 virtual machines from MDT/OST server to client's local disk.

> I realize small file performance is never fast, but this seems particularly 
> slow considering the data is all buffered in memory with little to no disk 
> activity.
> 
> RPC stats on the client shows only 1 RPC in flight at a time. max inflight is 
> set to 64. Is that expected behavior for a copy?

Well, it seems you are reading from Lustre. Small files too.
So Lustre reads a single file at a time (I assume you copy with somehing like 
cp - single threadedly), readahead does not come into play because file size
is smaller than 1 RPC.
So before we are done with a single file, we cannot guess there'd be another 
request to the next file. That's why you have only one RPC in flight.

Also Lustre metadata protocol is somewhat more heavy than NFS, which would 
explain why it's slower than NFS.
Situation should improve once you start trying bigger files.

Bye,
Oleg
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?

[Andrew Mast]

Hi Brett,

Sorry, I think my choice in wording is not correct.

One VM is holding the metadata and the objects. I guess that would mean ti
is the OSS and MDS?
Another VM is the client.It has mounted the lusture filesystem and also has
some local disks. The test is to just to use cp to read data to local disk.

Thanks,
Andy



On Fri, Jun 21, 2013 at 3:22 PM, Lee, Brett  wrote:

>  “test between 2 virtual machines from MDT/OST server to client's local
> disk.”
>
> ** **
>
> Andrew,
>
> ** **
>
> I’m confused by the description of your test.  Can you clarify?
>
> ** **
>
> --
>
> Brett Lee
>
> Sr. Systems Engineer
>
> Intel High Performance Data Division
>
> ** **
>
> *From:* lustre-discuss-boun...@lists.lustre.org [mailto:
> lustre-discuss-boun...@lists.lustre.org] *On Behalf Of *Andrew Mast
> *Sent:* Friday, June 21, 2013 3:42 PM
> *To:* lustre-discuss@lists.lustre.org
> *Subject:* [Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?
>
> ** **
>
> Hello, I am new to Lustre and wanted to run a small simple small copy test
> between 2 virtual machines from MDT/OST server to client's local disk.
>
> ** **
>
> I realize small file performance is never fast, but this seems
> particularly slow considering the data is all buffered in memory with
> little to no disk activity.
>
> ** **
>
> Setup Info
>
> Version is 2.4.50
>
> Average file size is small. < 10KB
>
> The amount of data being copied is about 250MB.
>
> The VMs are on separate hosts.
>
> ** **
>
> Performance
>
> 7 minutes over a gigabit network. 
>
> NFS takes only 3 minutes.
>
> ** **
>
> Observations
>
> iostat on the OST/MDT is usually 0% during the copy. Assuming all buffered.
> 
>
> Additional network traffic is minimal. 
>
> CPU load on the VMs is 15-20% during copy.
>
> ** **
>
> RPC stats on the client shows only 1 RPC in flight at a time. max inflight
> is set to 64. Is that expected behavior for a copy?
>
> ** **
>
> Here is a snapshot of rpc_stats early during the copy:
>
> ** **
>
>readwrite
>
> pages per rpc rpcs   % cum % |   rpcs   % cum %
>
> 1: 1653  90  90   |  0   0   0
>
> 2:  164   8  98   |  0   0   0
>
> 4:7   0  99   |  0   0   0
>
> 8:3   0  99   |  0   0   0
>
> 16:  3   0  99   |  0   0   0
>
> 32:  5   0  99   |  0   0   0
>
> 64:  0   0  99   |  0   0   0
>
> 128:1   0 100   |  0   0   0
>
> ** **
>
>   read write**
> **
>
> rpcs in flightrpcs   % cum % |   rpcs   % cum %
>
> 0:0   0   0   |  0   0   0
>
> 1: 1836 100 100   |  0   0   0
>
> ** **
>
>   read write**
> **
>
> offsetrpcs   % cum % |   rpcs   % cum %
>
> 0: 1836 100 100   |  0   0   0
>
> ** **
>
> As I am new, any suggestions for what to look for or improve would be
> greatly appreciated.
>
> ___
> Lustre-discuss mailing list
> Lustre-discuss@lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?

[Andrew Mast]

Oleg,

Very clear, thank you for the explanation, I misunderstood readahead. Yes
the 1gb and 10gb file transfer tests was on par with NFS.

Our use case is typically compiling and find/grep through (30gb) amounts of
source code so it seems we are stuck with small files.

Andy

On Fri, Jun 21, 2013 at 3:42 PM, Drokin, Oleg  wrote:

> Hello!
>
> On Jun 21, 2013, at 5:42 PM, Andrew Mast wrote:
>
> > Hello, I am new to Lustre and wanted to run a small simple small copy
> test between 2 virtual machines from MDT/OST server to client's local disk.
>
> > I realize small file performance is never fast, but this seems
> particularly slow considering the data is all buffered in memory with
> little to no disk activity.
> >
> > RPC stats on the client shows only 1 RPC in flight at a time. max
> inflight is set to 64. Is that expected behavior for a copy?
>
> Well, it seems you are reading from Lustre. Small files too.
> So Lustre reads a single file at a time (I assume you copy with somehing
> like cp - single threadedly), readahead does not come into play because
> file size
> is smaller than 1 RPC.
> So before we are done with a single file, we cannot guess there'd be
> another request to the next file. That's why you have only one RPC in
> flight.
>
> Also Lustre metadata protocol is somewhat more heavy than NFS, which would
> explain why it's slower than NFS.
> Situation should improve once you start trying bigger files.
>
> Bye,
> Oleg
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?

[Drokin, Oleg]

Hello!

On Jun 21, 2013, at 9:07 PM, Andrew Mast wrote:
> Very clear, thank you for the explanation, I misunderstood readahead. Yes the 
> 1gb and 10gb file transfer tests was on par with NFS.
> 
> Our use case is typically compiling and find/grep through (30gb) amounts of 
> source code so it seems we are stuck with small files.

Generally this sort of workload is pretty bad for network filesystems due to 
large amounts of synchronous RPC traffic that you cannot easily predict.
You can get certain speedup by doing several copies in parallel (e.g. one copy 
per top level subtree or whatever) as then you'll at least get concurrent RPCs.

I know some people try to combat this by running a block device on top of 
network filesystem and then running some sort of a local fs (say, ext4)
on top of that block device (loopback based). That allows readahead to work, 
caching to work much better and so on. But this is not without limitations too,
only single node could have this filesystem-file mounted at any single time.

IF you do not have any significant writes to this fileset (if any at all) but a 
lot of consecutive reads/greps…, you might want just store entire workset as a 
tar file, that you will read and unpack locally on a client (should be pretty 
fast) to say a ramfs (need tons of RAM of course) and then do the searches. 
Also not ideal, but at least network filesystem would then be doing what it's 
best suited for - large transfers.

If you can come up with some other way of storing large number of smaller files 
in a single large combined file that you will then access with special tools 
(like, I dunno, fuse-tarfs or whatever - assuming those don't read unneeded 
data, but just skip over it, or something more specific to your case) - this 
might be a winner too.

Bye,
Oleg
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss