Re: [Gluster-users] Kernel NFS on GlusterFS

[Ondrej Valousek]

You say that accessing Gluster via NFS is actually faster than native (fuse) 
client?
Still I would like to know why we can’t use kernel NFS server on the data 
bricks. I understand we can’t use it on MDS as it can’t support pNFS.

Ondrej

From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Jim Kinney
Sent: Wednesday, March 07, 2018 11:47 PM
To: gluster-users@gluster.org
Subject: Re: [Gluster-users] Kernel NFS on GlusterFS

Gluster does the sync part better than corosync. It's not an active/passive 
failover system. It more all active. Gluster handles the recovery once all 
nodes are back online.

That requires the client tool chain to understand that a write goes to all 
storage devices not just the active one.

3.10 is a long term support release. Upgrading to 3.12 or 4 is not a 
significant issue once a replacement for NFS-ganesha stabilizes.

Kernel NFS doesn't understand "write to two IP addresses". That's what 
NFS-Ganesha does. The gluster-fuse client works but is slower than most people 
like. I use the fuse process in my setup at work. Will be changing to 
NFS-Ganesha as part of the upgrade to 3.10.

On Wed, 2018-03-07 at 14:50 -0500, Ben Mason wrote:
Hello,

I'm designing a 2-node, HA NAS that must support NFS. I had planned on using 
GlusterFS native NFS until I saw that it is being deprecated. Then, I was going 
to use GlusterFS + NFS-Ganesha until I saw that the Ganesha HA support ended 
after 3.10 and its replacement is still a WIP. So, I landed on GlusterFS + 
kernel NFS + corosync & pacemaker, which seems to work quite well. Are there 
any performance issues or other concerns with using GlusterFS as a replication 
layer and kernel NFS on top of that?

Thanks!

___

Gluster-users mailing list

Gluster-users@gluster.org

http://lists.gluster.org/mailman/listinfo/gluster-users

--

James P. Kinney III



Every time you stop a school, you will have to build a jail. What you

gain at one end you lose at the other. It's like feeding a dog on his

own tail. It won't fatten the dog.

- Speech 11/23/1900 Mark Twain



http://heretothereideas.blogspot.com/

-

The information contained in this e-mail and in any attachments is confidential 
and is designated solely for the attention of the intended recipient(s). If you 
are not an intended recipient, you must not use, disclose, copy, distribute or 
retain this e-mail or any part thereof. If you have received this e-mail in 
error, please notify the sender by return e-mail and delete all copies of this 
e-mail from your computer system(s). Please direct any additional queries to: 
communicati...@s3group.com. Thank You. Silicon and Software Systems Limited (S3 
Group). Registered in Ireland no. 378073. Registered Office: South County 
Business Park, Leopardstown, Dublin 18.___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster for home directories?

[Ondrej Valousek]

Ok,
I agree Gluster fits more in certain scenarios - but you just can't expect the 
same performance you receive from NAS based solution - this is especially true 
when you deal with lots of relatively small files.
Ondrej


-Original Message-
From: Rik Theys [mailto:rik.th...@esat.kuleuven.be] 
Sent: Wednesday, March 07, 2018 7:38 PM
To: Ondrej Valousek <ondrej.valou...@s3group.com>
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] gluster for home directories?

Hi,

On 2018-03-07 16:35, Ondrej Valousek wrote:
> Why do you need to replace your existing solution?
> If you don't need to scale out due to the capacity reasons, the async 
> NFS server will always outperform GlusterFS

The current solution is 8 years old and is reaching its end of life.

The reason we are also looking into gluster is that we like that it uses 
standard components and that we can prevent forklift upgrades every
5,6,7 years by replacing a few bricks each year. Next to providing storage for 
home directories we would like to also use the hosts to run VM's in a 
hyperconverged setup (with their storage as an additional gluster volume on 
those bricks).

Regards,

Rik


--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT) Kasteelpark Arenberg 10 bus 2440  - 
B-3001 Leuven-Heverlee
+32(0)16/32.11.07

<>
-

The information contained in this e-mail and in any attachments is confidential 
and is designated solely for the attention of the intended recipient(s). If you 
are not an intended recipient, you must not use, disclose, copy, distribute or 
retain this e-mail or any part thereof. If you have received this e-mail in 
error, please notify the sender by return e-mail and delete all copies of this 
e-mail from your computer system(s). Please direct any additional queries to: 
communicati...@s3group.com. Thank You. Silicon and Software Systems Limited (S3 
Group). Registered in Ireland no. 378073. Registered Office: South County 
Business Park, Leopardstown, Dublin 18.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Expected performance for WORM scenario

[Ondrej Valousek]

Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.
Ondrej

From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traffic isn't 
going over some 2mbit wifi by accident.

Our main storage has always been really slow (write speed of roughly 1.5MiB/s), 
but I had long attributed that to the extremely slow disks we use to back it, 
so now that we're expanding I set up a new gluster cluster with state of the 
art NVMe SSD drives to boost performance. However, performance only hopped up 
to around 2.1MiB/s. Perplexed, I tried it first with a 3-node cluster using 2GB 
ramdrives, which got me up to 2.4MiB/s. My last resort was to use a single node 
running on ramdisk, just to 100% exclude any network shenanigans, but the write 
performance stayed at an absolutely abysmal 3MiB/s.

Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I don't 
actually remember the numbers, but my test that took 2 minutes with gluster 
completed before I had time to blink). Writing straight to the backing SSD 
drives gives me a throughput of 96MiB/sec.

The test itself writes 8494 files that I simply took randomly from our 
production environment, comprising a total of 63.4MiB (so average file size is 
just under 8k. Most are actually close to 4k though, with the occasional 
2-or-so MB file in there.

I have googled and read a *lot* of performance-tuning guides, but the 3MiB/sec 
on single-node ramdisk seems to be far beyond the crippling one can cause by 
misconfiguration of a single system.

With this in mind; What sort of write performance can one reasonably hope to 
get with gluster? Assume a 3-node cluster running on top of (small) ramdisks on 
a fast and stable network. Is it just a bad fit for our workload?

/Andreas

-

The information contained in this e-mail and in any attachments is confidential 
and is designated solely for the attention of the intended recipient(s). If you 
are not an intended recipient, you must not use, disclose, copy, distribute or 
retain this e-mail or any part thereof. If you have received this e-mail in 
error, please notify the sender by return e-mail and delete all copies of this 
e-mail from your computer system(s). Please direct any additional queries to: 
communicati...@s3group.com. Thank You. Silicon and Software Systems Limited (S3 
Group). Registered in Ireland no. 378073. Registered Office: South County 
Business Park, Leopardstown, Dublin 18.___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Expected performance for WORM scenario

[Ondrej Valousek]

Well, it might be close to the _synchronous_ nfs, but it is still well behind 
of the asynchronous nfs performance.
Simple script (bit extreme I know, but helps to draw the picture):

#!/bin/csh

set HOSTNAME=`/bin/hostname`
set j=1
while ($j <= 7000)
   echo ahoj > test.$HOSTNAME.$j
   @ j++
end
rm -rf test.$HOSTNAME.*


Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS – i.e. 
10 times slower.

Ondrej

From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, March 13, 2018 8:28 AM
To: Ondrej Valousek <ondrej.valou...@s3group.com>
Cc: Andreas Ericsson <andreas.erics...@findity.com>; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org>]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traffic isn't 
going over some 2mbit wifi by accident.

Our main storage has always been really slow (write speed of roughly 1.5MiB/s), 
but I had long attributed that to the extremely slow disks we use to back it, 
so now that we're expanding I set up a new gluster cluster with state of the 
art NVMe SSD drives to boost performance. However, performance only hopped up 
to around 2.1MiB/s. Perplexed, I tried it first with a 3-node cluster using 2GB 
ramdrives, which got me up to 2.4MiB/s. My last resort was to use a single node 
running on ramdisk, just to 100% exclude any network shenanigans, but the write 
performance stayed at an absolutely abysmal 3MiB/s.

Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I don't 
actually remember the numbers, but my test that took 2 minutes with gluster 
completed before I had time to blink). Writing straight to the backing SSD 
drives gives me a throughput of 96MiB/sec.

The test itself writes 8494 files that I simply took randomly from our 
production environment, comprising a total of 63.4MiB (so average file size is 
just under 8k. Most are actually close to 4k though, with the occasional 
2-or-so MB file in there.

I have googled and read a *lot* of performance-tuning guides, but the 3MiB/sec 
on single-node ramdisk seems to be far beyond the crippling one can cause by 
misconfiguration of a single system.

With this in mind; What sort of write performance can one reasonably hope to 
get with gluster? Assume a 3-node cluster running on top of (small) ramdisks on 
a fast and stable network. Is it just a bad fit for our workload?

/Andreas

-



The information contained in this e-mail and in any attachments is confidential 
and is designated solely for the attention of the intended recipient(s). If you 
are not an intended recipient, you must not use, disclose, copy, distribute or 
retain this e-mail or any part thereof. If you have received this e-mail in 
error, please notify the sender by return e-mail and delete all copies of this 
e-mail from your computer system(s). Please direct any additional queries to: 
communicati...@s3group.com<mailto:communicati...@s3group.com>. Thank You. 
Silicon and Software Systems Limited (S3 Group). Registered in Ireland no. 
378073. Registered Office: South County Business Park, Leopardstown, Dublin 18.


___

Re: [Gluster-users] Expected performance for WORM scenario

[Ondrej Valousek]

Sorry - no time to play with that. But it’s simple to reproduce, just set up 
your of async nfs server, take my script and you will see on your own.
Ondrej

From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, March 13, 2018 10:41 AM
To: Ondrej Valousek <ondrej.valou...@s3group.com>
Cc: Andreas Ericsson <andreas.erics...@findity.com>; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Tue, Mar 13, 2018 at 2:42 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Yes, I have had this in place already (well except of the negative cache, but 
enabling that did not make much effect).
To me, this is no surprise – nothing can match nfs performance for small files 
for obvious reasons:

Could you give profile info of the run you did with and without nl-cache? 
Please also provide your volume info output.


1.   Single server, does not have to deal with distributed locks

2.   Afaik, gluster does not support read/write delegations the same way 
NFS does.

3.   Glusterfs is FUSE based
Glusterfs supports NFS/SMB/fuse

4.   Glusterfs does not support async writes
It supports async writes. It has a feature called write-behind which does that.


Summary: If you do not need to scale out, stick with a single server (+DRBD 
optionally for HA), it will give you the best performance

Ondrej


From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Tuesday, March 13, 2018 9:10 AM

To: Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>>
Cc: Andreas Ericsson 
<andreas.erics...@findity.com<mailto:andreas.erics...@findity.com>>; 
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Tue, Mar 13, 2018 at 1:37 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Well, it might be close to the _synchronous_ nfs, but it is still well behind 
of the asynchronous nfs performance.
Simple script (bit extreme I know, but helps to draw the picture):

#!/bin/csh

set HOSTNAME=`/bin/hostname`
set j=1
while ($j <= 7000)
   echo ahoj > test.$HOSTNAME.$j
   @ j++
end
rm -rf test.$HOSTNAME.*


Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS – i.e. 
10 times slower.

Do you have the new features enabled?

performance.stat-prefetch=on
performance.cache-invalidation=on
performance.md-cache-timeout=600
network.inode-lru-limit=5
performance.nl-cache=on
performance.nl-cache-timeout=600
network.inode-lru-limit=5


Ondrej

From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Tuesday, March 13, 2018 8:28 AM
To: Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>>
Cc: Andreas Ericsson 
<andreas.erics...@findity.com<mailto:andreas.erics...@findity.com>>; 
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org>]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no not

Re: [Gluster-users] Expected performance for WORM scenario

[Ondrej Valousek]

Yes, I have had this in place already (well except of the negative cache, but 
enabling that did not make much effect).
To me, this is no surprise – nothing can match nfs performance for small files 
for obvious reasons:

1.   Single server, does not have to deal with distributed locks

2.   Afaik, gluster does not support read/write delegations the same way 
NFS does.

3.   Glusterfs is FUSE based

4.   Glusterfs does not support async writes

Summary: If you do not need to scale out, stick with a single server (+DRBD 
optionally for HA), it will give you the best performance

Ondrej


From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, March 13, 2018 9:10 AM
To: Ondrej Valousek <ondrej.valou...@s3group.com>
Cc: Andreas Ericsson <andreas.erics...@findity.com>; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Tue, Mar 13, 2018 at 1:37 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Well, it might be close to the _synchronous_ nfs, but it is still well behind 
of the asynchronous nfs performance.
Simple script (bit extreme I know, but helps to draw the picture):

#!/bin/csh

set HOSTNAME=`/bin/hostname`
set j=1
while ($j <= 7000)
   echo ahoj > test.$HOSTNAME.$j
   @ j++
end
rm -rf test.$HOSTNAME.*


Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS – i.e. 
10 times slower.

Do you have the new features enabled?

performance.stat-prefetch=on
performance.cache-invalidation=on
performance.md-cache-timeout=600
network.inode-lru-limit=5
performance.nl-cache=on
performance.nl-cache-timeout=600
network.inode-lru-limit=5


Ondrej

From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Tuesday, March 13, 2018 8:28 AM
To: Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>>
Cc: Andreas Ericsson 
<andreas.erics...@findity.com<mailto:andreas.erics...@findity.com>>; 
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org>]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traffic isn't 
going over some 2mbit wifi by accident.

Our main storage has always been really slow (write speed of roughly 1.5MiB/s), 
but I had long attributed that to the extremely slow disks we use to back it, 
so now that we're expanding I set up a new gluster cluster with state of the 
art NVMe SSD drives to boost performance. However, performance only hopped up 
to around 2.1MiB/s. Perplexed, I tried it first with a 3-node cluster using 2GB 
ramdrives, which got me up to 2.4MiB/s. My last resort was to use a single node 
running on ramdisk, just to 100% exclude any network shenanigans, but the write 
performance stayed at an absolutely abysmal 3MiB/s.

Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I don't 
actually remember the numbers, but my test that took 2 minutes with gluster 
completed before I had time to blink). Writing straight to the backing 

Re: [Gluster-users] Expected performance for WORM scenario

[Ondrej Valousek]

Use DRBD then, that will give you required redundancy.

From: Andreas Ericsson [mailto:andreas.erics...@findity.com]
Sent: Wednesday, March 14, 2018 11:32 AM
To: Ondrej Valousek <ondrej.valou...@s3group.com>
Cc: Pranith Kumar Karampuri <pkara...@redhat.com>; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario

We can't stick to single server because the law. Redundancy is a legal 
requirement for our business.

I'm sort of giving up on gluster though. It would seem a pretty stupid content 
addressable storage would suit our needs better.

On 13 March 2018 at 10:12, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Yes, I have had this in place already (well except of the negative cache, but 
enabling that did not make much effect).
To me, this is no surprise – nothing can match nfs performance for small files 
for obvious reasons:

1.   Single server, does not have to deal with distributed locks

2.   Afaik, gluster does not support read/write delegations the same way 
NFS does.

3.   Glusterfs is FUSE based

4.   Glusterfs does not support async writes

Summary: If you do not need to scale out, stick with a single server (+DRBD 
optionally for HA), it will give you the best performance

Ondrej


From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Tuesday, March 13, 2018 9:10 AM

To: Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>>
Cc: Andreas Ericsson 
<andreas.erics...@findity.com<mailto:andreas.erics...@findity.com>>; 
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Tue, Mar 13, 2018 at 1:37 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Well, it might be close to the _synchronous_ nfs, but it is still well behind 
of the asynchronous nfs performance.
Simple script (bit extreme I know, but helps to draw the picture):

#!/bin/csh

set HOSTNAME=`/bin/hostname`
set j=1
while ($j <= 7000)
   echo ahoj > test.$HOSTNAME.$j
   @ j++
end
rm -rf test.$HOSTNAME.*


Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS – i.e. 
10 times slower.

Do you have the new features enabled?

performance.stat-prefetch=on
performance.cache-invalidation=on
performance.md-cache-timeout=600
network.inode-lru-limit=5
performance.nl-cache=on
performance.nl-cache-timeout=600
network.inode-lru-limit=5


Ondrej

From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Tuesday, March 13, 2018 8:28 AM
To: Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>>
Cc: Andreas Ericsson 
<andreas.erics...@findity.com<mailto:andreas.erics...@findity.com>>; 
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org>]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traff

Re: [Gluster-users] Expected performance for WORM scenario

[Ondrej Valousek]

Gluster offers distributed filesystem. It will NEVER perform as good as a local 
filesystem because it can’t.
I also believe NFS will always outperform Gluster in certain situations as it 
does not have to deal with distributed locks.

It’s also using FUSE which isn’t great performance-wise.

O.


From: Andreas Ericsson [mailto:andreas.erics...@findity.com]
Sent: Wednesday, March 14, 2018 10:43 AM
To: Pranith Kumar Karampuri <pkara...@redhat.com>
Cc: Ondrej Valousek <ondrej.valou...@s3group.com>; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario

That seems unlikely. I pre-create the directory layout and then write to 
directories I know exist.

I don't quite understand how any settings at all can reduce performance to 
1/5000 of what I get when writing straight to ramdisk though, and especially 
when running on a single node instead of in a cluster. Has anyone else set this 
up and managed to get better write performance?

On 13 March 2018 at 08:28, Pranith Kumar Karampuri 
<pkara...@redhat.com<mailto:pkara...@redhat.com>> wrote:


On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org>]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traffic isn't 
going over some 2mbit wifi by accident.

Our main storage has always been really slow (write speed of roughly 1.5MiB/s), 
but I had long attributed that to the extremely slow disks we use to back it, 
so now that we're expanding I set up a new gluster cluster with state of the 
art NVMe SSD drives to boost performance. However, performance only hopped up 
to around 2.1MiB/s. Perplexed, I tried it first with a 3-node cluster using 2GB 
ramdrives, which got me up to 2.4MiB/s. My last resort was to use a single node 
running on ramdisk, just to 100% exclude any network shenanigans, but the write 
performance stayed at an absolutely abysmal 3MiB/s.

Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I don't 
actually remember the numbers, but my test that took 2 minutes with gluster 
completed before I had time to blink). Writing straight to the backing SSD 
drives gives me a throughput of 96MiB/sec.

The test itself writes 8494 files that I simply took randomly from our 
production environment, comprising a total of 63.4MiB (so average file size is 
just under 8k. Most are actually close to 4k though, with the occasional 
2-or-so MB file in there.

I have googled and read a *lot* of performance-tuning guides, but the 3MiB/sec 
on single-node ramdisk seems to be far beyond the crippling one can cause by 
misconfiguration of a single system.

With this in mind; What sort of write performance can one reasonably hope to 
get with gluster? Assume a 3-node cluster running on top of (small) ramdisks on 
a fast and stable network. Is it just a bad fit for our workload?

/Andreas

-



The information contained in this e-mail and in any attachments is confidential 
and is designated solely for the attention of the intended recipient(s). If you 
are not an intended recipient, you must not use, disclose, copy, distribute or 
retain this e-mail or any part thereof. If you have received this e-mail in 
error, please notify the sender by re

[Gluster-users] pNFS

[Ondrej Valousek]

Hi list,

I am wondering why do we need Ganesha user-land NFS server in order to get pNFS 
working?
I understand Ganesha is necessary on the MDS, but standard kernel based NFS 
server should be sufficient on DS bricks (which should bring us additional 
performance), right?

Could someone clarify?
Thanks,

Ondrej

-

The information contained in this e-mail and in any attachments is confidential 
and is designated solely for the attention of the intended recipient(s). If you 
are not an intended recipient, you must not use, disclose, copy, distribute or 
retain this e-mail or any part thereof. If you have received this e-mail in 
error, please notify the sender by return e-mail and delete all copies of this 
e-mail from your computer system(s). Please direct any additional queries to: 
communicati...@s3group.com. Thank You. Silicon and Software Systems Limited (S3 
Group). Registered in Ireland no. 378073. Registered Office: South County 
Business Park, Leopardstown, Dublin 18.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster for home directories?

[Ondrej Valousek]

Hi,
Why do you need to replace your existing solution?
If you don't need to scale out due to the capacity reasons, the async NFS 
server will always outperform GlusterFS

Ondrej
-

The information contained in this e-mail and in any attachments is confidential 
and is designated solely for the attention of the intended recipient(s). If you 
are not an intended recipient, you must not use, disclose, copy, distribute or 
retain this e-mail or any part thereof. If you have received this e-mail in 
error, please notify the sender by return e-mail and delete all copies of this 
e-mail from your computer system(s). Please direct any additional queries to: 
communicati...@s3group.com. Thank You. Silicon and Software Systems Limited (S3 
Group). Registered in Ireland no. 378073. Registered Office: South County 
Business Park, Leopardstown, Dublin 18.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster very poor performance when copying small files (1x (2+1) = 3, SSD)

[Ondrej Valousek]

Hi,
As I posted in my previous emails - glusterfs can never match NFS (especially 
async one) performance of small files/latency. That's given by the design.
Nothing you can do about it.
Ondrej

-Original Message-
From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Rik Theys
Sent: Monday, March 19, 2018 10:38 AM
To: gluster-users@gluster.org; mailingli...@smcleod.net
Subject: Re: [Gluster-users] Gluster very poor performance when copying small 
files (1x (2+1) = 3, SSD)

Hi,

I've done some similar tests and experience similar performance issues (see my 
'gluster for home directories?' thread on the list).

If I read your mail correctly, you are comparing an NFS mount of the brick disk 
against a gluster mount (using the fuse client)?

Which options do you have set on the NFS export (sync or async)?

>From my tests, I concluded that the issue was not bandwidth but latency.
Gluster will only return an IO operation once all bricks have confirmed that 
the data is on disk. If you are using a fuse mount, you might compare with 
using the 'direct-io-mode=disable' option on the client might help (no 
experience with this).

In our tests, I've used NFS-ganesha to serve the gluster volume over NFS. This 
makes things even worse as NFS-ganesha has no "async" mode, which makes 
performance terrible.

If you find a magic knob to make glusterfs fast on small-file workloads, do let 
me know!

Regards,

Rik

On 03/18/2018 11:13 PM, Sam McLeod wrote:
> Howdy all,
> 
> We're experiencing terrible small file performance when copying or 
> moving files on gluster clients.
> 
> In the example below, Gluster is taking 6mins~ to copy 128MB / 21,000 
> files sideways on a client, doing the same thing on NFS (which I know 
> is a totally different solution etc. etc.) takes approximately 10-15 
> seconds(!).
> 
> Any advice for tuning the volume or XFS settings would be greatly 
> appreciated.
> 
> Hopefully I've included enough relevant information below.
> 
> 
> ## Gluster Client
> 
> root@gluster-client:/mnt/gluster_perf_test/  # du -sh .
> 127M    .
> root@gluster-client:/mnt/gluster_perf_test/  # find . -type f | wc -l
> 21791
> root@gluster-client:/mnt/gluster_perf_test/  # du 9584toto9584.txt
> 4    9584toto9584.txt
> 
> 
> root@gluster-client:/mnt/gluster_perf_test/  # time cp -a private 
> private_perf_test
> 
> real    5m51.862s
> user    0m0.862s
> sys    0m8.334s
> 
> root@gluster-client:/mnt/gluster_perf_test/ # time rm -rf 
> private_perf_test/
> 
> real    0m49.702s
> user    0m0.087s
> sys    0m0.958s
> 
> 
> ## Hosts
> 
> - 16x Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz per Gluster host / 
> client
> - Storage: iSCSI provisioned (via 10Gbit DAC/Fibre), SSD disk, 50K 
> R/RW 4k IOP/s, 400MB/s per Gluster host
> - Volumes are replicated across two hosts and one arbiter only host
> - Networking is 10Gbit DAC/Fibre between Gluster hosts and clients
> - 18GB DDR4 ECC memory
> 
> ## Volume Info
> 
> root@gluster-host-01:~ # gluster pool list UUID          Hostname             
>            
> State ad02970b-e2aa-4ca8-998c-bd10d5970faa  gluster-host-02.fqdn 
> Connected ea116a94-c19e-48db-b108-0be3ae622e2e  gluster-host-03.fqdn 
> Connected
> 2e855c25-e7ac-4ff6-be85-e8bcc6f45ee4  localhost Connected
> 
> root@gluster-host-01:~ # gluster volume info uat_storage
> 
> Volume Name: uat_storage
> Type: Replicate
> Volume ID: 7918f1c5-5031-47b8-b054-56f6f0c569a2
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: gluster-host-01.fqdn:/mnt/gluster-storage/uat_storage
> Brick2: gluster-host-02.fqdn:/mnt/gluster-storage/uat_storage
> Brick3: gluster-host-03.fqdn:/mnt/gluster-storage/uat_storage 
> (arbiter) Options Reconfigured:
> performance.rda-cache-limit: 256MB
> network.inode-lru-limit: 5
> server.outstanding-rpc-limit: 256
> performance.client-io-threads: true
> nfs.disable: on
> transport.address-family: inet
> client.event-threads: 8
> cluster.eager-lock: true
> cluster.favorite-child-policy: size
> cluster.lookup-optimize: true
> cluster.readdir-optimize: true
> cluster.use-compound-fops: true
> diagnostics.brick-log-level: ERROR
> diagnostics.client-log-level: ERROR
> features.cache-invalidation-timeout: 600
> features.cache-invalidation: true
> network.ping-timeout: 15
> performance.cache-invalidation: true
> performance.cache-max-file-size: 6MB
> performance.cache-refresh-timeout: 60
> performance.cache-size: 1024MB
> performance.io -thread-count: 16
> performance.md-cache-timeout: 600
> performance.stat-prefetch: true
> performance.write-behind-window-size: 256MB
> server.event-threads: 8
> transport.listen-backlog: 2048
> 
> root@gluster-host-01:~ # xfs_info /dev/mapper/gluster-storage-unlocked
> meta-data=/dev/mapper/gluster-storage-unlocked isize=512    agcount=4,
> agsize=196607360 blks
>          =                       sectsz=512   attr=2, projid32bit=1
>          =