Re: [Gluster-users] Expected performance for WORM scenario

[Ondrej Valousek]

Use DRBD then, that will give you required redundancy.

From: Andreas Ericsson [mailto:andreas.erics...@findity.com]
Sent: Wednesday, March 14, 2018 11:32 AM
To: Ondrej Valousek <ondrej.valou...@s3group.com>
Cc: Pranith Kumar Karampuri <pkara...@redhat.com>; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario

We can't stick to single server because the law. Redundancy is a legal 
requirement for our business.

I'm sort of giving up on gluster though. It would seem a pretty stupid content 
addressable storage would suit our needs better.

On 13 March 2018 at 10:12, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Yes, I have had this in place already (well except of the negative cache, but 
enabling that did not make much effect).
To me, this is no surprise – nothing can match nfs performance for small files 
for obvious reasons:

1.   Single server, does not have to deal with distributed locks

2.   Afaik, gluster does not support read/write delegations the same way 
NFS does.

3.   Glusterfs is FUSE based

4.   Glusterfs does not support async writes

Summary: If you do not need to scale out, stick with a single server (+DRBD 
optionally for HA), it will give you the best performance

Ondrej


From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Tuesday, March 13, 2018 9:10 AM

To: Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>>
Cc: Andreas Ericsson 
<andreas.erics...@findity.com<mailto:andreas.erics...@findity.com>>; 
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Tue, Mar 13, 2018 at 1:37 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Well, it might be close to the _synchronous_ nfs, but it is still well behind 
of the asynchronous nfs performance.
Simple script (bit extreme I know, but helps to draw the picture):

#!/bin/csh

set HOSTNAME=`/bin/hostname`
set j=1
while ($j <= 7000)
   echo ahoj > test.$HOSTNAME.$j
   @ j++
end
rm -rf test.$HOSTNAME.*


Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS – i.e. 
10 times slower.

Do you have the new features enabled?

performance.stat-prefetch=on
performance.cache-invalidation=on
performance.md-cache-timeout=600
network.inode-lru-limit=5
performance.nl-cache=on
performance.nl-cache-timeout=600
network.inode-lru-limit=5


Ondrej

From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Tuesday, March 13, 2018 8:28 AM
To: Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>>
Cc: Andreas Ericsson 
<andreas.erics...@findity.com<mailto:andreas.erics...@findity.com>>; 
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org>]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traff

Re: [Gluster-users] Expected performance for WORM scenario

[Ingard Mevåg]

If you could replicate the problem you had and provide the volume info +
profile that was requested from the redhat guys that would help in trying
to understand what is happening with your workload. Also if possible the
script you used to generate the load.

We've had our share of difficulties with small files on glusterfs as well.
The last problem we encountered is described here
https://bugzilla.redhat.com/show_bug.cgi?id=1546732 and seems to be a
regression from 3.10 to 3.12, but it is related to stat calls taking a long
time and that could be your problem as well. The diagnostics profile output
would tell us if thats the case.

That being said we've started experimenting with
https://github.com/chrislusf/seaweedfs for some of our small file workload.

2018-03-14 11:31 GMT+01:00 Andreas Ericsson <andreas.erics...@findity.com>:

> We can't stick to single server because the law. Redundancy is a legal
> requirement for our business.
>
> I'm sort of giving up on gluster though. It would seem a pretty stupid
> content addressable storage would suit our needs better.
>
> On 13 March 2018 at 10:12, Ondrej Valousek <ondrej.valou...@s3group.com>
> wrote:
>
>> Yes, I have had this in place already (well except of the negative cache,
>> but enabling that did not make much effect).
>>
>> To me, this is no surprise – nothing can match nfs performance for small
>> files for obvious reasons:
>>
>> 1.   Single server, does not have to deal with distributed locks
>>
>> 2.   Afaik, gluster does not support read/write delegations the same
>> way NFS does.
>>
>> 3.   Glusterfs is FUSE based
>>
>> 4.   Glusterfs does not support async writes
>>
>>
>>
>> Summary: If you do not need to scale out, stick with a single server
>> (+DRBD optionally for HA), it will give you the best performance
>>
>>
>>
>> Ondrej
>>
>>
>>
>>
>>
>> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
>> *Sent:* Tuesday, March 13, 2018 9:10 AM
>>
>> *To:* Ondrej Valousek <ondrej.valou...@s3group.com>
>> *Cc:* Andreas Ericsson <andreas.erics...@findity.com>;
>> Gluster-users@gluster.org
>> *Subject:* Re: [Gluster-users] Expected performance for WORM scenario
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Mar 13, 2018 at 1:37 PM, Ondrej Valousek <
>> ondrej.valou...@s3group.com> wrote:
>>
>> Well, it might be close to the _*synchronous*_ nfs, but it is still well
>> behind of the asynchronous nfs performance.
>>
>> Simple script (bit extreme I know, but helps to draw the picture):
>>
>>
>>
>> #!/bin/csh
>>
>>
>>
>> set HOSTNAME=`/bin/hostname`
>>
>> set j=1
>>
>> while ($j <= 7000)
>>
>>echo ahoj > test.$HOSTNAME.$j
>>
>>@ j++
>>
>> end
>>
>> rm -rf test.$HOSTNAME.*
>>
>>
>>
>>
>>
>> Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS
>> – i.e. 10 times slower.
>>
>>
>>
>> Do you have the new features enabled?
>>
>>
>>
>> performance.stat-prefetch=on
>> performance.cache-invalidation=on
>> performance.md-cache-timeout=600
>> network.inode-lru-limit=5
>>
>> performance.nl-cache=on
>> performance.nl-cache-timeout=600
>> network.inode-lru-limit=5
>>
>>
>>
>>
>>
>> Ondrej
>>
>>
>>
>> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
>> *Sent:* Tuesday, March 13, 2018 8:28 AM
>> *To:* Ondrej Valousek <ondrej.valou...@s3group.com>
>> *Cc:* Andreas Ericsson <andreas.erics...@findity.com>;
>> Gluster-users@gluster.org
>> *Subject:* Re: [Gluster-users] Expected performance for WORM scenario
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek <
>> ondrej.valou...@s3group.com> wrote:
>>
>> Hi,
>>
>> Gluster will never perform well for small files.
>>
>> I believe there is  nothing you can do with this.
>>
>>
>>
>> It is bad compared to a disk filesystem but I believe it is much closer
>> to NFS now.
>>
>>
>>
>> Andreas,
>>
>>  Looking at your workload, I am suspecting there to be lot of LOOKUPs
>> which reduce performance. Is it possible to do the following?
>>
>>
>>
>> # gluster volume profile  info incremental
>>
>> #execute your workload
>>
>> # gluster volu

Re: [Gluster-users] Expected performance for WORM scenario

[Ondrej Valousek]

Gluster offers distributed filesystem. It will NEVER perform as good as a local 
filesystem because it can’t.
I also believe NFS will always outperform Gluster in certain situations as it 
does not have to deal with distributed locks.

It’s also using FUSE which isn’t great performance-wise.

O.


From: Andreas Ericsson [mailto:andreas.erics...@findity.com]
Sent: Wednesday, March 14, 2018 10:43 AM
To: Pranith Kumar Karampuri <pkara...@redhat.com>
Cc: Ondrej Valousek <ondrej.valou...@s3group.com>; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario

That seems unlikely. I pre-create the directory layout and then write to 
directories I know exist.

I don't quite understand how any settings at all can reduce performance to 
1/5000 of what I get when writing straight to ramdisk though, and especially 
when running on a single node instead of in a cluster. Has anyone else set this 
up and managed to get better write performance?

On 13 March 2018 at 08:28, Pranith Kumar Karampuri 
<pkara...@redhat.com<mailto:pkara...@redhat.com>> wrote:


On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org>]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traffic isn't 
going over some 2mbit wifi by accident.

Our main storage has always been really slow (write speed of roughly 1.5MiB/s), 
but I had long attributed that to the extremely slow disks we use to back it, 
so now that we're expanding I set up a new gluster cluster with state of the 
art NVMe SSD drives to boost performance. However, performance only hopped up 
to around 2.1MiB/s. Perplexed, I tried it first with a 3-node cluster using 2GB 
ramdrives, which got me up to 2.4MiB/s. My last resort was to use a single node 
running on ramdisk, just to 100% exclude any network shenanigans, but the write 
performance stayed at an absolutely abysmal 3MiB/s.

Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I don't 
actually remember the numbers, but my test that took 2 minutes with gluster 
completed before I had time to blink). Writing straight to the backing SSD 
drives gives me a throughput of 96MiB/sec.

The test itself writes 8494 files that I simply took randomly from our 
production environment, comprising a total of 63.4MiB (so average file size is 
just under 8k. Most are actually close to 4k though, with the occasional 
2-or-so MB file in there.

I have googled and read a *lot* of performance-tuning guides, but the 3MiB/sec 
on single-node ramdisk seems to be far beyond the crippling one can cause by 
misconfiguration of a single system.

With this in mind; What sort of write performance can one reasonably hope to 
get with gluster? Assume a 3-node cluster running on top of (small) ramdisks on 
a fast and stable network. Is it just a bad fit for our workload?

/Andreas

-



The information contained in this e-mail and in any attachments is confidential 
and is designated solely for the attention of the intended recipient(s). If you 
are not an intended recipient, you must not use, disclose, copy, distribute or 
retain this e-mail or any part thereof. If you have received this e-mail in 
error, please notify the sender by re

Re: [Gluster-users] Expected performance for WORM scenario

[Andreas Ericsson]

We can't stick to single server because the law. Redundancy is a legal
requirement for our business.

I'm sort of giving up on gluster though. It would seem a pretty stupid
content addressable storage would suit our needs better.

On 13 March 2018 at 10:12, Ondrej Valousek <ondrej.valou...@s3group.com>
wrote:

> Yes, I have had this in place already (well except of the negative cache,
> but enabling that did not make much effect).
>
> To me, this is no surprise – nothing can match nfs performance for small
> files for obvious reasons:
>
> 1.   Single server, does not have to deal with distributed locks
>
> 2.   Afaik, gluster does not support read/write delegations the same
> way NFS does.
>
> 3.   Glusterfs is FUSE based
>
> 4.   Glusterfs does not support async writes
>
>
>
> Summary: If you do not need to scale out, stick with a single server
> (+DRBD optionally for HA), it will give you the best performance
>
>
>
> Ondrej
>
>
>
>
>
> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> *Sent:* Tuesday, March 13, 2018 9:10 AM
>
> *To:* Ondrej Valousek <ondrej.valou...@s3group.com>
> *Cc:* Andreas Ericsson <andreas.erics...@findity.com>;
> Gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Expected performance for WORM scenario
>
>
>
>
>
>
>
> On Tue, Mar 13, 2018 at 1:37 PM, Ondrej Valousek <
> ondrej.valou...@s3group.com> wrote:
>
> Well, it might be close to the _*synchronous*_ nfs, but it is still well
> behind of the asynchronous nfs performance.
>
> Simple script (bit extreme I know, but helps to draw the picture):
>
>
>
> #!/bin/csh
>
>
>
> set HOSTNAME=`/bin/hostname`
>
> set j=1
>
> while ($j <= 7000)
>
>echo ahoj > test.$HOSTNAME.$j
>
>@ j++
>
> end
>
> rm -rf test.$HOSTNAME.*
>
>
>
>
>
> Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS –
> i.e. 10 times slower.
>
>
>
> Do you have the new features enabled?
>
>
>
> performance.stat-prefetch=on
> performance.cache-invalidation=on
> performance.md-cache-timeout=600
> network.inode-lru-limit=5
>
> performance.nl-cache=on
> performance.nl-cache-timeout=600
> network.inode-lru-limit=50000
>
>
>
>
>
> Ondrej
>
>
>
> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> *Sent:* Tuesday, March 13, 2018 8:28 AM
> *To:* Ondrej Valousek <ondrej.valou...@s3group.com>
> *Cc:* Andreas Ericsson <andreas.erics...@findity.com>;
> Gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Expected performance for WORM scenario
>
>
>
>
>
>
>
> On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek <
> ondrej.valou...@s3group.com> wrote:
>
> Hi,
>
> Gluster will never perform well for small files.
>
> I believe there is  nothing you can do with this.
>
>
>
> It is bad compared to a disk filesystem but I believe it is much closer to
> NFS now.
>
>
>
> Andreas,
>
>  Looking at your workload, I am suspecting there to be lot of LOOKUPs
> which reduce performance. Is it possible to do the following?
>
>
>
> # gluster volume profile  info incremental
>
> #execute your workload
>
> # gluster volume profile  info incremental >
> /path/to/file/that/you/need/to/send/us
>
>
>
> If the last line in there is LOOKUP, mostly we need to enable nl-cache
> feature and see how it performs.
>
>
>
> Ondrej
>
>
>
> *From:* gluster-users-boun...@gluster.org [mailto:gluster-users-bounces@
> gluster.org] *On Behalf Of *Andreas Ericsson
> *Sent:* Monday, March 12, 2018 1:47 PM
> *To:* Gluster-users@gluster.org
> *Subject:* [Gluster-users] Expected performance for WORM scenario
>
>
>
> Heya fellas.
>
>
>
> I've been struggling quite a lot to get glusterfs to perform even
> halfdecently with a write-intensive workload. Testnumbers are from gluster
> 3.10.7.
>
>
>
> We store a bunch of small files in a doubly-tiered sha1 hash fanout
> directory structure. The directories themselves aren't overly full. Most of
> the data we write to gluster is "write once, read probably never", so 99%
> of all operations are of the write variety.
>
>
>
> The network between servers is sound. 10gb network cards run over a 10gb
> (doh) switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 -
> 0.2 ms. There is no firewall, no packet inspection and no nothing between
> the servers, and the 10gb switch is the only path between the two machines,
> so traffic isn't going over some 2mbit wifi by accident.
>
>
>

Re: [Gluster-users] Expected performance for WORM scenario

[Andreas Ericsson]

That seems unlikely. I pre-create the directory layout and then write to
directories I know exist.

I don't quite understand how any settings at all can reduce performance to
1/5000 of what I get when writing straight to ramdisk though, and
especially when running on a single node instead of in a cluster. Has
anyone else set this up and managed to get better write performance?

On 13 March 2018 at 08:28, Pranith Kumar Karampuri <pkara...@redhat.com>
wrote:

>
>
> On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek <
> ondrej.valou...@s3group.com> wrote:
>
>> Hi,
>>
>> Gluster will never perform well for small files.
>>
>> I believe there is  nothing you can do with this.
>>
>
> It is bad compared to a disk filesystem but I believe it is much closer to
> NFS now.
>
> Andreas,
>  Looking at your workload, I am suspecting there to be lot of LOOKUPs
> which reduce performance. Is it possible to do the following?
>
> # gluster volume profile  info incremental
> #execute your workload
> # gluster volume profile  info incremental >
> /path/to/file/that/you/need/to/send/us
>
> If the last line in there is LOOKUP, mostly we need to enable nl-cache
> feature and see how it performs.
>
>
>> Ondrej
>>
>>
>>
>> *From:* gluster-users-boun...@gluster.org [mailto:gluster-users-bounces@
>> gluster.org] *On Behalf Of *Andreas Ericsson
>> *Sent:* Monday, March 12, 2018 1:47 PM
>> *To:* Gluster-users@gluster.org
>> *Subject:* [Gluster-users] Expected performance for WORM scenario
>>
>>
>>
>> Heya fellas.
>>
>>
>>
>> I've been struggling quite a lot to get glusterfs to perform even
>> halfdecently with a write-intensive workload. Testnumbers are from gluster
>> 3.10.7.
>>
>>
>>
>> We store a bunch of small files in a doubly-tiered sha1 hash fanout
>> directory structure. The directories themselves aren't overly full. Most of
>> the data we write to gluster is "write once, read probably never", so 99%
>> of all operations are of the write variety.
>>
>>
>>
>> The network between servers is sound. 10gb network cards run over a 10gb
>> (doh) switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 -
>> 0.2 ms. There is no firewall, no packet inspection and no nothing between
>> the servers, and the 10gb switch is the only path between the two machines,
>> so traffic isn't going over some 2mbit wifi by accident.
>>
>>
>>
>> Our main storage has always been really slow (write speed of roughly
>> 1.5MiB/s), but I had long attributed that to the extremely slow disks we
>> use to back it, so now that we're expanding I set up a new gluster cluster
>> with state of the art NVMe SSD drives to boost performance. However,
>> performance only hopped up to around 2.1MiB/s. Perplexed, I tried it first
>> with a 3-node cluster using 2GB ramdrives, which got me up to 2.4MiB/s. My
>> last resort was to use a single node running on ramdisk, just to 100%
>> exclude any network shenanigans, but the write performance stayed at an
>> absolutely abysmal 3MiB/s.
>>
>>
>>
>> Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I
>> don't actually remember the numbers, but my test that took 2 minutes with
>> gluster completed before I had time to blink). Writing straight to the
>> backing SSD drives gives me a throughput of 96MiB/sec.
>>
>>
>>
>> The test itself writes 8494 files that I simply took randomly from our
>> production environment, comprising a total of 63.4MiB (so average file size
>> is just under 8k. Most are actually close to 4k though, with the occasional
>> 2-or-so MB file in there.
>>
>>
>>
>> I have googled and read a *lot* of performance-tuning guides, but the
>> 3MiB/sec on single-node ramdisk seems to be far beyond the crippling one
>> can cause by misconfiguration of a single system.
>>
>>
>>
>> With this in mind; What sort of write performance can one reasonably hope
>> to get with gluster? Assume a 3-node cluster running on top of (small)
>> ramdisks on a fast and stable network. Is it just a bad fit for our
>> workload?
>>
>>
>>
>> /Andreas
>>
>> -
>>
>> The information contained in this e-mail and in any attachments is 
>> confidential and is designated solely for the attention of the intended 
>> recipient(s). If you are not an intended recipient, you must not use, 
>> disclose, copy, distribute or retain this e-mail or any part thereof. If you 
>> have rece

Re: [Gluster-users] Expected performance for WORM scenario

[Andreas Ericsson]

I no longer have the volume lying around. The most interesting one was a
2GB volume created on ramdisk for a single node. If I can't get that to go
faster than 3MB/sec for writes, I figured I wouldn't bother further.

I was using gluster fuse fs 3.10.7. Everything was running on ubuntu 16.04
servers.

On 12 March 2018 at 15:30, Nithya Balachandran  wrote:

> Hi,
>
> Can you send us the following details:
> 1. gluster volume info
> 2. What client you are using to run this?
>
> Thanks,
> Nithya
>
> On 12 March 2018 at 18:16, Andreas Ericsson 
> wrote:
>
>> Heya fellas.
>>
>> I've been struggling quite a lot to get glusterfs to perform even
>> halfdecently with a write-intensive workload. Testnumbers are from gluster
>> 3.10.7.
>>
>> We store a bunch of small files in a doubly-tiered sha1 hash fanout
>> directory structure. The directories themselves aren't overly full. Most of
>> the data we write to gluster is "write once, read probably never", so 99%
>> of all operations are of the write variety.
>>
>> The network between servers is sound. 10gb network cards run over a 10gb
>> (doh) switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 -
>> 0.2 ms. There is no firewall, no packet inspection and no nothing between
>> the servers, and the 10gb switch is the only path between the two machines,
>> so traffic isn't going over some 2mbit wifi by accident.
>>
>> Our main storage has always been really slow (write speed of roughly
>> 1.5MiB/s), but I had long attributed that to the extremely slow disks we
>> use to back it, so now that we're expanding I set up a new gluster cluster
>> with state of the art NVMe SSD drives to boost performance. However,
>> performance only hopped up to around 2.1MiB/s. Perplexed, I tried it first
>> with a 3-node cluster using 2GB ramdrives, which got me up to 2.4MiB/s. My
>> last resort was to use a single node running on ramdisk, just to 100%
>> exclude any network shenanigans, but the write performance stayed at an
>> absolutely abysmal 3MiB/s.
>>
>> Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I
>> don't actually remember the numbers, but my test that took 2 minutes with
>> gluster completed before I had time to blink). Writing straight to the
>> backing SSD drives gives me a throughput of 96MiB/sec.
>>
>> The test itself writes 8494 files that I simply took randomly from our
>> production environment, comprising a total of 63.4MiB (so average file size
>> is just under 8k. Most are actually close to 4k though, with the occasional
>> 2-or-so MB file in there.
>>
>> I have googled and read a *lot* of performance-tuning guides, but the
>> 3MiB/sec on single-node ramdisk seems to be far beyond the crippling one
>> can cause by misconfiguration of a single system.
>>
>> With this in mind; What sort of write performance can one reasonably hope
>> to get with gluster? Assume a 3-node cluster running on top of (small)
>> ramdisks on a fast and stable network. Is it just a bad fit for our
>> workload?
>>
>> /Andreas
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Expected performance for WORM scenario

[Ondrej Valousek]

Sorry - no time to play with that. But it’s simple to reproduce, just set up 
your of async nfs server, take my script and you will see on your own.
Ondrej

From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, March 13, 2018 10:41 AM
To: Ondrej Valousek <ondrej.valou...@s3group.com>
Cc: Andreas Ericsson <andreas.erics...@findity.com>; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Tue, Mar 13, 2018 at 2:42 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Yes, I have had this in place already (well except of the negative cache, but 
enabling that did not make much effect).
To me, this is no surprise – nothing can match nfs performance for small files 
for obvious reasons:

Could you give profile info of the run you did with and without nl-cache? 
Please also provide your volume info output.


1.   Single server, does not have to deal with distributed locks

2.   Afaik, gluster does not support read/write delegations the same way 
NFS does.

3.   Glusterfs is FUSE based
Glusterfs supports NFS/SMB/fuse

4.   Glusterfs does not support async writes
It supports async writes. It has a feature called write-behind which does that.


Summary: If you do not need to scale out, stick with a single server (+DRBD 
optionally for HA), it will give you the best performance

Ondrej


From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Tuesday, March 13, 2018 9:10 AM

To: Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>>
Cc: Andreas Ericsson 
<andreas.erics...@findity.com<mailto:andreas.erics...@findity.com>>; 
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Tue, Mar 13, 2018 at 1:37 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Well, it might be close to the _synchronous_ nfs, but it is still well behind 
of the asynchronous nfs performance.
Simple script (bit extreme I know, but helps to draw the picture):

#!/bin/csh

set HOSTNAME=`/bin/hostname`
set j=1
while ($j <= 7000)
   echo ahoj > test.$HOSTNAME.$j
   @ j++
end
rm -rf test.$HOSTNAME.*


Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS – i.e. 
10 times slower.

Do you have the new features enabled?

performance.stat-prefetch=on
performance.cache-invalidation=on
performance.md-cache-timeout=600
network.inode-lru-limit=5
performance.nl-cache=on
performance.nl-cache-timeout=600
network.inode-lru-limit=5


Ondrej

From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Tuesday, March 13, 2018 8:28 AM
To: Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>>
Cc: Andreas Ericsson 
<andreas.erics...@findity.com<mailto:andreas.erics...@findity.com>>; 
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org>]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no not

Re: [Gluster-users] Expected performance for WORM scenario

[Pranith Kumar Karampuri]

On Tue, Mar 13, 2018 at 2:42 PM, Ondrej Valousek <
ondrej.valou...@s3group.com> wrote:

> Yes, I have had this in place already (well except of the negative cache,
> but enabling that did not make much effect).
>
> To me, this is no surprise – nothing can match nfs performance for small
> files for obvious reasons:
>

Could you give profile info of the run you did with and without nl-cache?
Please also provide your volume info output.


> 1.   Single server, does not have to deal with distributed locks
>
> 2.   Afaik, gluster does not support read/write delegations the same
> way NFS does.
>
> 3.   Glusterfs is FUSE based
>
Glusterfs supports NFS/SMB/fuse

> 4.   Glusterfs does not support async writes
>
It supports async writes. It has a feature called write-behind which does
that.


>
> Summary: If you do not need to scale out, stick with a single server
> (+DRBD optionally for HA), it will give you the best performance
>
>
>
> Ondrej
>
>
>
>
>
> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> *Sent:* Tuesday, March 13, 2018 9:10 AM
>
> *To:* Ondrej Valousek <ondrej.valou...@s3group.com>
> *Cc:* Andreas Ericsson <andreas.erics...@findity.com>;
> Gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Expected performance for WORM scenario
>
>
>
>
>
>
>
> On Tue, Mar 13, 2018 at 1:37 PM, Ondrej Valousek <
> ondrej.valou...@s3group.com> wrote:
>
> Well, it might be close to the _*synchronous*_ nfs, but it is still well
> behind of the asynchronous nfs performance.
>
> Simple script (bit extreme I know, but helps to draw the picture):
>
>
>
> #!/bin/csh
>
>
>
> set HOSTNAME=`/bin/hostname`
>
> set j=1
>
> while ($j <= 7000)
>
>echo ahoj > test.$HOSTNAME.$j
>
>@ j++
>
> end
>
> rm -rf test.$HOSTNAME.*
>
>
>
>
>
> Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS –
> i.e. 10 times slower.
>
>
>
> Do you have the new features enabled?
>
>
>
> performance.stat-prefetch=on
> performance.cache-invalidation=on
> performance.md-cache-timeout=600
> network.inode-lru-limit=5
>
> performance.nl-cache=on
> performance.nl-cache-timeout=600
> network.inode-lru-limit=5
>
>
>
>
>
> Ondrej
>
>
>
> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> *Sent:* Tuesday, March 13, 2018 8:28 AM
> *To:* Ondrej Valousek <ondrej.valou...@s3group.com>
> *Cc:* Andreas Ericsson <andreas.erics...@findity.com>;
> Gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Expected performance for WORM scenario
>
>
>
>
>
>
>
> On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek <
> ondrej.valou...@s3group.com> wrote:
>
> Hi,
>
> Gluster will never perform well for small files.
>
> I believe there is  nothing you can do with this.
>
>
>
> It is bad compared to a disk filesystem but I believe it is much closer to
> NFS now.
>
>
>
> Andreas,
>
>  Looking at your workload, I am suspecting there to be lot of LOOKUPs
> which reduce performance. Is it possible to do the following?
>
>
>
> # gluster volume profile  info incremental
>
> #execute your workload
>
> # gluster volume profile  info incremental >
> /path/to/file/that/you/need/to/send/us
>
>
>
> If the last line in there is LOOKUP, mostly we need to enable nl-cache
> feature and see how it performs.
>
>
>
> Ondrej
>
>
>
> *From:* gluster-users-boun...@gluster.org [mailto:gluster-users-bounces@
> gluster.org] *On Behalf Of *Andreas Ericsson
> *Sent:* Monday, March 12, 2018 1:47 PM
> *To:* Gluster-users@gluster.org
> *Subject:* [Gluster-users] Expected performance for WORM scenario
>
>
>
> Heya fellas.
>
>
>
> I've been struggling quite a lot to get glusterfs to perform even
> halfdecently with a write-intensive workload. Testnumbers are from gluster
> 3.10.7.
>
>
>
> We store a bunch of small files in a doubly-tiered sha1 hash fanout
> directory structure. The directories themselves aren't overly full. Most of
> the data we write to gluster is "write once, read probably never", so 99%
> of all operations are of the write variety.
>
>
>
> The network between servers is sound. 10gb network cards run over a 10gb
> (doh) switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 -
> 0.2 ms. There is no firewall, no packet inspection and no nothing between
> the servers, and the 10gb switch is the only path between the two machines,
> so traffic isn't going over some 2mbit wifi by accident.
>
>
>

Re: [Gluster-users] Expected performance for WORM scenario

[Ondrej Valousek]

Yes, I have had this in place already (well except of the negative cache, but 
enabling that did not make much effect).
To me, this is no surprise – nothing can match nfs performance for small files 
for obvious reasons:

1.   Single server, does not have to deal with distributed locks

2.   Afaik, gluster does not support read/write delegations the same way 
NFS does.

3.   Glusterfs is FUSE based

4.   Glusterfs does not support async writes

Summary: If you do not need to scale out, stick with a single server (+DRBD 
optionally for HA), it will give you the best performance

Ondrej


From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, March 13, 2018 9:10 AM
To: Ondrej Valousek <ondrej.valou...@s3group.com>
Cc: Andreas Ericsson <andreas.erics...@findity.com>; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Tue, Mar 13, 2018 at 1:37 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Well, it might be close to the _synchronous_ nfs, but it is still well behind 
of the asynchronous nfs performance.
Simple script (bit extreme I know, but helps to draw the picture):

#!/bin/csh

set HOSTNAME=`/bin/hostname`
set j=1
while ($j <= 7000)
   echo ahoj > test.$HOSTNAME.$j
   @ j++
end
rm -rf test.$HOSTNAME.*


Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS – i.e. 
10 times slower.

Do you have the new features enabled?

performance.stat-prefetch=on
performance.cache-invalidation=on
performance.md-cache-timeout=600
network.inode-lru-limit=5
performance.nl-cache=on
performance.nl-cache-timeout=600
network.inode-lru-limit=5


Ondrej

From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Tuesday, March 13, 2018 8:28 AM
To: Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>>
Cc: Andreas Ericsson 
<andreas.erics...@findity.com<mailto:andreas.erics...@findity.com>>; 
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org>]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traffic isn't 
going over some 2mbit wifi by accident.

Our main storage has always been really slow (write speed of roughly 1.5MiB/s), 
but I had long attributed that to the extremely slow disks we use to back it, 
so now that we're expanding I set up a new gluster cluster with state of the 
art NVMe SSD drives to boost performance. However, performance only hopped up 
to around 2.1MiB/s. Perplexed, I tried it first with a 3-node cluster using 2GB 
ramdrives, which got me up to 2.4MiB/s. My last resort was to use a single node 
running on ramdisk, just to 100% exclude any network shenanigans, but the write 
performance stayed at an absolutely abysmal 3MiB/s.

Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I don't 
actually remember the numbers, but my test that took 2 minutes with gluster 
completed before I had time to blink). Writing straight to the backing 

Re: [Gluster-users] Expected performance for WORM scenario

[Ondrej Valousek]

Well, it might be close to the _synchronous_ nfs, but it is still well behind 
of the asynchronous nfs performance.
Simple script (bit extreme I know, but helps to draw the picture):

#!/bin/csh

set HOSTNAME=`/bin/hostname`
set j=1
while ($j <= 7000)
   echo ahoj > test.$HOSTNAME.$j
   @ j++
end
rm -rf test.$HOSTNAME.*


Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS – i.e. 
10 times slower.

Ondrej

From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, March 13, 2018 8:28 AM
To: Ondrej Valousek <ondrej.valou...@s3group.com>
Cc: Andreas Ericsson <andreas.erics...@findity.com>; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
<ondrej.valou...@s3group.com<mailto:ondrej.valou...@s3group.com>> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org>]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traffic isn't 
going over some 2mbit wifi by accident.

Our main storage has always been really slow (write speed of roughly 1.5MiB/s), 
but I had long attributed that to the extremely slow disks we use to back it, 
so now that we're expanding I set up a new gluster cluster with state of the 
art NVMe SSD drives to boost performance. However, performance only hopped up 
to around 2.1MiB/s. Perplexed, I tried it first with a 3-node cluster using 2GB 
ramdrives, which got me up to 2.4MiB/s. My last resort was to use a single node 
running on ramdisk, just to 100% exclude any network shenanigans, but the write 
performance stayed at an absolutely abysmal 3MiB/s.

Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I don't 
actually remember the numbers, but my test that took 2 minutes with gluster 
completed before I had time to blink). Writing straight to the backing SSD 
drives gives me a throughput of 96MiB/sec.

The test itself writes 8494 files that I simply took randomly from our 
production environment, comprising a total of 63.4MiB (so average file size is 
just under 8k. Most are actually close to 4k though, with the occasional 
2-or-so MB file in there.

I have googled and read a *lot* of performance-tuning guides, but the 3MiB/sec 
on single-node ramdisk seems to be far beyond the crippling one can cause by 
misconfiguration of a single system.

With this in mind; What sort of write performance can one reasonably hope to 
get with gluster? Assume a 3-node cluster running on top of (small) ramdisks on 
a fast and stable network. Is it just a bad fit for our workload?

/Andreas

-



The information contained in this e-mail and in any attachments is confidential 
and is designated solely for the attention of the intended recipient(s). If you 
are not an intended recipient, you must not use, disclose, copy, distribute or 
retain this e-mail or any part thereof. If you have received this e-mail in 
error, please notify the sender by return e-mail and delete all copies of this 
e-mail from your computer system(s). Please direct any additional queries to: 
communicati...@s3group.com<mailto:communicati...@s3group.com>. Thank You. 
Silicon and Software Systems Limited (S3 Group). Registered in Ireland no. 
378073. Registered Office: South County Business Park, Leopardstown, Dublin 18.


___

Re: [Gluster-users] Expected performance for WORM scenario

[Pranith Kumar Karampuri]

On Tue, Mar 13, 2018 at 12:58 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek <
> ondrej.valou...@s3group.com> wrote:
>
>> Hi,
>>
>> Gluster will never perform well for small files.
>>
>> I believe there is  nothing you can do with this.
>>
>
> It is bad compared to a disk filesystem but I believe it is much closer to
> NFS now.
>
> Andreas,
>  Looking at your workload, I am suspecting there to be lot of LOOKUPs
> which reduce performance. Is it possible to do the following?
>
> # gluster volume profile  info incremental
> #execute your workload
> # gluster volume profile  info incremental >
> /path/to/file/that/you/need/to/send/us
>
> If the last line in there is LOOKUP, mostly we need to enable nl-cache
> feature and see how it performs.
>

Please attach this as extra information along with what Nitya asked in the
previous mail.


>
>
>> Ondrej
>>
>>
>>
>> *From:* gluster-users-boun...@gluster.org [mailto:gluster-users-bounces@
>> gluster.org] *On Behalf Of *Andreas Ericsson
>> *Sent:* Monday, March 12, 2018 1:47 PM
>> *To:* Gluster-users@gluster.org
>> *Subject:* [Gluster-users] Expected performance for WORM scenario
>>
>>
>>
>> Heya fellas.
>>
>>
>>
>> I've been struggling quite a lot to get glusterfs to perform even
>> halfdecently with a write-intensive workload. Testnumbers are from gluster
>> 3.10.7.
>>
>>
>>
>> We store a bunch of small files in a doubly-tiered sha1 hash fanout
>> directory structure. The directories themselves aren't overly full. Most of
>> the data we write to gluster is "write once, read probably never", so 99%
>> of all operations are of the write variety.
>>
>>
>>
>> The network between servers is sound. 10gb network cards run over a 10gb
>> (doh) switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 -
>> 0.2 ms. There is no firewall, no packet inspection and no nothing between
>> the servers, and the 10gb switch is the only path between the two machines,
>> so traffic isn't going over some 2mbit wifi by accident.
>>
>>
>>
>> Our main storage has always been really slow (write speed of roughly
>> 1.5MiB/s), but I had long attributed that to the extremely slow disks we
>> use to back it, so now that we're expanding I set up a new gluster cluster
>> with state of the art NVMe SSD drives to boost performance. However,
>> performance only hopped up to around 2.1MiB/s. Perplexed, I tried it first
>> with a 3-node cluster using 2GB ramdrives, which got me up to 2.4MiB/s. My
>> last resort was to use a single node running on ramdisk, just to 100%
>> exclude any network shenanigans, but the write performance stayed at an
>> absolutely abysmal 3MiB/s.
>>
>>
>>
>> Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I
>> don't actually remember the numbers, but my test that took 2 minutes with
>> gluster completed before I had time to blink). Writing straight to the
>> backing SSD drives gives me a throughput of 96MiB/sec.
>>
>>
>>
>> The test itself writes 8494 files that I simply took randomly from our
>> production environment, comprising a total of 63.4MiB (so average file size
>> is just under 8k. Most are actually close to 4k though, with the occasional
>> 2-or-so MB file in there.
>>
>>
>>
>> I have googled and read a *lot* of performance-tuning guides, but the
>> 3MiB/sec on single-node ramdisk seems to be far beyond the crippling one
>> can cause by misconfiguration of a single system.
>>
>>
>>
>> With this in mind; What sort of write performance can one reasonably hope
>> to get with gluster? Assume a 3-node cluster running on top of (small)
>> ramdisks on a fast and stable network. Is it just a bad fit for our
>> workload?
>>
>>
>>
>> /Andreas
>>
>> -
>>
>> The information contained in this e-mail and in any attachments is 
>> confidential and is designated solely for the attention of the intended 
>> recipient(s). If you are not an intended recipient, you must not use, 
>> disclose, copy, distribute or retain this e-mail or any part thereof. If you 
>> have received this e-mail in error, please notify the sender by return 
>> e-mail and delete all copies of this e-mail from your computer system(s). 
>> Please direct any additional queries to: communicati...@s3group.com. Thank 
>> You. Silicon and Software Systems Limited (S3 Group). Registered in Ireland 
>> no. 378073. Registered Office: South County Business Park, Leopardstown, 
>> Dublin 18.
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
> --
> Pranith
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Expected performance for WORM scenario

[Nithya Balachandran]

Hi,

Can you send us the following details:
1. gluster volume info
2. What client you are using to run this?

Thanks,
Nithya

On 12 March 2018 at 18:16, Andreas Ericsson 
wrote:

> Heya fellas.
>
> I've been struggling quite a lot to get glusterfs to perform even
> halfdecently with a write-intensive workload. Testnumbers are from gluster
> 3.10.7.
>
> We store a bunch of small files in a doubly-tiered sha1 hash fanout
> directory structure. The directories themselves aren't overly full. Most of
> the data we write to gluster is "write once, read probably never", so 99%
> of all operations are of the write variety.
>
> The network between servers is sound. 10gb network cards run over a 10gb
> (doh) switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 -
> 0.2 ms. There is no firewall, no packet inspection and no nothing between
> the servers, and the 10gb switch is the only path between the two machines,
> so traffic isn't going over some 2mbit wifi by accident.
>
> Our main storage has always been really slow (write speed of roughly
> 1.5MiB/s), but I had long attributed that to the extremely slow disks we
> use to back it, so now that we're expanding I set up a new gluster cluster
> with state of the art NVMe SSD drives to boost performance. However,
> performance only hopped up to around 2.1MiB/s. Perplexed, I tried it first
> with a 3-node cluster using 2GB ramdrives, which got me up to 2.4MiB/s. My
> last resort was to use a single node running on ramdisk, just to 100%
> exclude any network shenanigans, but the write performance stayed at an
> absolutely abysmal 3MiB/s.
>
> Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I
> don't actually remember the numbers, but my test that took 2 minutes with
> gluster completed before I had time to blink). Writing straight to the
> backing SSD drives gives me a throughput of 96MiB/sec.
>
> The test itself writes 8494 files that I simply took randomly from our
> production environment, comprising a total of 63.4MiB (so average file size
> is just under 8k. Most are actually close to 4k though, with the occasional
> 2-or-so MB file in there.
>
> I have googled and read a *lot* of performance-tuning guides, but the
> 3MiB/sec on single-node ramdisk seems to be far beyond the crippling one
> can cause by misconfiguration of a single system.
>
> With this in mind; What sort of write performance can one reasonably hope
> to get with gluster? Assume a 3-node cluster running on top of (small)
> ramdisks on a fast and stable network. Is it just a bad fit for our
> workload?
>
> /Andreas
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Expected performance for WORM scenario

[Ondrej Valousek]

Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.
Ondrej

From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traffic isn't 
going over some 2mbit wifi by accident.

Our main storage has always been really slow (write speed of roughly 1.5MiB/s), 
but I had long attributed that to the extremely slow disks we use to back it, 
so now that we're expanding I set up a new gluster cluster with state of the 
art NVMe SSD drives to boost performance. However, performance only hopped up 
to around 2.1MiB/s. Perplexed, I tried it first with a 3-node cluster using 2GB 
ramdrives, which got me up to 2.4MiB/s. My last resort was to use a single node 
running on ramdisk, just to 100% exclude any network shenanigans, but the write 
performance stayed at an absolutely abysmal 3MiB/s.

Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I don't 
actually remember the numbers, but my test that took 2 minutes with gluster 
completed before I had time to blink). Writing straight to the backing SSD 
drives gives me a throughput of 96MiB/sec.

The test itself writes 8494 files that I simply took randomly from our 
production environment, comprising a total of 63.4MiB (so average file size is 
just under 8k. Most are actually close to 4k though, with the occasional 
2-or-so MB file in there.

I have googled and read a *lot* of performance-tuning guides, but the 3MiB/sec 
on single-node ramdisk seems to be far beyond the crippling one can cause by 
misconfiguration of a single system.

With this in mind; What sort of write performance can one reasonably hope to 
get with gluster? Assume a 3-node cluster running on top of (small) ramdisks on 
a fast and stable network. Is it just a bad fit for our workload?

/Andreas

-

The information contained in this e-mail and in any attachments is confidential 
and is designated solely for the attention of the intended recipient(s). If you 
are not an intended recipient, you must not use, disclose, copy, distribute or 
retain this e-mail or any part thereof. If you have received this e-mail in 
error, please notify the sender by return e-mail and delete all copies of this 
e-mail from your computer system(s). Please direct any additional queries to: 
communicati...@s3group.com. Thank You. Silicon and Software Systems Limited (S3 
Group). Registered in Ireland no. 378073. Registered Office: South County 
Business Park, Leopardstown, Dublin 18.___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Expected performance for WORM scenario

[Andreas Ericsson]

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even
halfdecently with a write-intensive workload. Testnumbers are from gluster
3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout
directory structure. The directories themselves aren't overly full. Most of
the data we write to gluster is "write once, read probably never", so 99%
of all operations are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb
(doh) switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 -
0.2 ms. There is no firewall, no packet inspection and no nothing between
the servers, and the 10gb switch is the only path between the two machines,
so traffic isn't going over some 2mbit wifi by accident.

Our main storage has always been really slow (write speed of roughly
1.5MiB/s), but I had long attributed that to the extremely slow disks we
use to back it, so now that we're expanding I set up a new gluster cluster
with state of the art NVMe SSD drives to boost performance. However,
performance only hopped up to around 2.1MiB/s. Perplexed, I tried it first
with a 3-node cluster using 2GB ramdrives, which got me up to 2.4MiB/s. My
last resort was to use a single node running on ramdisk, just to 100%
exclude any network shenanigans, but the write performance stayed at an
absolutely abysmal 3MiB/s.

Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I
don't actually remember the numbers, but my test that took 2 minutes with
gluster completed before I had time to blink). Writing straight to the
backing SSD drives gives me a throughput of 96MiB/sec.

The test itself writes 8494 files that I simply took randomly from our
production environment, comprising a total of 63.4MiB (so average file size
is just under 8k. Most are actually close to 4k though, with the occasional
2-or-so MB file in there.

I have googled and read a *lot* of performance-tuning guides, but the
3MiB/sec on single-node ramdisk seems to be far beyond the crippling one
can cause by misconfiguration of a single system.

With this in mind; What sort of write performance can one reasonably hope
to get with gluster? Assume a 3-node cluster running on top of (small)
ramdisks on a fast and stable network. Is it just a bad fit for our
workload?

/Andreas
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users