Re: [Gluster-users] Can't heal a volume: "Please check if all brick processes are running."

2018-03-13 Thread Karthik Subrahmanya 
On Wed, Mar 14, 2018 at 4:33 AM, Laura Bailey  wrote:

> Can we add a smarter error message for this situation by checking volume
> type first?

Yes we can. I will do that.

Thanks,
Karthik

>
> Cheers,
> Laura B
>
>
> On Wednesday, March 14, 2018, Karthik Subrahmanya 
> wrote:
>
>> Hi Anatoliy,
>>
>> The heal command is basically used to heal any mismatching contents
>> between replica copies of the files.
>> For the command "gluster volume heal " to succeed, you should
>> have the self-heal-daemon running,
>> which is true only if your volume is of type replicate/disperse.
>> In your case you have a plain distribute volume where you do not store
>> the replica of any files.
>> So the volume heal will return you the error.
>>
>> Regards,
>> Karthik
>>
>> On Tue, Mar 13, 2018 at 7:53 PM, Anatoliy Dmytriyev 
>> wrote:
>>
>>> Hi,
>>>
>>>
>>> Maybe someone can point me to a documentation or explain this? I can't
>>> find it myself.
>>> Do we have any other useful resources except doc.gluster.org? As I see
>>> many gluster options are not described there or there are no explanation
>>> what is doing...
>>>
>>>
>>>
>>> On 2018-03-12 15:58, Anatoliy Dmytriyev wrote:
>>>
 Hello,

 We have a very fresh gluster 3.10.10 installation.
 Our volume is created as distributed volume, 9 bricks 96TB in total
 (87TB after 10% of gluster disk space reservation)

 For some reasons I can’t “heal” the volume:
 # gluster volume heal gv0
 Launching heal operation to perform index self heal on volume gv0 has
 been unsuccessful on bricks that are down. Please check if all brick
 processes are running.

 Which processes should be run on every brick for heal operation?

 # gluster volume status
 Status of volume: gv0
 Gluster process TCP Port  RDMA Port
 Online  Pid
 
 --
 Brick cn01-ib:/gfs/gv0/brick1/brick 0 49152  Y
  70850
 Brick cn02-ib:/gfs/gv0/brick1/brick 0 49152  Y
  102951
 Brick cn03-ib:/gfs/gv0/brick1/brick 0 49152  Y
  57535
 Brick cn04-ib:/gfs/gv0/brick1/brick 0 49152  Y
  56676
 Brick cn05-ib:/gfs/gv0/brick1/brick 0 49152  Y
  56880
 Brick cn06-ib:/gfs/gv0/brick1/brick 0 49152  Y
  56889
 Brick cn07-ib:/gfs/gv0/brick1/brick 0 49152  Y
  56902
 Brick cn08-ib:/gfs/gv0/brick1/brick 0 49152  Y
  94920
 Brick cn09-ib:/gfs/gv0/brick1/brick 0 49152  Y
  56542

 Task Status of Volume gv0
 
 --
 There are no active volume tasks


 # gluster volume info gv0
 Volume Name: gv0
 Type: Distribute
 Volume ID: 8becaf78-cf2d-4991-93bf-f2446688154f
 Status: Started
 Snapshot Count: 0
 Number of Bricks: 9
 Transport-type: rdma
 Bricks:
 Brick1: cn01-ib:/gfs/gv0/brick1/brick
 Brick2: cn02-ib:/gfs/gv0/brick1/brick
 Brick3: cn03-ib:/gfs/gv0/brick1/brick
 Brick4: cn04-ib:/gfs/gv0/brick1/brick
 Brick5: cn05-ib:/gfs/gv0/brick1/brick
 Brick6: cn06-ib:/gfs/gv0/brick1/brick
 Brick7: cn07-ib:/gfs/gv0/brick1/brick
 Brick8: cn08-ib:/gfs/gv0/brick1/brick
 Brick9: cn09-ib:/gfs/gv0/brick1/brick
 Options Reconfigured:
 client.event-threads: 8
 performance.parallel-readdir: on
 performance.readdir-ahead: on
 cluster.nufa: on
 nfs.disable: on

>>>
>>> --
>>> Best regards,
>>> Anatoliy
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>
> --
> Laura Bailey
> Senior Technical Writer
> Customer Content Services BNE
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Announcing Gluster release 4.0.0 (Short Term Maintenance)

2018-03-13 Thread Shyam Ranganathan 
The Gluster community celebrates 13 years of development with this
latest release, Gluster 4.0. This release enables improved integration
with containers, an enhanced user experience, and a next-generation
management framework. The 4.0 release helps cloud-native app developers
choose Gluster as the default scale-out distributed file system.

We’re highlighting some of the announcements, major features and changes
here, but our release notes[1] have announcements, expanded major
changes and features, and bugs addressed in this release.

Major enhancements:

- Management
GlusterD2 (GD2) is a new management daemon for Gluster-4.0. It is a
complete rewrite, with all new internal core frameworks, that make it
more scalable, easier to integrate with and has lower maintenance
requirements. This replaces GlusterD.

A quick start guide [6] is available to get started with GD2.

Although GD2 is in tech preview for this release, it is ready to use for
forming and managing new clusters.

- Monitoring
With this release, GlusterFS enables a lightweight method to access
internal monitoring information.

- Performance
There are several enhancements to performance in the disperse translator
and in the client side metadata caching layers.

- Other enhancements of note
This release adds: ability to run Gluster on FIPS compliant systems,
ability to force permissions while creating files/directories, and
improved consistency in distributed volumes.

- Developer related
New on-wire protocol version and full type encoding of internal
dictionaries on the wire, Global translator to handle per-daemon
options, improved translator initialization structure, among a few other
improvements, that help streamline development of newer translators.

Release packages (or where to get them) are available at [2] and are
signed with [3]. The upgrade guide for this release can be found at [4]

Related announcements:

- As 3.13 was a short term maintenance release, it will reach end of
life (EOL) with the release of 4.0.0 [5].

- Releases that receive maintenance updates post 4.0 release are, 3.10,
3.12, 4.0 [5].

- With this release, the CentOS storage SIG will not build server
packages for CentOS6. Server packages will be available for CentOS7
only. For ease of migrations, client packages on CentOS6 will be
published and maintained, as announced here [7].

References:
[1] Release notes:
https://docs.gluster.org/en/latest/release-notes/4.0.0.md/
[2] Packages: https://download.gluster.org/pub/gluster/glusterfs/4.0/
[3] Packages signed with:
https://download.gluster.org/pub/gluster/glusterfs/4.0/rsa.pub
[4] Upgrade guide:
https://docs.gluster.org/en/latest/Upgrade-Guide/upgrade_to_4.0/
[5] Release schedule: https://www.gluster.org/release-schedule/
[6] GD2 quick start:
https://github.com/gluster/glusterd2/blob/master/doc/quick-start-user-guide.md
[7] CentOS Storage SIG CentOS6 support announcement:
http://lists.gluster.org/pipermail/gluster-users/2018-January/033212.html
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Can't heal a volume: "Please check if all brick processes are running."

2018-03-13 Thread Laura Bailey 
Can we add a smarter error message for this situation by checking volume
type first?

Cheers,
Laura B

On Wednesday, March 14, 2018, Karthik Subrahmanya 
wrote:

> Hi Anatoliy,
>
> The heal command is basically used to heal any mismatching contents
> between replica copies of the files.
> For the command "gluster volume heal " to succeed, you should
> have the self-heal-daemon running,
> which is true only if your volume is of type replicate/disperse.
> In your case you have a plain distribute volume where you do not store the
> replica of any files.
> So the volume heal will return you the error.
>
> Regards,
> Karthik
>
> On Tue, Mar 13, 2018 at 7:53 PM, Anatoliy Dmytriyev 
> wrote:
>
>> Hi,
>>
>>
>> Maybe someone can point me to a documentation or explain this? I can't
>> find it myself.
>> Do we have any other useful resources except doc.gluster.org? As I see
>> many gluster options are not described there or there are no explanation
>> what is doing...
>>
>>
>>
>> On 2018-03-12 15:58, Anatoliy Dmytriyev wrote:
>>
>>> Hello,
>>>
>>> We have a very fresh gluster 3.10.10 installation.
>>> Our volume is created as distributed volume, 9 bricks 96TB in total
>>> (87TB after 10% of gluster disk space reservation)
>>>
>>> For some reasons I can’t “heal” the volume:
>>> # gluster volume heal gv0
>>> Launching heal operation to perform index self heal on volume gv0 has
>>> been unsuccessful on bricks that are down. Please check if all brick
>>> processes are running.
>>>
>>> Which processes should be run on every brick for heal operation?
>>>
>>> # gluster volume status
>>> Status of volume: gv0
>>> Gluster process TCP Port  RDMA Port  Online
>>> Pid
>>> 
>>> --
>>> Brick cn01-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>>  70850
>>> Brick cn02-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>>  102951
>>> Brick cn03-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>>  57535
>>> Brick cn04-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>>  56676
>>> Brick cn05-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>>  56880
>>> Brick cn06-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>>  56889
>>> Brick cn07-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>>  56902
>>> Brick cn08-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>>  94920
>>> Brick cn09-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>>  56542
>>>
>>> Task Status of Volume gv0
>>> 
>>> --
>>> There are no active volume tasks
>>>
>>>
>>> # gluster volume info gv0
>>> Volume Name: gv0
>>> Type: Distribute
>>> Volume ID: 8becaf78-cf2d-4991-93bf-f2446688154f
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 9
>>> Transport-type: rdma
>>> Bricks:
>>> Brick1: cn01-ib:/gfs/gv0/brick1/brick
>>> Brick2: cn02-ib:/gfs/gv0/brick1/brick
>>> Brick3: cn03-ib:/gfs/gv0/brick1/brick
>>> Brick4: cn04-ib:/gfs/gv0/brick1/brick
>>> Brick5: cn05-ib:/gfs/gv0/brick1/brick
>>> Brick6: cn06-ib:/gfs/gv0/brick1/brick
>>> Brick7: cn07-ib:/gfs/gv0/brick1/brick
>>> Brick8: cn08-ib:/gfs/gv0/brick1/brick
>>> Brick9: cn09-ib:/gfs/gv0/brick1/brick
>>> Options Reconfigured:
>>> client.event-threads: 8
>>> performance.parallel-readdir: on
>>> performance.readdir-ahead: on
>>> cluster.nufa: on
>>> nfs.disable: on
>>>
>>
>> --
>> Best regards,
>> Anatoliy
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>

-- 
Laura Bailey
Senior Technical Writer
Customer Content Services BNE
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Can't heal a volume: "Please check if all brick processes are running."

2018-03-13 Thread Karthik Subrahmanya 
Hi Anatoliy,

The heal command is basically used to heal any mismatching contents between
replica copies of the files.
For the command "gluster volume heal " to succeed, you should have
the self-heal-daemon running,
which is true only if your volume is of type replicate/disperse.
In your case you have a plain distribute volume where you do not store the
replica of any files.
So the volume heal will return you the error.

Regards,
Karthik

On Tue, Mar 13, 2018 at 7:53 PM, Anatoliy Dmytriyev 
wrote:

> Hi,
>
>
> Maybe someone can point me to a documentation or explain this? I can't
> find it myself.
> Do we have any other useful resources except doc.gluster.org? As I see
> many gluster options are not described there or there are no explanation
> what is doing...
>
>
>
> On 2018-03-12 15:58, Anatoliy Dmytriyev wrote:
>
>> Hello,
>>
>> We have a very fresh gluster 3.10.10 installation.
>> Our volume is created as distributed volume, 9 bricks 96TB in total
>> (87TB after 10% of gluster disk space reservation)
>>
>> For some reasons I can’t “heal” the volume:
>> # gluster volume heal gv0
>> Launching heal operation to perform index self heal on volume gv0 has
>> been unsuccessful on bricks that are down. Please check if all brick
>> processes are running.
>>
>> Which processes should be run on every brick for heal operation?
>>
>> # gluster volume status
>> Status of volume: gv0
>> Gluster process TCP Port  RDMA Port  Online
>> Pid
>> 
>> --
>> Brick cn01-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>  70850
>> Brick cn02-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>  102951
>> Brick cn03-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>  57535
>> Brick cn04-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>  56676
>> Brick cn05-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>  56880
>> Brick cn06-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>  56889
>> Brick cn07-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>  56902
>> Brick cn08-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>  94920
>> Brick cn09-ib:/gfs/gv0/brick1/brick 0 49152  Y
>>  56542
>>
>> Task Status of Volume gv0
>> 
>> --
>> There are no active volume tasks
>>
>>
>> # gluster volume info gv0
>> Volume Name: gv0
>> Type: Distribute
>> Volume ID: 8becaf78-cf2d-4991-93bf-f2446688154f
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 9
>> Transport-type: rdma
>> Bricks:
>> Brick1: cn01-ib:/gfs/gv0/brick1/brick
>> Brick2: cn02-ib:/gfs/gv0/brick1/brick
>> Brick3: cn03-ib:/gfs/gv0/brick1/brick
>> Brick4: cn04-ib:/gfs/gv0/brick1/brick
>> Brick5: cn05-ib:/gfs/gv0/brick1/brick
>> Brick6: cn06-ib:/gfs/gv0/brick1/brick
>> Brick7: cn07-ib:/gfs/gv0/brick1/brick
>> Brick8: cn08-ib:/gfs/gv0/brick1/brick
>> Brick9: cn09-ib:/gfs/gv0/brick1/brick
>> Options Reconfigured:
>> client.event-threads: 8
>> performance.parallel-readdir: on
>> performance.readdir-ahead: on
>> cluster.nufa: on
>> nfs.disable: on
>>
>
> --
> Best regards,
> Anatoliy
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] trashcan on dist. repl. volume with geo-replication

2018-03-13 Thread Dietmar Putz 

Hi Kotresh,

...another test. this time the trashcan was enabled on master only. as 
in the test before it's a gfs 3.12.6 on ubuntu 16.04.4
the geo rep error appeared again and disabling the trashcan does not 
change anything.
as in the former test the error appears when i try to list files in the 
trashcan.
the shown gfid belongs to a directory in trashcan with just one file in 
it...like in the former test.


[2018-03-13 11:08:30.777489] E [master(/brick1/mvol1):784:log_failures] 
_GMaster: ENTRY FAILED  data=({'uid': 0, 'gfid': 
'71379ee0-c40a-49db-b3ed-9f3145ed409a', 'gid': 0, 'mode': 16877, 
'entry': '.gfid/4f59c068-6c77-40f2-b556-aa761834caf1/dir1', 'op': 
'MKDIR'}, 2, {'gfid_mismatch': False, 'dst': False})


below the setup, further informations and all activities.
is there anything else i could test or check...?

a generally question, is there a recommendation for the use of the 
trashcan feature in geo-replication envrionments...?
for my use-case it's not necessary to activate it on the slave...but is 
this needed to activate it on master and slave ?


best regards

Dietmar


master volume :
root@gl-node1:~# gluster volume info mvol1

Volume Name: mvol1
Type: Distributed-Replicate
Volume ID: 7590b6a0-520b-4c51-ad63-3ba5be0ed0df
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gl-node1-int:/brick1/mvol1
Brick2: gl-node2-int:/brick1/mvol1
Brick3: gl-node3-int:/brick1/mvol1
Brick4: gl-node4-int:/brick1/mvol1
Options Reconfigured:
changelog.changelog: on
geo-replication.ignore-pid-check: on
geo-replication.indexing: on
features.trash-max-filesize: 2GB
features.trash: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
root@gl-node1:~#


slave volume :
root@gl-node5:~# gluster volume info mvol1

Volume Name: mvol1
Type: Distributed-Replicate
Volume ID: aba4e057-7374-4a62-bcd7-c1c6f71e691b
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gl-node5-int:/brick1/mvol1
Brick2: gl-node6-int:/brick1/mvol1
Brick3: gl-node7-int:/brick1/mvol1
Brick4: gl-node8-int:/brick1/mvol1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
root@gl-node5:~#

root@gl-node1:~# gluster volume geo-replication mvol1 
gl-node5-int::mvol1 config

special_sync_mode: partial
state_socket_unencoded: 
/var/lib/glusterd/geo-replication/mvol1_gl-node5-int_mvol1/ssh%3A%2F%2Froot%40192.168.178.65%3Agluster%3A%2F%2F127.0.0.1%3Amvol1.socket
gluster_log_file: 
/var/log/glusterfs/geo-replication/mvol1/ssh%3A%2F%2Froot%40192.168.178.65%3Agluster%3A%2F%2F127.0.0.1%3Amvol1.gluster.log
ssh_command: ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no 
-i /var/lib/glusterd/geo-replication/secret.pem

ignore_deletes: false
change_detector: changelog
gluster_command_dir: /usr/sbin/
state_file: 
/var/lib/glusterd/geo-replication/mvol1_gl-node5-int_mvol1/monitor.status

remote_gsyncd: /nonexistent/gsyncd
log_file: 
/var/log/glusterfs/geo-replication/mvol1/ssh%3A%2F%2Froot%40192.168.178.65%3Agluster%3A%2F%2F127.0.0.1%3Amvol1.log
changelog_log_file: 
/var/log/glusterfs/geo-replication/mvol1/ssh%3A%2F%2Froot%40192.168.178.65%3Agluster%3A%2F%2F127.0.0.1%3Amvol1-changes.log

socketdir: /var/run/gluster
working_dir: 
/var/lib/misc/glusterfsd/mvol1/ssh%3A%2F%2Froot%40192.168.178.65%3Agluster%3A%2F%2F127.0.0.1%3Amvol1
state_detail_file: 
/var/lib/glusterd/geo-replication/mvol1_gl-node5-int_mvol1/ssh%3A%2F%2Froot%40192.168.178.65%3Agluster%3A%2F%2F127.0.0.1%3Amvol1-detail.status

use_meta_volume: true
ssh_command_tar: ssh -oPasswordAuthentication=no 
-oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/tar_ssh.pem
pid_file: 
/var/lib/glusterd/geo-replication/mvol1_gl-node5-int_mvol1/monitor.pid
georep_session_working_dir: 
/var/lib/glusterd/geo-replication/mvol1_gl-node5-int_mvol1/

access_mount: true
gluster_params: aux-gfid-mount acl
root@gl-node1:~#

root@gl-node1:~# gluster volume geo-replication mvol1 
gl-node5-int::mvol1 status


MASTER NODE MASTER VOL    MASTER BRICK SLAVE USER 
SLAVE  SLAVE NODE  STATUS CRAWL STATUS   
LAST_SYNCED


gl-node1-int    mvol1 /brick1/mvol1    root 
gl-node5-int::mvol1    gl-node5-int    Active Changelog Crawl    
2018-03-13 09:43:46
gl-node4-int    mvol1 /brick1/mvol1    root 
gl-node5-int::mvol1    gl-node8-int    Active Changelog Crawl    
2018-03-13 09:43:47
gl-node2-int    mvol1 /brick1/mvol1    root 
gl-node5-int::mvol1    gl-node6-int    Passive N/A    N/A
gl-node3-int    mvol1 /brick1/mvol1    root 
gl-node5-int::mvol1    gl-node7-int    Passive N/A    N/A

root@gl-node1:~#

volume's are locally mounted as :

gl-node1:/mvol1 20G   65M   20G 1% /m_vol

Re: [Gluster-users] Can't heal a volume: "Please check if all brick processes are running."

2018-03-13 Thread Anatoliy Dmytriyev 

Hi,


Maybe someone can point me to a documentation or explain this? I can't 
find it myself.
Do we have any other useful resources except doc.gluster.org? As I see 
many gluster options are not described there or there are no explanation 
what is doing...



On 2018-03-12 15:58, Anatoliy Dmytriyev wrote:

Hello,

We have a very fresh gluster 3.10.10 installation.
Our volume is created as distributed volume, 9 bricks 96TB in total
(87TB after 10% of gluster disk space reservation)

For some reasons I can’t “heal” the volume:
# gluster volume heal gv0
Launching heal operation to perform index self heal on volume gv0 has
been unsuccessful on bricks that are down. Please check if all brick
processes are running.

Which processes should be run on every brick for heal operation?

# gluster volume status
Status of volume: gv0
Gluster process TCP Port  RDMA Port  Online 
 Pid

--
Brick cn01-ib:/gfs/gv0/brick1/brick 0 49152  Y  
 70850
Brick cn02-ib:/gfs/gv0/brick1/brick 0 49152  Y  
 102951
Brick cn03-ib:/gfs/gv0/brick1/brick 0 49152  Y  
 57535
Brick cn04-ib:/gfs/gv0/brick1/brick 0 49152  Y  
 56676
Brick cn05-ib:/gfs/gv0/brick1/brick 0 49152  Y  
 56880
Brick cn06-ib:/gfs/gv0/brick1/brick 0 49152  Y  
 56889
Brick cn07-ib:/gfs/gv0/brick1/brick 0 49152  Y  
 56902
Brick cn08-ib:/gfs/gv0/brick1/brick 0 49152  Y  
 94920
Brick cn09-ib:/gfs/gv0/brick1/brick 0 49152  Y  
 56542


Task Status of Volume gv0
--
There are no active volume tasks


# gluster volume info gv0
Volume Name: gv0
Type: Distribute
Volume ID: 8becaf78-cf2d-4991-93bf-f2446688154f
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: rdma
Bricks:
Brick1: cn01-ib:/gfs/gv0/brick1/brick
Brick2: cn02-ib:/gfs/gv0/brick1/brick
Brick3: cn03-ib:/gfs/gv0/brick1/brick
Brick4: cn04-ib:/gfs/gv0/brick1/brick
Brick5: cn05-ib:/gfs/gv0/brick1/brick
Brick6: cn06-ib:/gfs/gv0/brick1/brick
Brick7: cn07-ib:/gfs/gv0/brick1/brick
Brick8: cn08-ib:/gfs/gv0/brick1/brick
Brick9: cn09-ib:/gfs/gv0/brick1/brick
Options Reconfigured:
client.event-threads: 8
performance.parallel-readdir: on
performance.readdir-ahead: on
cluster.nufa: on
nfs.disable: on


--
Best regards,
Anatoliy
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Expected performance for WORM scenario

2018-03-13 Thread Ondrej Valousek 
Sorry - no time to play with that. But it’s simple to reproduce, just set up 
your of async nfs server, take my script and you will see on your own.
Ondrej

From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, March 13, 2018 10:41 AM
To: Ondrej Valousek 
Cc: Andreas Ericsson ; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Tue, Mar 13, 2018 at 2:42 PM, Ondrej Valousek 
> wrote:
Yes, I have had this in place already (well except of the negative cache, but 
enabling that did not make much effect).
To me, this is no surprise – nothing can match nfs performance for small files 
for obvious reasons:

Could you give profile info of the run you did with and without nl-cache? 
Please also provide your volume info output.


1.   Single server, does not have to deal with distributed locks

2.   Afaik, gluster does not support read/write delegations the same way 
NFS does.

3.   Glusterfs is FUSE based
Glusterfs supports NFS/SMB/fuse

4.   Glusterfs does not support async writes
It supports async writes. It has a feature called write-behind which does that.


Summary: If you do not need to scale out, stick with a single server (+DRBD 
optionally for HA), it will give you the best performance

Ondrej


From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com]
Sent: Tuesday, March 13, 2018 9:10 AM

To: Ondrej Valousek 
>
Cc: Andreas Ericsson 
>; 
Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Tue, Mar 13, 2018 at 1:37 PM, Ondrej Valousek 
> wrote:
Well, it might be close to the _synchronous_ nfs, but it is still well behind 
of the asynchronous nfs performance.
Simple script (bit extreme I know, but helps to draw the picture):

#!/bin/csh

set HOSTNAME=`/bin/hostname`
set j=1
while ($j <= 7000)
   echo ahoj > test.$HOSTNAME.$j
   @ j++
end
rm -rf test.$HOSTNAME.*


Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS – i.e. 
10 times slower.

Do you have the new features enabled?

performance.stat-prefetch=on
performance.cache-invalidation=on
performance.md-cache-timeout=600
network.inode-lru-limit=5
performance.nl-cache=on
performance.nl-cache-timeout=600
network.inode-lru-limit=5


Ondrej

From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com]
Sent: Tuesday, March 13, 2018 8:28 AM
To: Ondrej Valousek 
>
Cc: Andreas Ericsson 
>; 
Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traffic isn't 
going over some 2mbit wifi by accident.

Our main

Re: [Gluster-users] Expected performance for WORM scenario

2018-03-13 Thread Pranith Kumar Karampuri 
On Tue, Mar 13, 2018 at 2:42 PM, Ondrej Valousek <
ondrej.valou...@s3group.com> wrote:

> Yes, I have had this in place already (well except of the negative cache,
> but enabling that did not make much effect).
>
> To me, this is no surprise – nothing can match nfs performance for small
> files for obvious reasons:
>

Could you give profile info of the run you did with and without nl-cache?
Please also provide your volume info output.


> 1.   Single server, does not have to deal with distributed locks
>
> 2.   Afaik, gluster does not support read/write delegations the same
> way NFS does.
>
> 3.   Glusterfs is FUSE based
>
Glusterfs supports NFS/SMB/fuse

> 4.   Glusterfs does not support async writes
>
It supports async writes. It has a feature called write-behind which does
that.


>
> Summary: If you do not need to scale out, stick with a single server
> (+DRBD optionally for HA), it will give you the best performance
>
>
>
> Ondrej
>
>
>
>
>
> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> *Sent:* Tuesday, March 13, 2018 9:10 AM
>
> *To:* Ondrej Valousek 
> *Cc:* Andreas Ericsson ;
> Gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Expected performance for WORM scenario
>
>
>
>
>
>
>
> On Tue, Mar 13, 2018 at 1:37 PM, Ondrej Valousek <
> ondrej.valou...@s3group.com> wrote:
>
> Well, it might be close to the _*synchronous*_ nfs, but it is still well
> behind of the asynchronous nfs performance.
>
> Simple script (bit extreme I know, but helps to draw the picture):
>
>
>
> #!/bin/csh
>
>
>
> set HOSTNAME=`/bin/hostname`
>
> set j=1
>
> while ($j <= 7000)
>
>echo ahoj > test.$HOSTNAME.$j
>
>@ j++
>
> end
>
> rm -rf test.$HOSTNAME.*
>
>
>
>
>
> Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS –
> i.e. 10 times slower.
>
>
>
> Do you have the new features enabled?
>
>
>
> performance.stat-prefetch=on
> performance.cache-invalidation=on
> performance.md-cache-timeout=600
> network.inode-lru-limit=5
>
> performance.nl-cache=on
> performance.nl-cache-timeout=600
> network.inode-lru-limit=5
>
>
>
>
>
> Ondrej
>
>
>
> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> *Sent:* Tuesday, March 13, 2018 8:28 AM
> *To:* Ondrej Valousek 
> *Cc:* Andreas Ericsson ;
> Gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Expected performance for WORM scenario
>
>
>
>
>
>
>
> On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek <
> ondrej.valou...@s3group.com> wrote:
>
> Hi,
>
> Gluster will never perform well for small files.
>
> I believe there is  nothing you can do with this.
>
>
>
> It is bad compared to a disk filesystem but I believe it is much closer to
> NFS now.
>
>
>
> Andreas,
>
>  Looking at your workload, I am suspecting there to be lot of LOOKUPs
> which reduce performance. Is it possible to do the following?
>
>
>
> # gluster volume profile  info incremental
>
> #execute your workload
>
> # gluster volume profile  info incremental >
> /path/to/file/that/you/need/to/send/us
>
>
>
> If the last line in there is LOOKUP, mostly we need to enable nl-cache
> feature and see how it performs.
>
>
>
> Ondrej
>
>
>
> *From:* gluster-users-boun...@gluster.org [mailto:gluster-users-bounces@
> gluster.org] *On Behalf Of *Andreas Ericsson
> *Sent:* Monday, March 12, 2018 1:47 PM
> *To:* Gluster-users@gluster.org
> *Subject:* [Gluster-users] Expected performance for WORM scenario
>
>
>
> Heya fellas.
>
>
>
> I've been struggling quite a lot to get glusterfs to perform even
> halfdecently with a write-intensive workload. Testnumbers are from gluster
> 3.10.7.
>
>
>
> We store a bunch of small files in a doubly-tiered sha1 hash fanout
> directory structure. The directories themselves aren't overly full. Most of
> the data we write to gluster is "write once, read probably never", so 99%
> of all operations are of the write variety.
>
>
>
> The network between servers is sound. 10gb network cards run over a 10gb
> (doh) switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 -
> 0.2 ms. There is no firewall, no packet inspection and no nothing between
> the servers, and the 10gb switch is the only path between the two machines,
> so traffic isn't going over some 2mbit wifi by accident.
>
>
>
> Our main storage has always been really slow (write speed of roughly
> 1.5MiB/s), but I had long attributed that to the extremely slow disks we
> use to back it, so now that we're expanding I set up a new gluster cluster
> with state of the art NVMe SSD drives to boost performance. However,
> performance only hopped up to around 2.1MiB/s. Perplexed, I tried it first
> with a 3-node cluster using 2GB ramdrives, which got me up to 2.4MiB/s. My
> last resort was to use a single node running on ramdisk, just to 100%
> exclude any network shenanigans, but the write performance stayed at an
> absolutely abysmal

Re: [Gluster-users] Expected performance for WORM scenario

2018-03-13 Thread Ondrej Valousek 
Yes, I have had this in place already (well except of the negative cache, but 
enabling that did not make much effect).
To me, this is no surprise – nothing can match nfs performance for small files 
for obvious reasons:

1.   Single server, does not have to deal with distributed locks

2.   Afaik, gluster does not support read/write delegations the same way 
NFS does.

3.   Glusterfs is FUSE based

4.   Glusterfs does not support async writes

Summary: If you do not need to scale out, stick with a single server (+DRBD 
optionally for HA), it will give you the best performance

Ondrej


From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, March 13, 2018 9:10 AM
To: Ondrej Valousek 
Cc: Andreas Ericsson ; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Tue, Mar 13, 2018 at 1:37 PM, Ondrej Valousek 
> wrote:
Well, it might be close to the _synchronous_ nfs, but it is still well behind 
of the asynchronous nfs performance.
Simple script (bit extreme I know, but helps to draw the picture):

#!/bin/csh

set HOSTNAME=`/bin/hostname`
set j=1
while ($j <= 7000)
   echo ahoj > test.$HOSTNAME.$j
   @ j++
end
rm -rf test.$HOSTNAME.*


Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS – i.e. 
10 times slower.

Do you have the new features enabled?

performance.stat-prefetch=on
performance.cache-invalidation=on
performance.md-cache-timeout=600
network.inode-lru-limit=5
performance.nl-cache=on
performance.nl-cache-timeout=600
network.inode-lru-limit=5


Ondrej

From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com]
Sent: Tuesday, March 13, 2018 8:28 AM
To: Ondrej Valousek 
>
Cc: Andreas Ericsson 
>; 
Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traffic isn't 
going over some 2mbit wifi by accident.

Our main storage has always been really slow (write speed of roughly 1.5MiB/s), 
but I had long attributed that to the extremely slow disks we use to back it, 
so now that we're expanding I set up a new gluster cluster with state of the 
art NVMe SSD drives to boost performance. However, performance only hopped up 
to around 2.1MiB/s. Perplexed, I tried it first with a 3-node cluster using 2GB 
ramdrives, which got me up to 2.4MiB/s. My last resort was to use a single node 
running on ramdisk, just to 100% exclude any network shenanigans, but the write 
performance stayed at an absolutely abysmal 3MiB/s.

Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I don't 
actually remember the numbers, but my test that took 2 minutes with gluster 
completed before I had time to blink). Writing straight to the backing SSD 
drives gives me a throughput of 96MiB/sec.

The test itself writes 8494 files that I simply took randomly from our

Re: [Gluster-users] Expected performance for WORM scenario

2018-03-13 Thread Ondrej Valousek 
Well, it might be close to the _synchronous_ nfs, but it is still well behind 
of the asynchronous nfs performance.
Simple script (bit extreme I know, but helps to draw the picture):

#!/bin/csh

set HOSTNAME=`/bin/hostname`
set j=1
while ($j <= 7000)
   echo ahoj > test.$HOSTNAME.$j
   @ j++
end
rm -rf test.$HOSTNAME.*


Takes 9 seconds to execute on the NFS share, but 90 seconds on GlusterFS – i.e. 
10 times slower.

Ondrej

From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, March 13, 2018 8:28 AM
To: Ondrej Valousek 
Cc: Andreas Ericsson ; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Expected performance for WORM scenario



On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek 
> wrote:
Hi,
Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

It is bad compared to a disk filesystem but I believe it is much closer to NFS 
now.

Andreas,
 Looking at your workload, I am suspecting there to be lot of LOOKUPs which 
reduce performance. Is it possible to do the following?

# gluster volume profile  info incremental
#execute your workload
# gluster volume profile  info incremental > 
/path/to/file/that/you/need/to/send/us

If the last line in there is LOOKUP, mostly we need to enable nl-cache feature 
and see how it performs.

Ondrej

From: 
gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org]
 On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traffic isn't 
going over some 2mbit wifi by accident.

Our main storage has always been really slow (write speed of roughly 1.5MiB/s), 
but I had long attributed that to the extremely slow disks we use to back it, 
so now that we're expanding I set up a new gluster cluster with state of the 
art NVMe SSD drives to boost performance. However, performance only hopped up 
to around 2.1MiB/s. Perplexed, I tried it first with a 3-node cluster using 2GB 
ramdrives, which got me up to 2.4MiB/s. My last resort was to use a single node 
running on ramdisk, just to 100% exclude any network shenanigans, but the write 
performance stayed at an absolutely abysmal 3MiB/s.

Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I don't 
actually remember the numbers, but my test that took 2 minutes with gluster 
completed before I had time to blink). Writing straight to the backing SSD 
drives gives me a throughput of 96MiB/sec.

The test itself writes 8494 files that I simply took randomly from our 
production environment, comprising a total of 63.4MiB (so average file size is 
just under 8k. Most are actually close to 4k though, with the occasional 
2-or-so MB file in there.

I have googled and read a *lot* of performance-tuning guides, but the 3MiB/sec 
on single-node ramdisk seems to be far beyond the crippling one can cause by 
misconfiguration of a single system.

With this in mind; What sort of write performance can one reasonably hope to 
get with gluster? Assume a 3-node cluster running on top of (small) ramdisks on 
a fast and stable network. Is it just a bad fit for our workload?

/Andreas

-



The information contained in this e-mail and in any attachments is confidential 
and is designated solely for the attention of the intended recipient(s). If you 
are not an intended recipient, you must not use, disclose, copy, distribute or 
retain this e-mail or any part thereof. If you have received this e-mail in 
error, please notify the sender by return e-mail and delete all copies of this 
e-mail from your computer system(s). Please direct any additional queries to: 
communicati...@s3group.com. Thank You. 
Silicon and Software Systems Limited (S3 Group). Registered in Ireland no. 
378073. Registered Office: South County Business Park, Leopardstown, Dublin 18.


___
Gluster-users mailing list

Re: [Gluster-users] Expected performance for WORM scenario

2018-03-13 Thread Pranith Kumar Karampuri 
On Tue, Mar 13, 2018 at 12:58 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Mon, Mar 12, 2018 at 6:23 PM, Ondrej Valousek <
> ondrej.valou...@s3group.com> wrote:
>
>> Hi,
>>
>> Gluster will never perform well for small files.
>>
>> I believe there is  nothing you can do with this.
>>
>
> It is bad compared to a disk filesystem but I believe it is much closer to
> NFS now.
>
> Andreas,
>  Looking at your workload, I am suspecting there to be lot of LOOKUPs
> which reduce performance. Is it possible to do the following?
>
> # gluster volume profile  info incremental
> #execute your workload
> # gluster volume profile  info incremental >
> /path/to/file/that/you/need/to/send/us
>
> If the last line in there is LOOKUP, mostly we need to enable nl-cache
> feature and see how it performs.
>

Please attach this as extra information along with what Nitya asked in the
previous mail.


>
>
>> Ondrej
>>
>>
>>
>> *From:* gluster-users-boun...@gluster.org [mailto:gluster-users-bounces@
>> gluster.org] *On Behalf Of *Andreas Ericsson
>> *Sent:* Monday, March 12, 2018 1:47 PM
>> *To:* Gluster-users@gluster.org
>> *Subject:* [Gluster-users] Expected performance for WORM scenario
>>
>>
>>
>> Heya fellas.
>>
>>
>>
>> I've been struggling quite a lot to get glusterfs to perform even
>> halfdecently with a write-intensive workload. Testnumbers are from gluster
>> 3.10.7.
>>
>>
>>
>> We store a bunch of small files in a doubly-tiered sha1 hash fanout
>> directory structure. The directories themselves aren't overly full. Most of
>> the data we write to gluster is "write once, read probably never", so 99%
>> of all operations are of the write variety.
>>
>>
>>
>> The network between servers is sound. 10gb network cards run over a 10gb
>> (doh) switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 -
>> 0.2 ms. There is no firewall, no packet inspection and no nothing between
>> the servers, and the 10gb switch is the only path between the two machines,
>> so traffic isn't going over some 2mbit wifi by accident.
>>
>>
>>
>> Our main storage has always been really slow (write speed of roughly
>> 1.5MiB/s), but I had long attributed that to the extremely slow disks we
>> use to back it, so now that we're expanding I set up a new gluster cluster
>> with state of the art NVMe SSD drives to boost performance. However,
>> performance only hopped up to around 2.1MiB/s. Perplexed, I tried it first
>> with a 3-node cluster using 2GB ramdrives, which got me up to 2.4MiB/s. My
>> last resort was to use a single node running on ramdisk, just to 100%
>> exclude any network shenanigans, but the write performance stayed at an
>> absolutely abysmal 3MiB/s.
>>
>>
>>
>> Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I
>> don't actually remember the numbers, but my test that took 2 minutes with
>> gluster completed before I had time to blink). Writing straight to the
>> backing SSD drives gives me a throughput of 96MiB/sec.
>>
>>
>>
>> The test itself writes 8494 files that I simply took randomly from our
>> production environment, comprising a total of 63.4MiB (so average file size
>> is just under 8k. Most are actually close to 4k though, with the occasional
>> 2-or-so MB file in there.
>>
>>
>>
>> I have googled and read a *lot* of performance-tuning guides, but the
>> 3MiB/sec on single-node ramdisk seems to be far beyond the crippling one
>> can cause by misconfiguration of a single system.
>>
>>
>>
>> With this in mind; What sort of write performance can one reasonably hope
>> to get with gluster? Assume a 3-node cluster running on top of (small)
>> ramdisks on a fast and stable network. Is it just a bad fit for our
>> workload?
>>
>>
>>
>> /Andreas
>>
>> -
>>
>> The information contained in this e-mail and in any attachments is 
>> confidential and is designated solely for the attention of the intended 
>> recipient(s). If you are not an intended recipient, you must not use, 
>> disclose, copy, distribute or retain this e-mail or any part thereof. If you 
>> have received this e-mail in error, please notify the sender by return 
>> e-mail and delete all copies of this e-mail from your computer system(s). 
>> Please direct any additional queries to: communicati...@s3group.com. Thank 
>> You. Silicon and Software Systems Limited (S3 Group). Registered in Ireland 
>> no. 378073. Registered Office: South County Business Park, Leopardstown, 
>> Dublin 18.
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
> --
> Pranith
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users