Re: [Gluster-users] [EXTERNAL] Re: New Gluster volume (10.3) not healing symlinks after brick offline

[Matt Rubright]

Hi Eli,

Thanks for the response. I had hoped for a simple fix here, but I think
perhaps there isn't one. I have built this as a part of a new environment,
eventually replacing a much older system built with Gluster 3.10 (yes -
that old). I appreciate the warning about 10.3 and will run some
comparative load testing against it and 9.5 both.

- Matt

On Fri, Feb 24, 2023 at 8:46 AM Eli V  wrote:

> I've seen issues with symlinks failing to heal as well. I never found
> a good solution on the glusterfs side of things. Most reliable fix I
> found is just rm and recreate the symlink in the fuse volume itself.
> Also, I'd strongly suggest heavy load testing before upgrading to 10.3
> in production, after upgrading from 9.5 -> 10.3 I've seen frequent
> brick process crashes(glusterfsd), whereas 9.5 was quite stable.
>
> On Mon, Jan 23, 2023 at 3:58 PM Matt Rubright  wrote:
> >
> > Hi friends,
> >
> > I have recently built a new replica 3 arbiter 1 volume on 10.3 servers
> and have been putting it through its paces before getting it ready for
> production use. The volume will ultimately contain about 200G of web
> content files shared among multiple frontends. Each will use the gluster
> fuse client to connect.
> >
> > What I am experiencing sounds very much like this post from 9 years ago:
> https://lists.gnu.org/archive/html/gluster-devel/2013-12/msg00103.html
> >
> > In short, if I perform these steps I can reliably end up with symlinks
> on the volume which will not heal either by initiating a 'full heal' from
> the cluster or using a fuse client to read each file:
> >
> > 1) Verify that all nodes are healthy, the volume is healthy, and there
> are no items needing to be healed
> > 2) Cleanly shut down one server hosting a brick
> > 3) Copy data, including some symlinks, from a fuse client to the volume
> > 4) Bring the brick back online and observe the number and type of items
> needing to be healed
> > 5) Initiate a full heal from one of the nodes
> > 6) Confirm that while files and directories are healed, symlinks are not
> >
> > Please help me determine if I have improper expectations here. I have
> some basic knowledge of managing gluster volumes, but I may be
> misunderstanding intended behavior.
> >
> > Here is the volume info and heal data at each step of the way:
> >
> > *** Verify that all nodes are healthy, the volume is healthy, and there
> are no items needing to be healed ***
> >
> > # gluster vol info cwsvol01
> >
> > Volume Name: cwsvol01
> > Type: Replicate
> > Volume ID: 7b28e6e6-4a73-41b7-83fe-863a45fd27fc
> > Status: Started
> > Snapshot Count: 0
> > Number of Bricks: 1 x (2 + 1) = 3
> > Transport-type: tcp
> > Bricks:
> > Brick1: glfs02-172-20-1:/data/brick01/cwsvol01
> > Brick2: glfs01-172-20-1:/data/brick01/cwsvol01
> > Brick3: glfsarb01-172-20-1:/data/arb01/cwsvol01 (arbiter)
> > Options Reconfigured:
> > performance.client-io-threads: off
> > nfs.disable: on
> > transport.address-family: inet
> > storage.fips-mode-rchecksum: on
> > cluster.granular-entry-heal: on
> >
> > # gluster vol status
> > Status of volume: cwsvol01
> > Gluster process TCP Port  RDMA Port  Online
> Pid
> >
> --
> > Brick glfs02-172-20-1:/data/brick01/cwsvol0
> > 1   50253 0  Y
>  1397
> > Brick glfs01-172-20-1:/data/brick01/cwsvol0
> > 1   56111 0  Y
>  1089
> > Brick glfsarb01-172-20-1:/data/arb01/cwsvol
> > 01  54517 0  Y
>  118704
> > Self-heal Daemon on localhost   N/A   N/AY
>  1413
> > Self-heal Daemon on glfs01-172-20-1 N/A   N/AY
>  3490
> > Self-heal Daemon on glfsarb01-172-20-1  N/A   N/AY
>  118720
> >
> > Task Status of Volume cwsvol01
> >
> --
> > There are no active volume tasks
> >
> > # gluster vol heal cwsvol01 info summary
> > Brick glfs02-172-20-1:/data/brick01/cwsvol01
> > Status: Connected
> > Total Number of entries: 0
> > Number of entries in heal pending: 0
> > Number of entries in split-brain: 0
> > Number of entries possibly healing: 0
> >
> > Brick glfs01-172-20-1:/data/brick01/cwsvol01
> > Status: Connected
> > Total Number of entries: 0
> > Number of entries in heal pending: 0
>

[Gluster-users] New Gluster volume (10.3) not healing symlinks after brick offline

[Matt Rubright]

Hi friends,

I have recently built a new replica 3 arbiter 1 volume on 10.3 servers and
have been putting it through its paces before getting it ready for
production use. The volume will ultimately contain about 200G of web
content files shared among multiple frontends. Each will use the gluster
fuse client to connect.

What I am experiencing sounds very much like this post from 9 years ago:
https://lists.gnu.org/archive/html/gluster-devel/2013-12/msg00103.html

In short, if I perform these steps I can reliably end up with symlinks on
the volume which will not heal either by initiating a 'full heal' from the
cluster or using a fuse client to read each file:

1) Verify that all nodes are healthy, the volume is healthy, and there are
no items needing to be healed
2) Cleanly shut down one server hosting a brick
3) Copy data, including some symlinks, from a fuse client to the volume
4) Bring the brick back online and observe the number and type of items
needing to be healed
5) Initiate a full heal from one of the nodes
6) Confirm that while files and directories are healed, symlinks are not

Please help me determine if I have improper expectations here. I have some
basic knowledge of managing gluster volumes, but I may be misunderstanding
intended behavior.

Here is the volume info and heal data at each step of the way:

*** Verify that all nodes are healthy, the volume is healthy, and there are
no items needing to be healed ***

# gluster vol info cwsvol01

Volume Name: cwsvol01
Type: Replicate
Volume ID: 7b28e6e6-4a73-41b7-83fe-863a45fd27fc
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: glfs02-172-20-1:/data/brick01/cwsvol01
Brick2: glfs01-172-20-1:/data/brick01/cwsvol01
Brick3: glfsarb01-172-20-1:/data/arb01/cwsvol01 (arbiter)
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
storage.fips-mode-rchecksum: on
cluster.granular-entry-heal: on

# gluster vol status
Status of volume: cwsvol01
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick glfs02-172-20-1:/data/brick01/cwsvol0
1   50253 0  Y
1397
Brick glfs01-172-20-1:/data/brick01/cwsvol0
1   56111 0  Y
1089
Brick glfsarb01-172-20-1:/data/arb01/cwsvol
01  54517 0  Y
118704
Self-heal Daemon on localhost   N/A   N/AY
1413
Self-heal Daemon on glfs01-172-20-1 N/A   N/AY
3490
Self-heal Daemon on glfsarb01-172-20-1  N/A   N/AY
118720

Task Status of Volume cwsvol01
--
There are no active volume tasks

# gluster vol heal cwsvol01 info summary
Brick glfs02-172-20-1:/data/brick01/cwsvol01
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick glfs01-172-20-1:/data/brick01/cwsvol01
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick glfsarb01-172-20-1:/data/arb01/cwsvol01
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

*** Cleanly shut down one server hosting a brick ***

*** Copy data, including some symlinks, from a fuse client to the volume ***

# gluster vol heal cwsvol01 info summary
Brick glfs02-172-20-1:/data/brick01/cwsvol01
Status: Transport endpoint is not connected
Total Number of entries: -
Number of entries in heal pending: -
Number of entries in split-brain: -
Number of entries possibly healing: -

Brick glfs01-172-20-1:/data/brick01/cwsvol01
Status: Connected
Total Number of entries: 810
Number of entries in heal pending: 810
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick glfsarb01-172-20-1:/data/arb01/cwsvol01
Status: Connected
Total Number of entries: 810
Number of entries in heal pending: 810
Number of entries in split-brain: 0
Number of entries possibly healing: 0

*** Bring the brick back online and observe the number and type of entities
needing to be healed ***

# gluster vol heal cwsvol01 info summary
Brick glfs02-172-20-1:/data/brick01/cwsvol01
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick glfs01-172-20-1:/data/brick01/cwsvol01
Status: Connected
Total Number of entries: 769
Number of entries in heal pending: 769
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick glfsarb01-172-20-1:/data/arb01/cwsvol01
Status: Connected
Total Number of 

Re: [Gluster-users] Unable to create new files or folders using samba and vfs_glusterfs

[Matt Waymack]

I've been using these for a few weeks now without any issues, thank you!

-Original Message-
From: gluster-users-boun...@gluster.org  On 
Behalf Of Matt Waymack
Sent: Thursday, December 27, 2018 10:56 AM
To: Diego Remolina 
Cc: gluster-users@gluster.org List 
Subject: Re: [Gluster-users] Unable to create new files or folders using samba 
and vfs_glusterfs

OK, I'm back from the holiday and updated using the following packages:
libsmbclient-4.8.3-4.el7.0.1.x86_64.rpm  
libwbclient-4.8.3-4.el7.0.1.x86_64.rpm   
samba-4.8.3-4.el7.0.1.x86_64.rpm 
samba-client-4.8.3-4.el7.0.1.x86_64.rpm   
samba-client-libs-4.8.3-4.el7.0.1.x86_64.rpm  
samba-common-4.8.3-4.el7.0.1.noarch.rpm   
samba-common-libs-4.8.3-4.el7.0.1.x86_64.rpm
samba-common-tools-4.8.3-4.el7.0.1.x86_64.rpm
samba-libs-4.8.3-4.el7.0.1.x86_64.rpm
samba-vfs-glusterfs-4.8.3-4.el7.0.1.x86_64.rpm

First impressions are good!  We're able to create files/folders.  I'll keep you 
updated with stability.

Thank you!
-Original Message-
From: Diego Remolina 
Sent: Thursday, December 20, 2018 1:36 PM
To: Matt Waymack 
Cc: gluster-users@gluster.org List 
Subject: Re: [Gluster-users] Unable to create new files or folders using samba 
and vfs_glusterfs

Hi Matt,

The update is slightly different, has the .1 at the end:

Fast-track -> samba-4.8.3-4.el7.0.1.x86_64.rpm

vs general -> samba-4.8.3-4.el7.x86_64

I think these are built, but not pushed to fasttrack repo until they get 
feedback the packages are good. So you may need to use wget to download them 
and update your packages with these for the test.

Diego

On Thu, Dec 20, 2018 at 1:06 PM Matt Waymack  wrote:
>
> Hi all,
>
>
>
> I’m looking to update Samba from fasttrack, but I only still se 4.8.3 and yum 
> is not wanting to update.  The test build is also showing 4.8.3.
>
>
>
> Thank you!
>
>
>
>
>
> From: gluster-users-boun...@gluster.org 
>  On Behalf Of Matt Waymack
> Sent: Sunday, December 16, 2018 1:55 PM
> To: Diego Remolina 
> Cc: gluster-users@gluster.org List 
> Subject: Re: [Gluster-users] Unable to create new files or folders 
> using samba and vfs_glusterfs
>
>
>
> Hi all, sorry for the delayed response.
>
>
>
> I can test this out and will report back.  It may be as late as Tuesday 
> before I can test the build.
>
>
>
> Thank you!
>
>
>
> On Dec 15, 2018 7:46 AM, Diego Remolina  wrote:
>
> Matt,
>
>
>
> Can you test the updated samba packages that the CentOS team has built for 
> FasTrack?
>
>
>
> A NOTE has been added to this issue.
>
> --
>  (0033351) pgreco (developer) - 2018-12-15 13:43
>  https://bugs.centos.org/view.php?id=15586#c33351
> --
> @dijur...@gmail.com
> Here's the link for the test build
> https://buildlogs.centos.org/c7-fasttrack.x86_64/samba/20181214164659/
> 4.8.3-4.el7.0.1.x86_64/
> .
> Please let us know how it goes. Thanks for testing!
> Pablo.
> --
>
> Diego
>
>
>
>
> On Fri, Dec 14, 2018 at 12:52 AM Anoop C S  wrote:
> >
> > On Thu, 2018-12-13 at 15:31 +, Matt Waymack wrote:
> > > Hi all,
> > >
> > > I’m having an issue on Windows clients accessing shares via smb 
> > > when using vfs_glusterfs.  They are unable to create any file or 
> > > folders at the root of the share and get the error “The file is 
> > > too large for the destination file system.”  When I change from 
> > > vfs_glusterfs to just using a filesystem path to the same 
> > > location, it works fine (except for the performance hit).  All my 
> > > searches have led to bug 1619108, and that seems to be the 
> > > symptom, but there doesn’t appear to be any clear resolution.
> >
> > You figured out the right bug and following is the upstream Samba bug:
> >
> > https://bugzilla.samba.org/show_bug.cgi?id=13585
> >
> > Unfortunately it is only available with v4.8.6 and higher. If 
> > required I can patch it up and provide a build.
> >
> > > I’m on the latest version of samba available on CentOS 7 (4.8.3) 
> > > and I’m on the latest available glusterfs 4.1 (4.1.6).  Is there 
> > > something simple I’m missing to get this going?
> > >
> > > Thank you!
> > > ___
> > > Gluster-users mailing list
> > > Gluster-users@gluster.org
> > > https://lists.gluster.org/mailman/listinfo/gluster-users
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Input/output error on FUSE log

[Matt Waymack]

Has anyone any other ideas where to look?  This is only affecting FUSE clients. 
 SMB clients are unaffected by this problem.

Thanks!

From: gluster-users-boun...@gluster.org  On 
Behalf Of Matt Waymack
Sent: Monday, January 7, 2019 1:19 PM
To: Raghavendra Gowdappa 
Cc: gluster-users@gluster.org List 
Subject: Re: [Gluster-users] Input/output error on FUSE log

Attached are the logs from when a failure occurred with diagnostics set to 
trace.

Thank you!

From: Raghavendra Gowdappa mailto:rgowd...@redhat.com>>
Sent: Saturday, January 5, 2019 8:32 PM
To: Matt Waymack mailto:mwaym...@nsgdv.com>>
Cc: gluster-users@gluster.org<mailto:gluster-users@gluster.org> List 
mailto:gluster-users@gluster.org>>
Subject: Re: [Gluster-users] Input/output error on FUSE log



On Sun, Jan 6, 2019 at 7:58 AM Raghavendra Gowdappa 
mailto:rgowd...@redhat.com>> wrote:


On Sun, Jan 6, 2019 at 4:19 AM Matt Waymack 
mailto:mwaym...@nsgdv.com>> wrote:

Hi all,



I'm having a problem writing to our volume.  When writing files larger than 
about 2GB, I get an intermittent issue where the write will fail and return 
Input/Output error.  This is also shown in the FUSE log of the client (this is 
affecting all clients).  A snip of a client log is below:

[2019-01-05 22:39:44.581371] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51040978: WRITE => -1 
gfid=82a0b5c4-7ef3-43c2-ad86-41e16673d7c2 fd=0x7f949839a368 (Input/output error)

[2019-01-05 22:39:44.598392] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51040979: FLUSH() ERR => -1 (Input/output error)

[2019-01-05 22:39:47.420920] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51041266: WRITE => -1 
gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949809b7f8 (Input/output error)

[2019-01-05 22:39:47.433377] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51041267: FLUSH() ERR => -1 (Input/output error)

[2019-01-05 22:39:50.441531] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51041548: WRITE => -1 
gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949839a368 (Input/output error)

[2019-01-05 22:39:50.451914] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51041549: FLUSH() ERR => -1 (Input/output error)

The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: 
no subvolume for hash (value) = 1311504267" repeated 1721 times between 
[2019-01-05 22:39:33.906241] and [2019-01-05 22:39:44.598371]

The message "E [MSGID: 101046] [dht-common.c:1502:dht_lookup_dir_cbk] 
0-gv1-dht: dict is null" repeated 1714 times between [2019-01-05 
22:39:33.925981] and [2019-01-05 22:39:50.451862]

The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: 
no subvolume for hash (value) = 1137142622" repeated 1707 times between 
[2019-01-05 22:39:39.636552] and [2019-01-05 22:39:50.451895]

This looks to be a DHT issue. Some questions:
* Are all subvolumes of DHT up and client is connected to them? Particularly 
the subvolume which contains the file in question.
* Can you get all extended attributes of parent directory of the file from all 
bricks?
* set diagnostics.client-log-level to TRACE, capture these errors again and 
attach the client log file.

I spoke a bit early. dht_writev doesn't search hashed subvolume as its already 
been looked up in lookup. So, these msgs looks to be of a different issue - not 
 writev failure.


This is intermittent for most files, but eventually if a file is large enough 
it will not write.  The workflow is SFTP tot he client which then writes to the 
volume over FUSE.  When files get to a certain point,w e can no longer write to 
them.  The file sizes are different as well, so it's not like they all get to 
the same size and just stop either.  I've ruled out a free space issue, our 
files at their largest are only a few hundred GB and we have tens of terrabytes 
free on each brick.  We are also sharding at 1GB.

I'm not sure where to go from here as the error seems vague and I can only see 
it on the client log.  I'm not seeing these errors on the nodes themselves.  
This is also seen if I mount the volume via FUSE on any of the nodes as well 
and it is only reflected in the FUSE log.

Here is the volume info:
Volume Name: gv1
Type: Distributed-Replicate
Volume ID: 1472cc78-e2a0-4c3f-9571-dab840239b3c
Status: Started
Snapshot Count: 0
Number of Bricks: 8 x (2 + 1) = 24
Transport-type: tcp
Bricks:
Brick1: tpc-glus4:/exp/b1/gv1
Brick2: tpc-glus2:/exp/b1/gv1
Brick3: tpc-arbiter1:/exp/b1/gv1 (arbiter)
Brick4: tpc-glus2:/exp/b2/gv1
Brick5: tpc-glus4:/exp/b2/gv1
Brick6: tpc-arbiter1:/exp/b2/gv1 (arbiter)
Brick7: tpc-glus4:/exp/b3/gv1
Brick8: tpc-glus2:/exp/b3/gv1
Brick9: tpc-arbiter1:/exp/b3/gv1 (arbiter)
Brick10: tpc-glus4:/exp/b4/gv1
Brick11: tpc-glus2:/exp/b4/gv1
Brick12: tpc-arbiter1:/exp/b4/gv1 (arbiter)
Brick13: tpc-glus1:/exp/b5/gv1
Brick14: tpc-glus3:/exp/b5/gv1
Brick15: tpc-arbiter2

Re: [Gluster-users] [External] Re: Input/output error on FUSE log

[Matt Waymack]

Yep, first unmount/remounted, then rebooted clients.  Stopped/started the 
volumes, and rebooted all nodes.

From: Davide Obbi 
Sent: Monday, January 7, 2019 12:47 PM
To: Matt Waymack 
Cc: Raghavendra Gowdappa ; gluster-users@gluster.org List 

Subject: Re: [External] Re: [Gluster-users] Input/output error on FUSE log

i guess you tried already unmounting, stop/star and mounting?

On Mon, Jan 7, 2019 at 7:44 PM Matt Waymack 
mailto:mwaym...@nsgdv.com>> wrote:
Yes, all volumes use sharding.

From: Davide Obbi mailto:davide.o...@booking.com>>
Sent: Monday, January 7, 2019 12:43 PM
To: Matt Waymack mailto:mwaym...@nsgdv.com>>
Cc: Raghavendra Gowdappa mailto:rgowd...@redhat.com>>; 
gluster-users@gluster.org<mailto:gluster-users@gluster.org> List 
mailto:gluster-users@gluster.org>>
Subject: Re: [External] Re: [Gluster-users] Input/output error on FUSE log

are all the volumes being configured with sharding?

On Mon, Jan 7, 2019 at 5:35 PM Matt Waymack 
mailto:mwaym...@nsgdv.com>> wrote:
I think that I can rule out network as I have multiple volumes on the same 
nodes and not all volumes are affected.  Additionally, access via SMB using 
samba-vfs-glusterfs is not affected, even on the same volumes.   This is 
seemingly only affecting the FUSE clients.

From: Davide Obbi mailto:davide.o...@booking.com>>
Sent: Sunday, January 6, 2019 12:26 PM
To: Raghavendra Gowdappa mailto:rgowd...@redhat.com>>
Cc: Matt Waymack mailto:mwaym...@nsgdv.com>>; 
gluster-users@gluster.org<mailto:gluster-users@gluster.org> List 
mailto:gluster-users@gluster.org>>
Subject: Re: [External] Re: [Gluster-users] Input/output error on FUSE log

Hi,

i would start doing some checks like: "(Input/output error)" seems returned by 
the operating system, this happens for instance trying to access a file system 
which is on a device not available so i would check the network connectivity 
between the client to servers  and server to server during the reported time.

Regards
Davide

On Sun, Jan 6, 2019 at 3:32 AM Raghavendra Gowdappa 
mailto:rgowd...@redhat.com>> wrote:


On Sun, Jan 6, 2019 at 7:58 AM Raghavendra Gowdappa 
mailto:rgowd...@redhat.com>> wrote:


On Sun, Jan 6, 2019 at 4:19 AM Matt Waymack 
mailto:mwaym...@nsgdv.com>> wrote:

Hi all,



I'm having a problem writing to our volume.  When writing files larger than 
about 2GB, I get an intermittent issue where the write will fail and return 
Input/Output error.  This is also shown in the FUSE log of the client (this is 
affecting all clients).  A snip of a client log is below:

[2019-01-05 22:39:44.581371] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51040978: WRITE => -1 
gfid=82a0b5c4-7ef3-43c2-ad86-41e16673d7c2 fd=0x7f949839a368 (Input/output error)

[2019-01-05 22:39:44.598392] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51040979: FLUSH() ERR => -1 (Input/output error)

[2019-01-05 22:39:47.420920] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51041266: WRITE => -1 
gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949809b7f8 (Input/output error)

[2019-01-05 22:39:47.433377] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51041267: FLUSH() ERR => -1 (Input/output error)

[2019-01-05 22:39:50.441531] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51041548: WRITE => -1 
gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949839a368 (Input/output error)

[2019-01-05 22:39:50.451914] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51041549: FLUSH() ERR => -1 (Input/output error)

The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: 
no subvolume for hash (value) = 1311504267" repeated 1721 times between 
[2019-01-05 22:39:33.906241] and [2019-01-05 22:39:44.598371]

The message "E [MSGID: 101046] [dht-common.c:1502:dht_lookup_dir_cbk] 
0-gv1-dht: dict is null" repeated 1714 times between [2019-01-05 
22:39:33.925981] and [2019-01-05 22:39:50.451862]

The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: 
no subvolume for hash (value) = 1137142622" repeated 1707 times between 
[2019-01-05 22:39:39.636552] and [2019-01-05 22:39:50.451895]

This looks to be a DHT issue. Some questions:
* Are all subvolumes of DHT up and client is connected to them? Particularly 
the subvolume which contains the file in question.
* Can you get all extended attributes of parent directory of the file from all 
bricks?
* set diagnostics.client-log-level to TRACE, capture these errors again and 
attach the client log file.

I spoke a bit early. dht_writev doesn't search hashed subvolume as its already 
been looked up in lookup. So, these msgs looks to be of a different issue - not 
 writev failure.


This is intermittent for most files, but eventually if a file is large enough 
it will not write.  The workflow is SFTP tot he client which then writes to

Re: [Gluster-users] [External] Re: Input/output error on FUSE log

[Matt Waymack]

Yes, all volumes use sharding.

From: Davide Obbi 
Sent: Monday, January 7, 2019 12:43 PM
To: Matt Waymack 
Cc: Raghavendra Gowdappa ; gluster-users@gluster.org List 

Subject: Re: [External] Re: [Gluster-users] Input/output error on FUSE log

are all the volumes being configured with sharding?

On Mon, Jan 7, 2019 at 5:35 PM Matt Waymack 
mailto:mwaym...@nsgdv.com>> wrote:
I think that I can rule out network as I have multiple volumes on the same 
nodes and not all volumes are affected.  Additionally, access via SMB using 
samba-vfs-glusterfs is not affected, even on the same volumes.   This is 
seemingly only affecting the FUSE clients.

From: Davide Obbi mailto:davide.o...@booking.com>>
Sent: Sunday, January 6, 2019 12:26 PM
To: Raghavendra Gowdappa mailto:rgowd...@redhat.com>>
Cc: Matt Waymack mailto:mwaym...@nsgdv.com>>; 
gluster-users@gluster.org<mailto:gluster-users@gluster.org> List 
mailto:gluster-users@gluster.org>>
Subject: Re: [External] Re: [Gluster-users] Input/output error on FUSE log

Hi,

i would start doing some checks like: "(Input/output error)" seems returned by 
the operating system, this happens for instance trying to access a file system 
which is on a device not available so i would check the network connectivity 
between the client to servers  and server to server during the reported time.

Regards
Davide

On Sun, Jan 6, 2019 at 3:32 AM Raghavendra Gowdappa 
mailto:rgowd...@redhat.com>> wrote:


On Sun, Jan 6, 2019 at 7:58 AM Raghavendra Gowdappa 
mailto:rgowd...@redhat.com>> wrote:


On Sun, Jan 6, 2019 at 4:19 AM Matt Waymack 
mailto:mwaym...@nsgdv.com>> wrote:

Hi all,



I'm having a problem writing to our volume.  When writing files larger than 
about 2GB, I get an intermittent issue where the write will fail and return 
Input/Output error.  This is also shown in the FUSE log of the client (this is 
affecting all clients).  A snip of a client log is below:

[2019-01-05 22:39:44.581371] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51040978: WRITE => -1 
gfid=82a0b5c4-7ef3-43c2-ad86-41e16673d7c2 fd=0x7f949839a368 (Input/output error)

[2019-01-05 22:39:44.598392] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51040979: FLUSH() ERR => -1 (Input/output error)

[2019-01-05 22:39:47.420920] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51041266: WRITE => -1 
gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949809b7f8 (Input/output error)

[2019-01-05 22:39:47.433377] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51041267: FLUSH() ERR => -1 (Input/output error)

[2019-01-05 22:39:50.441531] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51041548: WRITE => -1 
gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949839a368 (Input/output error)

[2019-01-05 22:39:50.451914] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51041549: FLUSH() ERR => -1 (Input/output error)

The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: 
no subvolume for hash (value) = 1311504267" repeated 1721 times between 
[2019-01-05 22:39:33.906241] and [2019-01-05 22:39:44.598371]

The message "E [MSGID: 101046] [dht-common.c:1502:dht_lookup_dir_cbk] 
0-gv1-dht: dict is null" repeated 1714 times between [2019-01-05 
22:39:33.925981] and [2019-01-05 22:39:50.451862]

The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: 
no subvolume for hash (value) = 1137142622" repeated 1707 times between 
[2019-01-05 22:39:39.636552] and [2019-01-05 22:39:50.451895]

This looks to be a DHT issue. Some questions:
* Are all subvolumes of DHT up and client is connected to them? Particularly 
the subvolume which contains the file in question.
* Can you get all extended attributes of parent directory of the file from all 
bricks?
* set diagnostics.client-log-level to TRACE, capture these errors again and 
attach the client log file.

I spoke a bit early. dht_writev doesn't search hashed subvolume as its already 
been looked up in lookup. So, these msgs looks to be of a different issue - not 
 writev failure.


This is intermittent for most files, but eventually if a file is large enough 
it will not write.  The workflow is SFTP tot he client which then writes to the 
volume over FUSE.  When files get to a certain point,w e can no longer write to 
them.  The file sizes are different as well, so it's not like they all get to 
the same size and just stop either.  I've ruled out a free space issue, our 
files at their largest are only a few hundred GB and we have tens of terrabytes 
free on each brick.  We are also sharding at 1GB.

I'm not sure where to go from here as the error seems vague and I can only see 
it on the client log.  I'm not seeing these errors on the nodes themselves.  
This is also seen if I mount the volume via FUSE on any of the nodes as well 
and it is only reflected in the FUSE log.

Here 

Re: [Gluster-users] [External] Re: Input/output error on FUSE log

[Matt Waymack]

I think that I can rule out network as I have multiple volumes on the same 
nodes and not all volumes are affected.  Additionally, access via SMB using 
samba-vfs-glusterfs is not affected, even on the same volumes.   This is 
seemingly only affecting the FUSE clients.

From: Davide Obbi 
Sent: Sunday, January 6, 2019 12:26 PM
To: Raghavendra Gowdappa 
Cc: Matt Waymack ; gluster-users@gluster.org List 

Subject: Re: [External] Re: [Gluster-users] Input/output error on FUSE log

Hi,

i would start doing some checks like: "(Input/output error)" seems returned by 
the operating system, this happens for instance trying to access a file system 
which is on a device not available so i would check the network connectivity 
between the client to servers  and server to server during the reported time.

Regards
Davide

On Sun, Jan 6, 2019 at 3:32 AM Raghavendra Gowdappa 
mailto:rgowd...@redhat.com>> wrote:


On Sun, Jan 6, 2019 at 7:58 AM Raghavendra Gowdappa 
mailto:rgowd...@redhat.com>> wrote:


On Sun, Jan 6, 2019 at 4:19 AM Matt Waymack 
mailto:mwaym...@nsgdv.com>> wrote:

Hi all,



I'm having a problem writing to our volume.  When writing files larger than 
about 2GB, I get an intermittent issue where the write will fail and return 
Input/Output error.  This is also shown in the FUSE log of the client (this is 
affecting all clients).  A snip of a client log is below:

[2019-01-05 22:39:44.581371] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51040978: WRITE => -1 
gfid=82a0b5c4-7ef3-43c2-ad86-41e16673d7c2 fd=0x7f949839a368 (Input/output error)

[2019-01-05 22:39:44.598392] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51040979: FLUSH() ERR => -1 (Input/output error)

[2019-01-05 22:39:47.420920] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51041266: WRITE => -1 
gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949809b7f8 (Input/output error)

[2019-01-05 22:39:47.433377] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51041267: FLUSH() ERR => -1 (Input/output error)

[2019-01-05 22:39:50.441531] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51041548: WRITE => -1 
gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949839a368 (Input/output error)

[2019-01-05 22:39:50.451914] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51041549: FLUSH() ERR => -1 (Input/output error)

The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: 
no subvolume for hash (value) = 1311504267" repeated 1721 times between 
[2019-01-05 22:39:33.906241] and [2019-01-05 22:39:44.598371]

The message "E [MSGID: 101046] [dht-common.c:1502:dht_lookup_dir_cbk] 
0-gv1-dht: dict is null" repeated 1714 times between [2019-01-05 
22:39:33.925981] and [2019-01-05 22:39:50.451862]

The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: 
no subvolume for hash (value) = 1137142622" repeated 1707 times between 
[2019-01-05 22:39:39.636552] and [2019-01-05 22:39:50.451895]

This looks to be a DHT issue. Some questions:
* Are all subvolumes of DHT up and client is connected to them? Particularly 
the subvolume which contains the file in question.
* Can you get all extended attributes of parent directory of the file from all 
bricks?
* set diagnostics.client-log-level to TRACE, capture these errors again and 
attach the client log file.

I spoke a bit early. dht_writev doesn't search hashed subvolume as its already 
been looked up in lookup. So, these msgs looks to be of a different issue - not 
 writev failure.


This is intermittent for most files, but eventually if a file is large enough 
it will not write.  The workflow is SFTP tot he client which then writes to the 
volume over FUSE.  When files get to a certain point,w e can no longer write to 
them.  The file sizes are different as well, so it's not like they all get to 
the same size and just stop either.  I've ruled out a free space issue, our 
files at their largest are only a few hundred GB and we have tens of terrabytes 
free on each brick.  We are also sharding at 1GB.

I'm not sure where to go from here as the error seems vague and I can only see 
it on the client log.  I'm not seeing these errors on the nodes themselves.  
This is also seen if I mount the volume via FUSE on any of the nodes as well 
and it is only reflected in the FUSE log.

Here is the volume info:
Volume Name: gv1
Type: Distributed-Replicate
Volume ID: 1472cc78-e2a0-4c3f-9571-dab840239b3c
Status: Started
Snapshot Count: 0
Number of Bricks: 8 x (2 + 1) = 24
Transport-type: tcp
Bricks:
Brick1: tpc-glus4:/exp/b1/gv1
Brick2: tpc-glus2:/exp/b1/gv1
Brick3: tpc-arbiter1:/exp/b1/gv1 (arbiter)
Brick4: tpc-glus2:/exp/b2/gv1
Brick5: tpc-glus4:/exp/b2/gv1
Brick6: tpc-arbiter1:/exp/b2/gv1 (arbiter)
Brick7: tpc-glus4:/exp/b3/gv1
Brick8: tpc-glus2:/exp/b3/gv1
Brick9: tpc-arbiter1:/exp/b3/gv1 (arbiter)
Brick10: tpc-glus4:/exp/b4/gv1
Brick11: tpc-glus2:/ex

[Gluster-users] Input/output error on FUSE log

[Matt Waymack]

Hi all,


I'm having a problem writing to our volume.  When writing files larger than 
about 2GB, I get an intermittent issue where the write will fail and return 
Input/Output error.  This is also shown in the FUSE log of the client (this is 
affecting all clients).  A snip of a client log is below:

[2019-01-05 22:39:44.581371] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51040978: WRITE => -1 
gfid=82a0b5c4-7ef3-43c2-ad86-41e16673d7c2 fd=0x7f949839a368 (Input/output error)

[2019-01-05 22:39:44.598392] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51040979: FLUSH() ERR => -1 (Input/output error)

[2019-01-05 22:39:47.420920] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51041266: WRITE => -1 
gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949809b7f8 (Input/output error)

[2019-01-05 22:39:47.433377] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51041267: FLUSH() ERR => -1 (Input/output error)

[2019-01-05 22:39:50.441531] W [fuse-bridge.c:2474:fuse_writev_cbk] 
0-glusterfs-fuse: 51041548: WRITE => -1 
gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949839a368 (Input/output error)

[2019-01-05 22:39:50.451914] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 51041549: FLUSH() ERR => -1 (Input/output error)

The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: 
no subvolume for hash (value) = 1311504267" repeated 1721 times between 
[2019-01-05 22:39:33.906241] and [2019-01-05 22:39:44.598371]

The message "E [MSGID: 101046] [dht-common.c:1502:dht_lookup_dir_cbk] 
0-gv1-dht: dict is null" repeated 1714 times between [2019-01-05 
22:39:33.925981] and [2019-01-05 22:39:50.451862]

The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: 
no subvolume for hash (value) = 1137142622" repeated 1707 times between 
[2019-01-05 22:39:39.636552] and [2019-01-05 22:39:50.451895]

This is intermittent for most files, but eventually if a file is large enough 
it will not write.  The workflow is SFTP tot he client which then writes to the 
volume over FUSE.  When files get to a certain point,w e can no longer write to 
them.  The file sizes are different as well, so it's not like they all get to 
the same size and just stop either.  I've ruled out a free space issue, our 
files at their largest are only a few hundred GB and we have tens of terrabytes 
free on each brick.  We are also sharding at 1GB.

I'm not sure where to go from here as the error seems vague and I can only see 
it on the client log.  I'm not seeing these errors on the nodes themselves.  
This is also seen if I mount the volume via FUSE on any of the nodes as well 
and it is only reflected in the FUSE log.

Here is the volume info:
Volume Name: gv1
Type: Distributed-Replicate
Volume ID: 1472cc78-e2a0-4c3f-9571-dab840239b3c
Status: Started
Snapshot Count: 0
Number of Bricks: 8 x (2 + 1) = 24
Transport-type: tcp
Bricks:
Brick1: tpc-glus4:/exp/b1/gv1
Brick2: tpc-glus2:/exp/b1/gv1
Brick3: tpc-arbiter1:/exp/b1/gv1 (arbiter)
Brick4: tpc-glus2:/exp/b2/gv1
Brick5: tpc-glus4:/exp/b2/gv1
Brick6: tpc-arbiter1:/exp/b2/gv1 (arbiter)
Brick7: tpc-glus4:/exp/b3/gv1
Brick8: tpc-glus2:/exp/b3/gv1
Brick9: tpc-arbiter1:/exp/b3/gv1 (arbiter)
Brick10: tpc-glus4:/exp/b4/gv1
Brick11: tpc-glus2:/exp/b4/gv1
Brick12: tpc-arbiter1:/exp/b4/gv1 (arbiter)
Brick13: tpc-glus1:/exp/b5/gv1
Brick14: tpc-glus3:/exp/b5/gv1
Brick15: tpc-arbiter2:/exp/b5/gv1 (arbiter)
Brick16: tpc-glus1:/exp/b6/gv1
Brick17: tpc-glus3:/exp/b6/gv1
Brick18: tpc-arbiter2:/exp/b6/gv1 (arbiter)
Brick19: tpc-glus1:/exp/b7/gv1
Brick20: tpc-glus3:/exp/b7/gv1
Brick21: tpc-arbiter2:/exp/b7/gv1 (arbiter)
Brick22: tpc-glus1:/exp/b8/gv1
Brick23: tpc-glus3:/exp/b8/gv1
Brick24: tpc-arbiter2:/exp/b8/gv1 (arbiter)
Options Reconfigured:
performance.cache-samba-metadata: on
performance.cache-invalidation: off
features.shard-block-size: 1000MB
features.shard: on
transport.address-family: inet
nfs.disable: on
cluster.lookup-optimize: on

I'm a bit stumped on this, any help is appreciated.  Thank you!

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Unable to create new files or folders using samba and vfs_glusterfs

[Matt Waymack]

OK, I'm back from the holiday and updated using the following packages:
libsmbclient-4.8.3-4.el7.0.1.x86_64.rpm  
libwbclient-4.8.3-4.el7.0.1.x86_64.rpm   
samba-4.8.3-4.el7.0.1.x86_64.rpm 
samba-client-4.8.3-4.el7.0.1.x86_64.rpm   
samba-client-libs-4.8.3-4.el7.0.1.x86_64.rpm  
samba-common-4.8.3-4.el7.0.1.noarch.rpm   
samba-common-libs-4.8.3-4.el7.0.1.x86_64.rpm
samba-common-tools-4.8.3-4.el7.0.1.x86_64.rpm
samba-libs-4.8.3-4.el7.0.1.x86_64.rpm
samba-vfs-glusterfs-4.8.3-4.el7.0.1.x86_64.rpm

First impressions are good!  We're able to create files/folders.  I'll keep you 
updated with stability.

Thank you!
-Original Message-
From: Diego Remolina  
Sent: Thursday, December 20, 2018 1:36 PM
To: Matt Waymack 
Cc: gluster-users@gluster.org List 
Subject: Re: [Gluster-users] Unable to create new files or folders using samba 
and vfs_glusterfs

Hi Matt,

The update is slightly different, has the .1 at the end:

Fast-track -> samba-4.8.3-4.el7.0.1.x86_64.rpm

vs general -> samba-4.8.3-4.el7.x86_64

I think these are built, but not pushed to fasttrack repo until they get 
feedback the packages are good. So you may need to use wget to download them 
and update your packages with these for the test.

Diego

On Thu, Dec 20, 2018 at 1:06 PM Matt Waymack  wrote:
>
> Hi all,
>
>
>
> I’m looking to update Samba from fasttrack, but I only still se 4.8.3 and yum 
> is not wanting to update.  The test build is also showing 4.8.3.
>
>
>
> Thank you!
>
>
>
>
>
> From: gluster-users-boun...@gluster.org 
>  On Behalf Of Matt Waymack
> Sent: Sunday, December 16, 2018 1:55 PM
> To: Diego Remolina 
> Cc: gluster-users@gluster.org List 
> Subject: Re: [Gluster-users] Unable to create new files or folders 
> using samba and vfs_glusterfs
>
>
>
> Hi all, sorry for the delayed response.
>
>
>
> I can test this out and will report back.  It may be as late as Tuesday 
> before I can test the build.
>
>
>
> Thank you!
>
>
>
> On Dec 15, 2018 7:46 AM, Diego Remolina  wrote:
>
> Matt,
>
>
>
> Can you test the updated samba packages that the CentOS team has built for 
> FasTrack?
>
>
>
> A NOTE has been added to this issue.
>
> --
>  (0033351) pgreco (developer) - 2018-12-15 13:43
>  https://bugs.centos.org/view.php?id=15586#c33351
> --
> @dijur...@gmail.com
> Here's the link for the test build
> https://buildlogs.centos.org/c7-fasttrack.x86_64/samba/20181214164659/
> 4.8.3-4.el7.0.1.x86_64/
> .
> Please let us know how it goes. Thanks for testing!
> Pablo.
> --
>
> Diego
>
>
>
>
> On Fri, Dec 14, 2018 at 12:52 AM Anoop C S  wrote:
> >
> > On Thu, 2018-12-13 at 15:31 +, Matt Waymack wrote:
> > > Hi all,
> > >
> > > I’m having an issue on Windows clients accessing shares via smb 
> > > when using vfs_glusterfs.  They are unable to create any file or 
> > > folders at the root of the share and get the error “The file is 
> > > too large for the destination file system.”  When I change from 
> > > vfs_glusterfs to just using a filesystem path to the same 
> > > location, it works fine (except for the performance hit).  All my 
> > > searches have led to bug 1619108, and that seems to be the 
> > > symptom, but there doesn’t appear to be any clear resolution.
> >
> > You figured out the right bug and following is the upstream Samba bug:
> >
> > https://bugzilla.samba.org/show_bug.cgi?id=13585
> >
> > Unfortunately it is only available with v4.8.6 and higher. If 
> > required I can patch it up and provide a build.
> >
> > > I’m on the latest version of samba available on CentOS 7 (4.8.3) 
> > > and I’m on the latest available glusterfs 4.1 (4.1.6).  Is there 
> > > something simple I’m missing to get this going?
> > >
> > > Thank you!
> > > ___
> > > Gluster-users mailing list
> > > Gluster-users@gluster.org
> > > https://lists.gluster.org/mailman/listinfo/gluster-users
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Unable to create new files or folders using samba and vfs_glusterfs

[Matt Waymack]

Hi all,

I'm looking to update Samba from fasttrack, but I only still se 4.8.3 and yum 
is not wanting to update.  The test build is also showing 4.8.3.

Thank you!


From: gluster-users-boun...@gluster.org  On 
Behalf Of Matt Waymack
Sent: Sunday, December 16, 2018 1:55 PM
To: Diego Remolina 
Cc: gluster-users@gluster.org List 
Subject: Re: [Gluster-users] Unable to create new files or folders using samba 
and vfs_glusterfs

Hi all, sorry for the delayed response.

I can test this out and will report back.  It may be as late as Tuesday before 
I can test the build.

Thank you!

On Dec 15, 2018 7:46 AM, Diego Remolina 
mailto:dijur...@gmail.com>> wrote:
Matt,

Can you test the updated samba packages that the CentOS team has built for 
FasTrack?

A NOTE has been added to this issue.

--
 (0033351) pgreco (developer) - 2018-12-15 13:43
 https://bugs.centos.org/view.php?id=15586#c33351
--
@dijur...@gmail.com<mailto:dijur...@gmail.com>
Here's the link for the test build
https://buildlogs.centos.org/c7-fasttrack.x86_64/samba/20181214164659/4.8.3-4.el7.0.1.x86_64/
.
Please let us know how it goes. Thanks for testing!
Pablo.
--
Diego



On Fri, Dec 14, 2018 at 12:52 AM Anoop C S 
mailto:anoo...@cryptolab.net>> wrote:
>
> On Thu, 2018-12-13 at 15:31 +, Matt Waymack wrote:
> > Hi all,
> >
> > I'm having an issue on Windows clients accessing shares via smb when
> > using vfs_glusterfs.  They are unable to create any file or folders
> > at the root of the share and get the error "The file is too large for
> > the destination file system."  When I change from vfs_glusterfs to
> > just using a filesystem path to the same location, it works fine
> > (except for the performance hit).  All my searches have led to bug
> > 1619108, and that seems to be the symptom, but there doesn't appear
> > to be any clear resolution.
>
> You figured out the right bug and following is the upstream Samba bug:
>
> https://bugzilla.samba.org/show_bug.cgi?id=13585
>
> Unfortunately it is only available with v4.8.6 and higher. If required
> I can patch it up and provide a build.
>
> > I'm on the latest version of samba available on CentOS 7 (4.8.3) and
> > I'm on the latest available glusterfs 4.1 (4.1.6).  Is there
> > something simple I'm missing to get this going?
> >
> > Thank you!
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
> > https://lists.gluster.org/mailman/listinfo/gluster-users
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Unable to create new files or folders using samba and vfs_glusterfs

[Matt Waymack]

Hi all, sorry for the delayed response.

I can test this out and will report back.  It may be as late as Tuesday before 
I can test the build.

Thank you!

On Dec 15, 2018 7:46 AM, Diego Remolina  wrote:
Matt,

Can you test the updated samba packages that the CentOS team has built for 
FasTrack?

A NOTE has been added to this issue.

--
 (0033351) pgreco (developer) - 2018-12-15 13:43
 https://bugs.centos.org/view.php?id=15586#c33351
--
@dijur...@gmail.com<mailto:dijur...@gmail.com>
Here's the link for the test build
https://buildlogs.centos.org/c7-fasttrack.x86_64/samba/20181214164659/4.8.3-4.el7.0.1.x86_64/
.
Please let us know how it goes. Thanks for testing!
Pablo.
--

Diego



On Fri, Dec 14, 2018 at 12:52 AM Anoop C S 
mailto:anoo...@cryptolab.net>> wrote:
>
> On Thu, 2018-12-13 at 15:31 +, Matt Waymack wrote:
> > Hi all,
> >
> > I'm having an issue on Windows clients accessing shares via smb when
> > using vfs_glusterfs.  They are unable to create any file or folders
> > at the root of the share and get the error "The file is too large for
> > the destination file system."  When I change from vfs_glusterfs to
> > just using a filesystem path to the same location, it works fine
> > (except for the performance hit).  All my searches have led to bug
> > 1619108, and that seems to be the symptom, but there doesn't appear
> > to be any clear resolution.
>
> You figured out the right bug and following is the upstream Samba bug:
>
> https://bugzilla.samba.org/show_bug.cgi?id=13585
>
> Unfortunately it is only available with v4.8.6 and higher. If required
> I can patch it up and provide a build.
>
> > I'm on the latest version of samba available on CentOS 7 (4.8.3) and
> > I'm on the latest available glusterfs 4.1 (4.1.6).  Is there
> > something simple I'm missing to get this going?
> >
> > Thank you!
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
> > https://lists.gluster.org/mailman/listinfo/gluster-users
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Unable to create new files or folders using samba and vfs_glusterfs

[Matt Waymack]

Hi all,

I'm having an issue on Windows clients accessing shares via smb when using 
vfs_glusterfs.  They are unable to create any file or folders at the root of 
the share and get the error "The file is too large for the destination file 
system."  When I change from vfs_glusterfs to just using a filesystem path to 
the same location, it works fine (except for the performance hit).  All my 
searches have led to bug 1619108, and that seems to be the symptom, but there 
doesn't appear to be any clear resolution.  I'm on the latest version of samba 
available on CentOS 7 (4.8.3) and I'm on the latest available glusterfs 4.1 
(4.1.6).  Is there something simple I'm missing to get this going?

Thank you!
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Snapshot size

[matt]

Hi list,

I was wondering if anyone knows if it's possible to get the size of a 
snapshot, ideally a list, but I'd take the size of just one?


I'm aware that you can use lvs and the Data% column gives you an idea.  
But it's not really very neat, so I was wondering if anyone knows of a 
better way?


Cheers,

Matt

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] How to make sure self-heal backlog is empty ?

[Matt Waymack]

Mine also has a list of files that seemingly never heal.  They are usually 
isolated on my arbiter bricks, but not always.  I would also like to find an 
answer for this behavior.

-Original Message-
From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Hoggins!
Sent: Tuesday, December 19, 2017 12:26 PM
To: gluster-users 
Subject: [Gluster-users] How to make sure self-heal backlog is empty ?

Hello list,

I'm not sure what to look for here, not sure if what I'm seeing is the actual 
"backlog" (that we need to make sure is empty while performing a rolling 
upgrade before going to the next node), how can I tell, while reading this, if 
it's okay to reboot / upgrade my next node in the pool ?
Here is what I do for checking :

for i in `gluster volume list`; do gluster volume heal $i info; done

And here is what I get :

Brick ngluster-1.network.hoggins.fr:/export/brick/clem
Status: Connected
Number of entries: 0

Brick ngluster-2.network.hoggins.fr:/export/brick/clem
Status: Connected
Number of entries: 0

Brick ngluster-3.network.hoggins.fr:/export/brick/clem
Status: Connected
Number of entries: 0

Brick ngluster-1.network.hoggins.fr:/export/brick/mailer
Status: Connected
Number of entries: 0

Brick ngluster-2.network.hoggins.fr:/export/brick/mailer
Status: Connected
Number of entries: 0

Brick ngluster-3.network.hoggins.fr:/export/brick/mailer

Status: Connected
Number of entries: 1

Brick ngluster-1.network.hoggins.fr:/export/brick/rom
Status: Connected
Number of entries: 0

Brick ngluster-2.network.hoggins.fr:/export/brick/rom
Status: Connected
Number of entries: 0

Brick ngluster-3.network.hoggins.fr:/export/brick/rom

Status: Connected
Number of entries: 1

Brick ngluster-1.network.hoggins.fr:/export/brick/thedude
Status: Connected
Number of entries: 0

Brick ngluster-2.network.hoggins.fr:/export/brick/thedude

Status: Connected
Number of entries: 1

Brick ngluster-3.network.hoggins.fr:/export/brick/thedude
Status: Connected
Number of entries: 0

Brick ngluster-1.network.hoggins.fr:/export/brick/web
Status: Connected
Number of entries: 0

Brick ngluster-2.network.hoggins.fr:/export/brick/web



Status: Connected
Number of entries: 3

Brick ngluster-3.network.hoggins.fr:/export/brick/web











Status: Connected
Number of entries: 11


Should I be worrying with this never ending ?

    Thank you,

        Hoggins!

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Production Volume will not start

[Matt Waymack]

Hi thank you for the reply.  Ultimately the volume did eventually start after 
about 1.5 hours from the volume start command.  Could it have something to do 
with the amount of files on the volume?

From: Atin Mukherjee [mailto:amukh...@redhat.com]
Sent: Monday, December 18, 2017 1:26 AM
To: Matt Waymack <mwaym...@nsgdv.com>
Cc: gluster-users <Gluster-users@gluster.org>
Subject: Re: [Gluster-users] Production Volume will not start



On Sat, Dec 16, 2017 at 12:45 AM, Matt Waymack 
<mwaym...@nsgdv.com<mailto:mwaym...@nsgdv.com>> wrote:

Hi all,



I have an issue where our volume will not start from any node.  When attempting 
to start the volume it will eventually return:

Error: Request timed out



For some time after that, the volume is locked and we either have to wait or 
restart Gluster services.  In the gluserd.log, it shows the following:



[2017-12-15 18:00:12.423478] I [glusterd-utils.c:5926:glusterd_brick_start] 
0-management: starting a fresh brick process for brick /exp/b1/gv0

[2017-12-15 18:03:12.673885] I 
[glusterd-locks.c:729:gd_mgmt_v3_unlock_timer_cbk] 0-management: In 
gd_mgmt_v3_unlock_timer_cbk

[2017-12-15 18:06:34.304868] I [MSGID: 106499] 
[glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: 
Received status volume req for volume gv0

[2017-12-15 18:06:34.306603] E [MSGID: 106301] 
[glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 
'Volume Status' failed on localhost : Volume gv0 is not started

[2017-12-15 18:11:39.412700] I [glusterd-utils.c:5926:glusterd_brick_start] 
0-management: starting a fresh brick process for brick /exp/b2/gv0

[2017-12-15 18:11:42.405966] I [MSGID: 106143] 
[glusterd-pmap.c:280:pmap_registry_bind] 0-pmap: adding brick /exp/b2/gv0 on 
port 49153

[2017-12-15 18:11:42.406415] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-management: setting frame-timeout to 600

[2017-12-15 18:11:42.406669] I [glusterd-utils.c:5926:glusterd_brick_start] 
0-management: starting a fresh brick process for brick /exp/b3/gv0

[2017-12-15 18:14:39.737192] I 
[glusterd-locks.c:729:gd_mgmt_v3_unlock_timer_cbk] 0-management: In 
gd_mgmt_v3_unlock_timer_cbk

[2017-12-15 18:35:20.856849] I [MSGID: 106143] 
[glusterd-pmap.c:280:pmap_registry_bind] 0-pmap: adding brick /exp/b1/gv0 on 
port 49152

[2017-12-15 18:35:20.857508] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-management: setting frame-timeout to 600

[2017-12-15 18:35:20.858277] I [glusterd-utils.c:5926:glusterd_brick_start] 
0-management: starting a fresh brick process for brick /exp/b4/gv0

[2017-12-15 18:46:07.953995] I [MSGID: 106143] 
[glusterd-pmap.c:280:pmap_registry_bind] 0-pmap: adding brick /exp/b3/gv0 on 
port 49154

[2017-12-15 18:46:07.954432] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-management: setting frame-timeout to 600

[2017-12-15 18:46:07.971355] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-snapd: setting frame-timeout to 600

[2017-12-15 18:46:07.989392] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-nfs: setting frame-timeout to 600

[2017-12-15 18:46:07.989543] I [MSGID: 106132] 
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped

[2017-12-15 18:46:07.989562] I [MSGID: 106568] 
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped

[2017-12-15 18:46:07.989575] I [MSGID: 106600] 
[glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so 
xlator is not installed

[2017-12-15 18:46:07.989601] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-glustershd: setting frame-timeout to 600

[2017-12-15 18:46:08.003011] I [MSGID: 106132] 
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already 
stopped

[2017-12-15 18:46:08.003039] I [MSGID: 106568] 
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is 
stopped

[2017-12-15 18:46:08.003079] I [MSGID: 106567] 
[glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd 
service

[2017-12-15 18:46:09.005173] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-quotad: setting frame-timeout to 600

[2017-12-15 18:46:09.005569] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-bitd: setting frame-timeout to 600

[2017-12-15 18:46:09.005673] I [MSGID: 106132] 
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped

[2017-12-15 18:46:09.005689] I [MSGID: 106568] 
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is 
stopped

[2017-12-15 18:46:09.005712] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-scrub: setting frame-timeout to 600

[2017-12-15 18:46:09.005892] I [MSGID: 106132] 
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped

[2017-12-15 18:46:09.005912] I [MSGID: 106568] 
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is 
stopped

[2017-12-15 18:46:09.026559] I [socket.c:3672:socket_submit_reply] 
0-socket.management: not connected (priv->c

[Gluster-users] Production Volume will not start

[Matt Waymack]

Hi all,

I have an issue where our volume will not start from any node.  When attempting 
to start the volume it will eventually return:

Error: Request timed out

For some time after that, the volume is locked and we either have to wait or 
restart Gluster services.  In the gluserd.log, it shows the following:

[2017-12-15 18:00:12.423478] I [glusterd-utils.c:5926:glusterd_brick_start] 
0-management: starting a fresh brick process for brick /exp/b1/gv0
[2017-12-15 18:03:12.673885] I 
[glusterd-locks.c:729:gd_mgmt_v3_unlock_timer_cbk] 0-management: In 
gd_mgmt_v3_unlock_timer_cbk
[2017-12-15 18:06:34.304868] I [MSGID: 106499] 
[glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: 
Received status volume req for volume gv0
[2017-12-15 18:06:34.306603] E [MSGID: 106301] 
[glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 
'Volume Status' failed on localhost : Volume gv0 is not started
[2017-12-15 18:11:39.412700] I [glusterd-utils.c:5926:glusterd_brick_start] 
0-management: starting a fresh brick process for brick /exp/b2/gv0
[2017-12-15 18:11:42.405966] I [MSGID: 106143] 
[glusterd-pmap.c:280:pmap_registry_bind] 0-pmap: adding brick /exp/b2/gv0 on 
port 49153
[2017-12-15 18:11:42.406415] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-management: setting frame-timeout to 600
[2017-12-15 18:11:42.406669] I [glusterd-utils.c:5926:glusterd_brick_start] 
0-management: starting a fresh brick process for brick /exp/b3/gv0
[2017-12-15 18:14:39.737192] I 
[glusterd-locks.c:729:gd_mgmt_v3_unlock_timer_cbk] 0-management: In 
gd_mgmt_v3_unlock_timer_cbk
[2017-12-15 18:35:20.856849] I [MSGID: 106143] 
[glusterd-pmap.c:280:pmap_registry_bind] 0-pmap: adding brick /exp/b1/gv0 on 
port 49152
[2017-12-15 18:35:20.857508] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-management: setting frame-timeout to 600
[2017-12-15 18:35:20.858277] I [glusterd-utils.c:5926:glusterd_brick_start] 
0-management: starting a fresh brick process for brick /exp/b4/gv0
[2017-12-15 18:46:07.953995] I [MSGID: 106143] 
[glusterd-pmap.c:280:pmap_registry_bind] 0-pmap: adding brick /exp/b3/gv0 on 
port 49154
[2017-12-15 18:46:07.954432] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-management: setting frame-timeout to 600
[2017-12-15 18:46:07.971355] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-snapd: setting frame-timeout to 600
[2017-12-15 18:46:07.989392] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-nfs: setting frame-timeout to 600
[2017-12-15 18:46:07.989543] I [MSGID: 106132] 
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-12-15 18:46:07.989562] I [MSGID: 106568] 
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-12-15 18:46:07.989575] I [MSGID: 106600] 
[glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so 
xlator is not installed
[2017-12-15 18:46:07.989601] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-glustershd: setting frame-timeout to 600
[2017-12-15 18:46:08.003011] I [MSGID: 106132] 
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already 
stopped
[2017-12-15 18:46:08.003039] I [MSGID: 106568] 
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is 
stopped
[2017-12-15 18:46:08.003079] I [MSGID: 106567] 
[glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd 
service
[2017-12-15 18:46:09.005173] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-quotad: setting frame-timeout to 600
[2017-12-15 18:46:09.005569] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-bitd: setting frame-timeout to 600
[2017-12-15 18:46:09.005673] I [MSGID: 106132] 
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-12-15 18:46:09.005689] I [MSGID: 106568] 
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is 
stopped
[2017-12-15 18:46:09.005712] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 
0-scrub: setting frame-timeout to 600
[2017-12-15 18:46:09.005892] I [MSGID: 106132] 
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-12-15 18:46:09.005912] I [MSGID: 106568] 
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is 
stopped
[2017-12-15 18:46:09.026559] I [socket.c:3672:socket_submit_reply] 
0-socket.management: not connected (priv->connected = -1)
[2017-12-15 18:46:09.026568] E [rpcsvc.c:1364:rpcsvc_submit_generic] 
0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, 
ProgVers: 2, Proc: 27) to rpc-transport (socket.management)
[2017-12-15 18:46:09.026582] E [MSGID: 106430] 
[glusterd-utils.c:568:glusterd_submit_reply] 0-glusterd: Reply submission failed
[2017-12-15 18:56:17.962251] E [rpc-clnt.c:185:call_bail] 0-management: bailing 
out frame type(glusterd mgmt v3) op(--(4)) xid = 0x14 sent = 2017-12-15 
18:46:09.005976. timeout = 600 for 10.17.100.208:24007
[2017-12-15 18:56:17.962324] E [MSGID: 

Re: [Gluster-users] gfid entries in volume heal info that do not heal

[Matt Waymack]

In my case I was able to delete the hard links in the .glusterfs folders of the 
bricks and it seems to have done the trick, thanks!

From: Karthik Subrahmanya [mailto:ksubr...@redhat.com]
Sent: Monday, October 23, 2017 1:52 AM
To: Jim Kinney <jim.kin...@gmail.com>; Matt Waymack <mwaym...@nsgdv.com>
Cc: gluster-users <Gluster-users@gluster.org>
Subject: Re: [Gluster-users] gfid entries in volume heal info that do not heal

Hi Jim & Matt,
Can you also check for the link count in the stat output of those hardlink 
entries in the .glusterfs folder on the bricks.
If the link count is 1 on all the bricks for those entries, then they are 
orphaned entries and you can delete those hardlinks.
To be on the safer side have a backup before deleting any of the entries.
Regards,
Karthik

On Fri, Oct 20, 2017 at 3:18 AM, Jim Kinney 
<jim.kin...@gmail.com<mailto:jim.kin...@gmail.com>> wrote:
I've been following this particular thread as I have a similar issue (RAID6 
array failed out with 3 dead drives at once while a 12 TB load was being copied 
into one mounted space - what a mess)

I have >700K GFID entries that have no path data:
Example:
getfattr -d -e hex -m . .glusterfs/00/00/a5ef-5af7-401b-84b5-ff2a51c10421
# file: .glusterfs/00/00/a5ef-5af7-401b-84b5-ff2a51c10421
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.bit-rot.version=0x020059b1b316000270e7
trusted.gfid=0xa5ef5af7401b84b5ff2a51c10421

[root@bmidata1<mailto:root@bmidata1> brick]# getfattr -d -n 
trusted.glusterfs.pathinfo -e hex -m . 
.glusterfs/00/00/a5ef-5af7-401b-84b5-ff2a51c10421
.glusterfs/00/00/a5ef-5af7-401b-84b5-ff2a51c10421: 
trusted.glusterfs.pathinfo: No such attribute

I had to totally rebuild the dead RAID array and did a copy from the live one 
before activating gluster on the rebuilt system. I accidentally copied over the 
.glusterfs folder from the working side
(replica 2 only for now - adding arbiter node as soon as I can get this one 
cleaned up).

I've run the methods from 
"http://docs.gluster.org/en/latest/Troubleshooting/gfid-to-path/; with no 
results using random GFIDs. A full systemic run using the script from method 3 
crashes with "too many nested links" error (or something similar).

When I run gluster volume heal volname info, I get 700K+ GFIDs. Oh. gluster 
3.8.4 on Centos 7.3

Should I just remove the contents of the .glusterfs folder on both and restart 
gluster and run a ls/stat on every file?


When I run a heal, it no longer has a decreasing number of files to heal so 
that's an improvement over the last 2-3 weeks :-)

On Tue, 2017-10-17 at 14:34 +, Matt Waymack wrote:

Attached is the heal log for the volume as well as the shd log.







Run these commands on all the bricks of the replica pair to get the attrs set 
on the backend.







[root@tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m . 
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2

getfattr: Removing leading '/' from absolute path names

# file: exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2

security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000

trusted.afr.dirty=0x

trusted.afr.gv0-client-2=0x0001

trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2

trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466332d346463622d393630322d3839356136396461363131662f435f564f4c2d623030312d693637342d63642d63772e6d6435



[root@tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m . 
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2

getfattr: Removing leading '/' from absolute path names

# file: exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2

security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000

trusted.afr.dirty=0x

trusted.afr.gv0-client-2=0x0001

trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2

trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466332d346463622d393630322d3839356136396461363131662f435f564f4c2d623030312d693637342d63642d63772e6d6435



[root@tpc-arbiter1-100617 ~]# getfattr -d -e hex -m . 
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2

getfattr: /exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2: No 
such file or directory





[root@tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m . 
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3

getfattr: Removing leading '/' from absolute path names

# file: exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3

security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000

trusted.afr.dirty=0x

trusted.afr.gv0-client-11=0x0001

trusted.gfid=0xe0c56bf78b

Re: [Gluster-users] gfid entries in volume heal info that do not heal

[Matt Waymack]

It looks like these entries don't have a corresponding file path, they exist 
only in .glusters and appear to be orphaned:

[root@tpc-cent-glus2-081017 ~]# find /exp/b4/gv0 -samefile 
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3

[root@tpc-cent-glus2-081017 ~]# find /exp/b4/gv0 -samefile 
/exp/b4/gv0/.glusterfs/6f/0a/6f0a0549-8669-46de-8823-d6677fdca8e3
/exp/b4/gv0/.glusterfs/6f/0a/6f0a0549-8669-46de-8823-d6677fdca8e3

[root@tpc-cent-glus1-081017 ~]# find /exp/b1/gv0 -samefile 
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2

[root@tpc-cent-glus1-081017 ~]# find /exp/b4/gv0 -samefile 
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3

Occasionally I would get these gfid entries in the heal info output but would 
just run tree against the volume.  After that the files would trigger 
self-heal, and the gfid entries would be updated with their volume paths.  That 
does not seem to be the case here so I feel that all of these entries are 
orphaned.

From: Karthik Subrahmanya [mailto:ksubr...@redhat.com] 
Sent: Wednesday, October 18, 2017 4:34 AM
To: Matt Waymack <mwaym...@nsgdv.com>
Cc: gluster-users <Gluster-users@gluster.org>
Subject: Re: [Gluster-users] gfid entries in volume heal info that do not heal

Hey Matt,
From the xattr output, it looks like the files are not present on the arbiter 
brick & needs healing. But on the parent it does not have the pending markers 
set for those entries.
The workaround for this is you need to do a lookup on the file which needs heal 
from the mount, so it will create the entry on the arbiter brick and then run 
the volume heal to do the healing.
Follow these steps to resolve the issue: (first try this on one file and check 
whether it gets healed. If it gets healed then do this for all the remaining 
files)
1. Get the file path for the gfids you got from heal info output.
    find  -samefile //
2. Do ls/stat on the file from mount.
3. Run volume heal.
4. Check the heal info output to see whether the file got healed.
If one file gets healed, then do step 1 & 2 for the rest of the files and do 
step 3 & 4 once at the end.
Let me know if that resolves the issue.

Thanks & Regards,
Karthik

On Tue, Oct 17, 2017 at 8:04 PM, Matt Waymack <mailto:mwaym...@nsgdv.com> wrote:
Attached is the heal log for the volume as well as the shd log.

>> Run these commands on all the bricks of the replica pair to get the attrs 
>> set on the backend.

[root@tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m . 
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
getfattr: Removing leading '/' from absolute path names
# file: exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.gv0-client-2=0x0001
trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2
trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466332d346463622d393630322d3839356136396461363131662f435f564f4c2d623030312d693637342d63642d63772e6d6435

[root@tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m . 
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
getfattr: Removing leading '/' from absolute path names
# file: exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.gv0-client-2=0x0001
trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2
trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466332d346463622d393630322d3839356136396461363131662f435f564f4c2d623030312d693637342d63642d63772e6d6435

[root@tpc-arbiter1-100617 ~]# getfattr -d -e hex -m . 
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
getfattr: /exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2: No 
such file or directory


[root@tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m . 
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
getfattr: Removing leading '/' from absolute path names
# file: exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.gv0-client-11=0x0001
trusted.gfid=0xe0c56bf78bfe46cabde1e46b92d33df3
trusted.gfid2path.be3ba24c3ef95ff2=0x63323366353834652d353566652d343033382d393131622d3866373063656334616136662f435f564f4c2d623030332d69313331342d63642d636d2d63722e6d6435

[root@tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m . 
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e4

Re: [Gluster-users] gfid entries in volume heal info that do not heal

[Matt Waymack]

Attached is the heal log for the volume as well as the shd log. 

>> Run these commands on all the bricks of the replica pair to get the attrs 
>> set on the backend.

[root@tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m . 
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
getfattr: Removing leading '/' from absolute path names
# file: exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.gv0-client-2=0x0001
trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2
trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466332d346463622d393630322d3839356136396461363131662f435f564f4c2d623030312d693637342d63642d63772e6d6435

[root@tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m . 
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
getfattr: Removing leading '/' from absolute path names
# file: exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.gv0-client-2=0x0001
trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2
trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466332d346463622d393630322d3839356136396461363131662f435f564f4c2d623030312d693637342d63642d63772e6d6435

[root@tpc-arbiter1-100617 ~]# getfattr -d -e hex -m . 
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
getfattr: /exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2: No 
such file or directory


[root@tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m . 
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
getfattr: Removing leading '/' from absolute path names
# file: exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.gv0-client-11=0x0001
trusted.gfid=0xe0c56bf78bfe46cabde1e46b92d33df3
trusted.gfid2path.be3ba24c3ef95ff2=0x63323366353834652d353566652d343033382d393131622d3866373063656334616136662f435f564f4c2d623030332d69313331342d63642d636d2d63722e6d6435

[root@tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m . 
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
getfattr: Removing leading '/' from absolute path names
# file: exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.gv0-client-11=0x0001
trusted.gfid=0xe0c56bf78bfe46cabde1e46b92d33df3
trusted.gfid2path.be3ba24c3ef95ff2=0x63323366353834652d353566652d343033382d393131622d3866373063656334616136662f435f564f4c2d623030332d69313331342d63642d636d2d63722e6d6435

[root@tpc-arbiter1-100617 ~]# getfattr -d -e hex -m . 
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
getfattr: /exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3: No 
such file or directory

>> And the output of "gluster volume heal  info split-brain"

[root@tpc-cent-glus1-081017 ~]# gluster volume heal gv0 info split-brain
Brick tpc-cent-glus1-081017:/exp/b1/gv0
Status: Connected
Number of entries in split-brain: 0

Brick tpc-cent-glus2-081017:/exp/b1/gv0
Status: Connected
Number of entries in split-brain: 0

Brick tpc-arbiter1-100617:/exp/b1/gv0
Status: Connected
Number of entries in split-brain: 0

Brick tpc-cent-glus1-081017:/exp/b2/gv0
Status: Connected
Number of entries in split-brain: 0

Brick tpc-cent-glus2-081017:/exp/b2/gv0
Status: Connected
Number of entries in split-brain: 0

Brick tpc-arbiter1-100617:/exp/b2/gv0
Status: Connected
Number of entries in split-brain: 0

Brick tpc-cent-glus1-081017:/exp/b3/gv0
Status: Connected
Number of entries in split-brain: 0

Brick tpc-cent-glus2-081017:/exp/b3/gv0
Status: Connected
Number of entries in split-brain: 0

Brick tpc-arbiter1-100617:/exp/b3/gv0
Status: Connected
Number of entries in split-brain: 0

Brick tpc-cent-glus1-081017:/exp/b4/gv0
Status: Connected
Number of entries in split-brain: 0

Brick tpc-cent-glus2-081017:/exp/b4/gv0
Status: Connected
Number of entries in split-brain: 0

Brick tpc-arbiter1-100617:/exp/b4/gv0
Status: Connected
Number of entries in split-brain: 0

-Matt

From: Karthik Subrahmanya [mailto:ksubr...@redhat.com] 
Sent: Tuesday, October 17, 2017 1:26 AM
To: Matt Waymack <mwaym...@nsgdv.com>
Cc: gluster-users <Gluster-users@gluster.org>
Subject: Re: [Gluster-users] gfid entries in volume heal info that do not heal

Hi Matt,

Run these commands on all the bricks of the replica pair to get the attrs set 
on the backend.

On the bricks of first replica set:
getfattr -d -e hex -m . /.glusterfs/10/86/108694db-c

Re: [Gluster-users] gfid entries in volume heal info that do not heal

[Matt Waymack]

OK, so here’s my output of the volume info and the heal info. I have not yet 
tracked down physical location of these files, any tips to finding them would 
be appreciated, but I’m definitely just wanting them gone.  I forgot to mention 
earlier that the cluster is running 3.12 and was upgraded from 3.10; these 
files were likely stuck like this when it was on 3.10.

[root@tpc-cent-glus1-081017 ~]# gluster volume info gv0

Volume Name: gv0
Type: Distributed-Replicate
Volume ID: 8f07894d-e3ab-4a65-bda1-9d9dd46db007
Status: Started
Snapshot Count: 0
Number of Bricks: 4 x (2 + 1) = 12
Transport-type: tcp
Bricks:
Brick1: tpc-cent-glus1-081017:/exp/b1/gv0
Brick2: tpc-cent-glus2-081017:/exp/b1/gv0
Brick3: tpc-arbiter1-100617:/exp/b1/gv0 (arbiter)
Brick4: tpc-cent-glus1-081017:/exp/b2/gv0
Brick5: tpc-cent-glus2-081017:/exp/b2/gv0
Brick6: tpc-arbiter1-100617:/exp/b2/gv0 (arbiter)
Brick7: tpc-cent-glus1-081017:/exp/b3/gv0
Brick8: tpc-cent-glus2-081017:/exp/b3/gv0
Brick9: tpc-arbiter1-100617:/exp/b3/gv0 (arbiter)
Brick10: tpc-cent-glus1-081017:/exp/b4/gv0
Brick11: tpc-cent-glus2-081017:/exp/b4/gv0
Brick12: tpc-arbiter1-100617:/exp/b4/gv0 (arbiter)
Options Reconfigured:
nfs.disable: on
transport.address-family: inet

[root@tpc-cent-glus1-081017 ~]# gluster volume heal gv0 info
Brick tpc-cent-glus1-081017:/exp/b1/gv0








Status: Connected
Number of entries: 118

Brick tpc-cent-glus2-081017:/exp/b1/gv0








Status: Connected
Number of entries: 118

Brick tpc-arbiter1-100617:/exp/b1/gv0
Status: Connected
Number of entries: 0

Brick tpc-cent-glus1-081017:/exp/b2/gv0
Status: Connected
Number of entries: 0

Brick tpc-cent-glus2-081017:/exp/b2/gv0
Status: Connected
Number of entries: 0

Brick tpc-arbiter1-100617:/exp/b2/gv0
Status: Connected
Number of entries: 0

Brick tpc-cent-glus1-081017:/exp/b3/gv0
Status: Connected
Number of entries: 0

Brick tpc-cent-glus2-081017:/exp/b3/gv0
Status: Connected
Number of entries: 0

Brick tpc-arbiter1-100617:/exp/b3/gv0
Status: Connected
Number of entries: 0

Brick tpc-cent-glus1-081017:/exp/b4/gv0
























Status: Connected
Number of entries: 24

Brick tpc-cent-glus2-081017:/exp/b4/gv0
























Status: Connected
Number of entries: 24

Brick tpc-arbiter1-100617:/exp/b4/gv0
Status: Connected
Number of entries: 0

Thank you for your help!

From: Karthik Subrahmanya [mailto:ksubr...@redhat.com]
Sent: Monday, October 16, 2017 10:27 AM
To: Matt Waymack <mwaym...@nsgdv.com>
Cc: gluster-users <Gluster-users@gluster.org>
Subject: Re: [Gluster-users] gfid entries in volume heal info that do not heal

Hi Matt,

The files might be in split brain. Could you please send the outputs of these?
gluster volume info 
gluster volume heal  info
And also the getfattr output of the files which are in the heal info output 
from all the bricks of that replica pair.
getfattr -d -e hex -m . 

Thanks &  Regards
Karthik

On 16-Oct-2017 8:16 PM, "Matt Waymack" 
<mwaym...@nsgdv.com<mailto:mwaym...@nsgdv.com>> wrote:
Hi all,

I have a volume where the output of volume heal info shows several gfid entries 
to be healed, but they’ve been there for weeks and have not healed.  Any normal 
file that shows up on the heal info does get healed as expected, but these gfid 
entries do not.  Is there any way to remove these orphaned entries from the 
volume so they are no longer stuck in the heal process?

Thank you!

___
Gluster-users mailing list
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] gfid entries in volume heal info that do not heal

[Matt Waymack]

Hi all,

I have a volume where the output of volume heal info shows several gfid entries 
to be healed, but they've been there for weeks and have not healed.  Any normal 
file that shows up on the heal info does get healed as expected, but these gfid 
entries do not.  Is there any way to remove these orphaned entries from the 
volume so they are no longer stuck in the heal process?

Thank you!
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Cache performance degrades over time

[Matt]

Hello list,

I have about a half a dozen nginx servers sitting in front of Gluster 
(3.4.6, I know it's old) serving a mix videos and images. It's a 
moderate amount of traffic, each of two geo-repped sites will do 2-4 
gigs/second throughout the day.


Here's the problem. Reads from gluster, due to the way nginx buffers 
the video, can far exceed what's being served out to the internet. 
Serving 1 gig of video may read 3 gigs from Gluster.


I can fix this by setting the performance cache on the volume to a 
pretty large size, right now it's at 2 gigs. This works great, gluster 
uses 1.5 - 2 gigs of RAM and the in/out bandwidth on the nginx machines 
becomes a healthy 1:1 or better.


For a few days. Over time, as the machines vfs cache fills, gluster 
starts to use less RAM, and that ratio gets worse. Rebooting the nginx 
boxes (or I presume, simply dropping their caches) fixes it immediately.


I'm going to try increasing vfs.cache_pressure on the nginx boxes, as 
this doc recommends:


https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Linux%20Kernel%20Tuning/#vmvfs95cache95pressure

Does that make sense to tune this on the clients? Is Gluster competing 
with the kernel cache? That's sort of my understanding but I can't find 
a clear explanation.


Other recommendations would be welcome, though tweaking the direct-io 
options is unfortunately not an option in my setup.


-Matt
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Matt Robinson]

If you don't trust the hardware raid, then steer clear of raid-6 as mdadm raid 
6 is stupidly slow.
I don't completely trust hardware raid either, but rebuild times should be 
under a day and in order to lose a raid-6 array you have to lose 3 disks.
My own systems are hardware raid-6.
If you're not terribly worried about maximising usable storage, then mdadm 
raid-10 is your friend.


> On 4 Jul 2016, at 18:15:26, Gandalf Corvotempesta 
> <gandalf.corvotempe...@gmail.com> wrote:
> 
> 2016-07-04 17:01 GMT+02:00 Matt Robinson <m.robin...@sheffield.ac.uk>:
>> Hi Gandalf,
>> 
>> Are you using hardware raid or mdadm?
>> On high quality hardware raid, a 12 disk raid-6 is pretty solid.  With mdadm 
>> any raid6 (especially with 12 disks) will be rubbish.
> 
> I can use both.
> I don't like very much hardware raid, even high quality. Recently i'm
> having too many issue with hardware raid (like multiple disks kicked
> out with no apparent reasons and virtual-disk failed with data loss)
> 
> A RAID-6 with 12x2TB SATA disks would take days to rebuild, in the
> meanwhile, multiple disks could fail resulting in data loss.
> Yes, gluster is able to recover from this, but I prefere to avoid have
> to resync 24TB of data via networks.
> 
> What about a software RAID-1 ? 6 raid for each gluster nodes and 6
> disks wasted but SATA disks are cheaper.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Matt Robinson]

Hi Gandalf,

Are you using hardware raid or mdadm?
On high quality hardware raid, a 12 disk raid-6 is pretty solid.  With mdadm 
any raid6 (especially with 12 disks) will be rubbish.

Matt.

> On 4 Jul 2016, at 15:54:44, Gandalf Corvotempesta 
> <gandalf.corvotempe...@gmail.com> wrote:
> 
> No suggestions ?
> 
> Il 14 giu 2016 10:01 AM, "Gandalf Corvotempesta" 
> <gandalf.corvotempe...@gmail.com> ha scritto:
> Let's assume a small cluster made by 3 servers, 12 disks/bricks each.
> This cluster would be expanded to a maximum of 15 servers in near future.
> 
> What do you suggest, a JBOD or a RAID? Which RAID level?
> 
> 15 servers with 12 disks/bricks in JBOD are 180 bricks. Is this an
> acceptable value?
> Multiple raid6 for each servers? In example, RAID-6 with 6 disks and
> another RAID-6 with the other 6 disks. I'll loose 4 disks on each
> servers, performance would be affected and rebuild times would be huge
> (by using 2TB/4TB disks)
> 
> Any suggestions?
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] root-squash permission denied on rename

[Matt Robinson]

Hi,

I'm fairly new to gluster so please forgive me if I'm in any way out of 
protocol for this list.

I have a distributed volume with 2 servers hosting bricks and a third just 
managing the system.
I really want root squashing, as there are a large number of clients and I do 
not want a bad keystroke on one to wipe out the contents of the gluster 
file-system.
I'm using gluster 3.7.10 on Scientific Linux 6.7.

I just cannot get gluster to work properly for normal users with root-squashing 
enabled.  The problem is easiest to reproduce if one creates a directory with 
mkdir, creates a file with say 'echo hi > filename' and then tries to rename 
the latter to place it in the former using mv.  This fails about 50% of the 
time.  My reading suggests that it occurs when gluster decides to move the file 
from one brick to another as it renames it.  rebalance and fix-layout have been 
run, but have long finished and the problem persists.

I've spent a fair amount of time googling this issue and it's clearly not 
unprecedented, but it's supposedly fixed long before  v3.7.10.
I really would appreciate it if somebody could rescue me.  For the moment I'm 
running with server.root-squash turned off.

Thanks,

Matt.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [ovirt-users] Gluster + ovirt + resize2fs

[Matt Wells]

Thanks Sahina; an item I should have added as well.

On Wed, Jun 1, 2016 at 10:58 PM Sahina Bose <sab...@redhat.com> wrote:

> [+gluster-users]
>
>
> On 06/01/2016 11:30 PM, Matt Wells wrote:
>
> Apologies, it's XFS so would be an xfs_growfs
>
> On Wed, Jun 1, 2016 at 10:58 AM, Matt Wells <matt.we...@mosaic451.com>
> wrote:
>
>> Hi everyone, I had a quick question that I really needed to bounce off
>> someone; one of those measure twice cut once moments.
>>
>> My primary datastore is on a gluster volume and the short story is I'm
>> going to grow it.  I've thought of two options
>>
>> 1 - add a brick with the new space
>> ** Was wondering from the gluster point of view if anyone had a best
>> practice for this.  I've looked around and find many people explaining
>> their stories but not a definitive best practices.
>>
>>
>> 2 - as I'm sitting atop LVMs grow the LVM.
>> ** This is the one that makes me a little nervous.  I've done many
>> resize2fs and never had issues, but I've never had gluster running atop
>> that volume and my VM's atop that.  Has anyone had any experiences they
>> could share?
>>
>> Thanks all -
>> Wells
>>
>
>
> ___
> Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] lots of nfs.log activity since upgrading to 3.4.6

[Matt]

We have all these sites setup in a test environment, I'll see about 
updating.


On Fri, Mar 27, 2015 at 3:47 PM, Justin Clift jus...@gluster.org 
wrote:

As a thought, would you be ok to test our just-released 3.4.7 beta4,
and verify back to us if its fixed for you? :)

  
http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.4.7beta4/


+ Justin


On 27 Mar 2015, at 20:05, Matt m...@mattlantis.com wrote:

 Wonderful, thanks.

 On Thu, Mar 26, 2015 at 6:04 AM, Nithya Balachandran 
nbala...@redhat.com wrote:


 Hi Matt,

 The fix is available at :


 http://review.gluster.org/#/c/10008


 This will be taken in for 3.4.7.


 Regards,
 Nithya


--
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Geo-replication stops every few days

[Matt]

-aux-mount-6Hn48F'.


$ cat 
8ed3c554-8d5d-4304-9cd1-cb9a17c0fd64:gluster%3A%2F%2F127.0.0.1%3Agtmmedia-storage.log
[2015-03-16 13:19:54.624100] I [gsyncd(slave):404:main_i] top: 
syncing: gluster://localhost:media-vol
[2015-03-16 13:19:55.752953] I [resource(slave):483:service_loop] 
GLUSTER: slave listening
[2015-03-16 14:02:40.654535] I [repce(slave):78:service_loop] 
RepceServer: terminating on reaching EOF.
[2015-03-16 14:02:40.661967] I [syncdutils(slave):148:finalize] top: 
exiting.
[2015-03-16 14:02:51.987721] I [gsyncd(slave):404:main_i] top: 
syncing: gluster://localhost:media-vol
[2015-03-16 14:02:53.141150] I [resource(slave):483:service_loop] 
GLUSTER: slave listening
[2015-03-17 17:31:25.696300] I [repce(slave):78:service_loop] 
RepceServer: terminating on reaching EOF.
[2015-03-17 17:31:25.703775] I [syncdutils(slave):148:finalize] top: 
exiting.
[2015-03-17 17:31:37.139935] I [gsyncd(slave):404:main_i] top: 
syncing: gluster://localhost:media-vol
[2015-03-17 17:31:38.228033] I [resource(slave):483:service_loop] 
GLUSTER: slave listening
[2015-03-19 20:51:55.965342] I [resource(slave):489:service_loop] 
GLUSTER: connection inactive for 120 seconds, stopping
[2015-03-19 20:51:55.979207] I [syncdutils(slave):148:finalize] top: 
exiting.


Thanks,

-Matt
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] lots of nfs.log activity since upgrading to 3.4.6

[Matt]

Wonderful, thanks.

On Thu, Mar 26, 2015 at 6:04 AM, Nithya Balachandran 
nbala...@redhat.com wrote:


Hi Matt,

The fix is available at :

http://review.gluster.org/#/c/10008

This will be taken in for 3.4.7.


Regards,
Nithya

- Original Message -
From: Matt m...@mattlantis.com
To: Nithya Balachandran nbala...@redhat.com
Sent: Wednesday, 25 March, 2015 6:24:25 PM
Subject: Re: [Gluster-users] lots of nfs.log activity since upgrading 
to 3.4.6


Thanks. Sounds like it is just the log level at least, no big problem.

Could it be fixed in 3.4.7 before the 3.4 series closes out? I've 
never

opened a bug for Gluster before, but I guess I could get on that.

On Wed, Mar 25, 2015 at 3:48 AM, Nithya Balachandran
nbala...@redhat.com wrote:

 Hi,

 This was inadvertently introduced with another patch. The fix was
 made to master (http://review.gluster.org/#/c/8621/) 3.6 and 3.5 but
 it looks like it was not backported to the 3.4 branch

 Regards,
 Nithya

 - Original Message -
 From: Matt m...@mattlantis.com
 To: gluster-users@gluster.org
 Sent: Tuesday, March 24, 2015 7:29:23 PM
 Subject: [Gluster-users] lots of nfs.log activity since upgrading to
 3.4.6

 Hello list,

 I have a few Wordpress sites served via NFS on Gluster. Since
 upgrading 3.4.6, I'm seeing a non-trivial amount (1-2 million
 depending on how busy the blogs are, about 4 gigs in the last three
 weeks) entries like this appear in nfs.log:

 [2015-03-24 13:49:17.314003] I
 [dht-common.c:1000:dht_lookup_everywhere_done] 0-wpblog-storage-dht:
 STATUS: hashed_subvol wpblog-storage-replicate-0 cached_subvol null
 [2015-03-24 13:49:17.355722] I
 [dht-common.c:1000:dht_lookup_everywhere_done] 0-wpblog-storage-dht:
 STATUS: hashed_subvol wpblog-storage-replicate-0 cached_subvol null
 [2015-03-24 13:49:17.616073] I
 [dht-common.c:1000:dht_lookup_everywhere_done] 0-wpblog-storage-dht:
 STATUS: hashed_subvol wpblog-storage-replicate-0 cached_subvol null

 It doesn't seem to be a big problem, it's just an INFO log, but it
 definitely wasn't there with 3.4.5. Can anyone give me any insight
 into what's going on?

 -Matt

 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] lots of nfs.log activity since upgrading to 3.4.6

[Matt]

Hello list,

I have a few Wordpress sites served via NFS on Gluster. Since upgrading 
3.4.6, I'm seeing a non-trivial amount (1-2 million depending on how 
busy the blogs are, about 4 gigs in the last three weeks) entries like 
this appear in nfs.log:


[2015-03-24 13:49:17.314003] I 
[dht-common.c:1000:dht_lookup_everywhere_done] 0-wpblog-storage-dht: 
STATUS: hashed_subvol wpblog-storage-replicate-0 cached_subvol null
[2015-03-24 13:49:17.355722] I 
[dht-common.c:1000:dht_lookup_everywhere_done] 0-wpblog-storage-dht: 
STATUS: hashed_subvol wpblog-storage-replicate-0 cached_subvol null
[2015-03-24 13:49:17.616073] I 
[dht-common.c:1000:dht_lookup_everywhere_done] 0-wpblog-storage-dht: 
STATUS: hashed_subvol wpblog-storage-replicate-0 cached_subvol null


It doesn't seem to be a big problem, it's just an INFO log, but it 
definitely wasn't there with 3.4.5. Can anyone give me any insight into 
what's going on?


-Matt
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Diagnosing Intermittent Performance Problems Possibly Caused by Gremlins

[Matt]

Thanks Pranith, Will do. Sunday night we put some things in place seem 
to be mitigating it and thankfully haven't seen it again, but if we do 
I'll send the profile info to the list. I was able to collect some 
profile info under normal load.


We added some caching to some files we noticed had become really 
popular, and when that didn't entirely stop the problem, also stopped 
the most recently added gluster volume. It's odd that volume would have 
any impact as it was only used to archive backups and was almost never 
active, but several times we'd stop it during the month just because it 
was most recently added and the issue would go away, start it back up 
and it would come back. Since then it's been quiet.


On Thu, Feb 5, 2015 at 5:14 AM, Pranith Kumar Karampuri 
pkara...@redhat.com wrote:


On 02/03/2015 11:16 AM, Matt wrote:

Hello List,

So I've been frustraded by intermittent performance problems 
throughout January. The problem occurs on a two node setup running 
3.4.5, 16 gigs of ram with a bunch of local disk. For sometimes an 
hour for sometimes weeks at a time (I have extensive graphs in 
OpenNMS) our Gluster boxes will get their CPUs pegged, and in vmstat 
they'll show extremely high numbers of context switches and 
interrupts. Eventually things calm down. During this time, memory 
usage actually drops. Overall usage on the box goes from between 
6-10 gigs to right around 4 gigs, and stays there. That's what 
really puzzles me.


When performance is problematic, sar shows one device, the device 
corresponding to the glusterfsd problem using all the CPU doing lots 
of little reads, Sometimes 70k/second, very small avg rq size, say 
10-12. Afraid I don't have any saved output handy, but I can try to 
capture some next time it happens. I have tons of information 
frankly, but am trying to keep this reasonably brief.


There are more than a dozen volumes on this two node setup. The CPU 
usage is pretty much entirely contained to one volume, a 1.5 TB 
volume that is just shy of 70% full. It stores uploaded files for a 
web app. What I hate about this app and so am always suspicious of, 
is that it stores a directory for every user in one level, so under 
the /data directory in the volume, there are 450,000 sub directories 
at this point.


The only real mitigation step that's been taken so far was to turn 
off the self-heal daemon on the volume, as I thought maybe crawling 
that large directory was getting expensive. This doesn't seem to 
have done anything as the problem still occurs.


At this point I figure there are one of two things sorts of things 
happening really broadly: one we're running into some sort of bug or 
performance problem with gluster we should either fix perhaps by 
upgrading or tuning around, or two, some process we're running but 
not aware of is hammering the file system causing problems.


If it's the latter option, can anyone give me any tips on figuring 
out what might be hammering the system? I can use volume top to see 
what a brick is doing, but I can't figure out how to tell what 
clients are doing what.


Apologies for the somewhat broad nature of the question, any input 
thoughts would be much appreciated. I can certainly provide more 
info about some things if it would help, but I've tried not to write 
a novel here.


Thanks,
Could you enable 'gluster volume profile volname start' for this 
volume?
When next time this issue happens, keep collecting 'gluster volume 
profile volname info' outputs. Mail them and lets see what is 
happening.


Pranith


-Matt


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Diagnosing Intermittent Performance Problems Possibly Caused by Gremlins

[Matt]

I’ve been trying for weeks to reproduce the performance problems in 
our preproduction environments but can’t. As a result, selling that 
just upgrading to 3.6.x and hoping it goes away might be tricky. 3.6 is 
perceived as a little too bleeding edge, and we’ve actually had some 
other not fully explained issues with this cluster recently that make 
us hesitate. I don’t think they’re related.


On Tue, Feb 3, 2015 at 4:58 AM, Justin Clift jus...@gluster.org wrote:

- Original Message -

 Hello List,

 So I've been frustraded by intermittent performance problems 
throughout
 January. The problem occurs on a two node setup running 3.4.5, 16 
gigs
 of ram with a bunch of local disk. For sometimes an hour for 
sometimes
 weeks at a time (I have extensive graphs in OpenNMS) our Gluster 
boxes
 will get their CPUs pegged, and in vmstat they'll show extremely 
high

 numbers of context switches and interrupts. Eventually things calm
 down. During this time, memory usage actually drops. Overall usage 
on
 the box goes from between 6-10 gigs to right around 4 gigs, and 
stays

 there. That's what really puzzles me.

 When performance is problematic, sar shows one device, the device
 corresponding to the glusterfsd problem using all the CPU doing 
lots of
 little reads, Sometimes 70k/second, very small avg rq size, say 
10-12.

 Afraid I don't have any saved output handy, but I can try to capture
 some next time it happens. I have tons of information frankly, but 
am

 trying to keep this reasonably brief.

 There are more than a dozen volumes on this two node setup. The CPU
 usage is pretty much entirely contained to one volume, a 1.5 TB 
volume
 that is just shy of 70% full. It stores uploaded files for a web 
app.
 What I hate about this app and so am always suspicious of, is that 
it

 stores a directory for every user in one level, so under the /data
 directory in the volume, there are 450,000 sub directories at this
 point.

 The only real mitigation step that's been taken so far was to turn 
off

 the self-heal daemon on the volume, as I thought maybe crawling that
 large directory was getting expensive. This doesn't seem to have 
done

 anything as the problem still occurs.

 At this point I figure there are one of two things sorts of things
 happening really broadly: one we're running into some sort of bug or
 performance problem with gluster we should either fix perhaps by
 upgrading or tuning around, or two, some process we're running but 
not

 aware of is hammering the file system causing problems.

 If it's the latter option, can anyone give me any tips on figuring 
out
 what might be hammering the system? I can use volume top to see 
what a

 brick is doing, but I can't figure out how to tell what clients are
 doing what.

 Apologies for the somewhat broad nature of the question, any input
 thoughts would be much appreciated. I can certainly provide more 
info
 about some things if it would help, but I've tried not to write a 
novel

 here.


Out of curiosity, are you able to test using GlusterFS 3.6.2?  We've
had a bunch of pretty in-depth upstream testing at decent scale (100+
nodes) from 3.5.x onwards, with lots of performance issues identified
and fixed on the way through.

So, I'm kinda hopeful the problem you're describing is fixed in newer
releases. :D

Regards and best wishes,

Justin Clift

--
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Diagnosing Intermittent Performance Problems Possibly Caused by Gremlins

[Matt]

Hello List,

So I've been frustraded by intermittent performance problems throughout 
January. The problem occurs on a two node setup running 3.4.5, 16 gigs 
of ram with a bunch of local disk. For sometimes an hour for sometimes 
weeks at a time (I have extensive graphs in OpenNMS) our Gluster boxes 
will get their CPUs pegged, and in vmstat they'll show extremely high 
numbers of context switches and interrupts. Eventually things calm 
down. During this time, memory usage actually drops. Overall usage on 
the box goes from between 6-10 gigs to right around 4 gigs, and stays 
there. That's what really puzzles me.


When performance is problematic, sar shows one device, the device 
corresponding to the glusterfsd problem using all the CPU doing lots of 
little reads, Sometimes 70k/second, very small avg rq size, say 10-12. 
Afraid I don't have any saved output handy, but I can try to capture 
some next time it happens. I have tons of information frankly, but am 
trying to keep this reasonably brief.


There are more than a dozen volumes on this two node setup. The CPU 
usage is pretty much entirely contained to one volume, a 1.5 TB volume 
that is just shy of 70% full. It stores uploaded files for a web app. 
What I hate about this app and so am always suspicious of, is that it 
stores a directory for every user in one level, so under the /data 
directory in the volume, there are 450,000 sub directories at this 
point.


The only real mitigation step that's been taken so far was to turn off 
the self-heal daemon on the volume, as I thought maybe crawling that 
large directory was getting expensive. This doesn't seem to have done 
anything as the problem still occurs.


At this point I figure there are one of two things sorts of things 
happening really broadly: one we're running into some sort of bug or 
performance problem with gluster we should either fix perhaps by 
upgrading or tuning around, or two, some process we're running but not 
aware of is hammering the file system causing problems.


If it's the latter option, can anyone give me any tips on figuring out 
what might be hammering the system? I can use volume top to see what a 
brick is doing, but I can't figure out how to tell what clients are 
doing what.


Apologies for the somewhat broad nature of the question, any input 
thoughts would be much appreciated. I can certainly provide more info 
about some things if it would help, but I've tried not to write a novel 
here.


Thanks,

-Matt
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.4.2 on Ubuntu 12.04 LTS Server - Upstart No Go

[Matt Edwards]

I could be completely off-base, but were you trying to mount with noatime?

I ran into a similar problem where, after upgrading to 3.4.2, all my client
mounts failed.  I discovered (by accident) that it was only due to
incompatible mount options (namely noatime).  In previous versions, the
client just gave a warning message about unsupported options, but now it
fails without mentioning the cause.  I believe there is a bug report for
this issue (that I discovered afterwards).

Anyways, I thought this had a chance of being the issue, but maybe it's
something different in your case.

Matt


On Tue, Mar 4, 2014 at 2:09 PM, Brock Nanson br...@nanson.net wrote:

 I thought I would share my experience in case it helps someone else avoid
 a bruised forehead.

 I'm running the semiosis Gluster 3.4.2 on Ubuntu 12.04 (two nodes,
 replicated) as server and client on both.  Going through all the typical
 configuration work, all seemed good.  The problem came when I tried to get
 the boxes to start Gluster and mount the volume at boot... without user
 intervention.  Thinking that the most difficult part of the installation
 should be configuring Gluster (newbie to distributed file systems in
 general, never used Gluster before), I foolishly wasted a pile of time
 trying to fix the *simple* boot problem.  Well, I did fix it, but only by
 finally disabling the upstart files bundled in the install
 (glusterfs-server.conf and mounting-glusterfs.conf).

 This was a clean install of Ubuntu 12.04.  Actually, two clean installs
 for testing.  And then another clean install on one box to rule out the
 original install!  Same behaviour on all installs.

 I spent some time experimenting with different fstab entries and found I
 could change the mount behaviour, but only to determine *when* the mount
 would fail... it always failed!

 My solution was to disable the two Gluster upstart conf files with
 .override files containing 'manual'.  Then a simple script to a) start
 glusterfs-server; b) sleep for a few seconds; c) mount the Gluster volume.
  The script is called from rc.local.

 Hardly an elegant solution I know, but at some point *'works'* beats *'pretty
 but non-functional'*.

 It's possible there is something quirky with my install that could be
 responsible.  But given that other than installing and configuring Gluster,
 nothing on the servers was tweaked, I'm thinking there is a problem with
 the Gluster upstart conf files.  Or something in 12.04 has rendered them
 inoperable. Or maybe it's as a friend in Poland says... (direct
 translation) it's the Malice of Things...  Truly a better description
 than anything we have in English! ;-)

 I'm not looking for a solution unless someone has an obvious fix that
 Google couldn't locate.  I have a quick and dirty work around that works.
  I just hope to save someone else the pain.

 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://supercolony.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster, Samba, and VFS

[Matt Miller]

Stumbled upon
https://forge.gluster.org/samba-glusterfs/samba-glusterfs-vfs/commits/masterwhen
trying to find info on how to make Gluster and Samba play nice as a
general purpose file server.  I have had severe performance problems in the
past with mounting the Gluster volume as a Fuse mount, then exporting the
Fuse mount thru Samba.  As I found out after setting up the cluster this is
somewhat expected when serving out lots of small files.  Was hoping VFS
would provide better performance when serving out lots and lots of small
files.

Is anyone using VFS extensions in production?  Is it ready for prime time?
I could not find a single reference to it on Gluster's main website (maybe
I am looking in the wrong place), so not sure of the stability or
supported-ness of this.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Typical setup questions

[Matt Weil]

Guys,

Thanks for the responses it is appreciated.

On 8/28/12 5:28 PM, Bryan Whitehead wrote:

I'f found pricing for Infiniband switches / cards to be cheaper than
10G cards/switches with the addition of being 4X fast.


I will look into this but putting all of our compute on Infiniband may 
be cost prohibitive.




On Tue, Aug 28, 2012 at 11:44 AM, Joe Topjian j...@topjian.net wrote:

Hi Matt,

On Tue, Aug 28, 2012 at 9:29 AM, Matt Weil mw...@genome.wustl.edu wrote:


Since we are on the subject of hardware what would be the perfect fit for
a gluster brick. We where looking at a PowerEdge C2100 Rack Server.



Just a note: the c2100 has been superseded by the Dell r720xd. Although the
r720 is not part of the c-series, it's their official replacement.


I looked at these but they only hold 8 3.5 drives verses 12 and two on 
the inside on the 2100.  I will ask our rep about this.


Do you typically run hot spares or just keep cold spares handy?





During testing I found it pretty easy to saturate 1 Gig network links.
This was also the case when multiple links where bonded together.  Are there
any cheap 10 gig switch alternatives that anyone would suggest?



While not necessarily cheap, I've had great luck with Arista 7050 switches.


also looking at dell's new force 10 switches.  Wonder how they compare 
price wise.




We implement them in sets of two, linked together. We then use dual-port
10gb NICs and connect each NIC to each switch. It gives multiple layers of
redundancy + a theoretical 20gb throughput per server.

Thanks,
Joe

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Typical setup questions

[Matt Weil]

Brian

thanks for this response.

Since we are on the subject of hardware what would be the perfect fit 
for a gluster brick. We where looking at a PowerEdge C2100 Rack Server.


During testing I found it pretty easy to saturate 1 Gig network links. 
This was also the case when multiple links where bonded together.  Are 
there any cheap 10 gig switch alternatives that anyone would suggest?


Matt

On 8/24/12 4:28 PM, Brian Candler wrote:

On Fri, Aug 24, 2012 at 10:51:24AM -0500, Matt Weil wrote:

I am curious what is used typically for the file system replication
and how do you make sure that it is consistent.

So for example when using large 3TB+ sata/NL-sas drives.  Is is
typical to replicate three times to get similar protection to raid
6?


Gluster sits on top of existing filesystems on the storage bricks, so it's
fine to continue to use RAID10 (for performance) or RAID6 (for capacity) on
those nodes.  Gluster replicated volumes, and/or gluster geo-replication,
then give you an additional layer of replication on top of that, and the
ability to handle entire servers going out of service.

If I were you, I would not want to have a non-resilient array like a RAID0
on my storage bricks.

Whilst in principle you could have lots of separate 3TB filesystems and put
them into a large distributed/replicated set, I think this is likely to be
difficult to manage.  In particular, the process of replacing a failed disk
requires more skill than a simple RAID drive swap.

One word of warning: when choosing 3TB SATA drives, ensure they support
error recovery control (a.k.a. time-limited error recovery).

Enterprise drives do, but many consumer ones don't. The Hitachi consumer
ones do, for now anyway; Seagate ones do not.

To attempt to enable it on a particular drive:

 # smartctl -l scterc,70,70 /dev/sda

If the drive supports it, you'll see:

 SCT Error Recovery Control set to:
Read: 70 (7.0 seconds)
   Write: 70 (7.0 seconds)

There's plenty of discussion on the linux-raid mailing list if you want to
go through the archives.


Also what is typically done to ensure that all replicas are in place
and consistent?  A cron that stats of ls's the file system from a
single client?


I don't have a good answer to that. Stat'ing all files recursively used to
be required for gluster 3.3 to force healing.  As of gluster 3.3, there is
a self-healing daemon which handles this automatically.  So basically, you
trust gluster to do its job.

I guess there could be value in running a recursive md5sum on each replica
locally and comparing the results (but you'd have to allow for files which
were in the process of changing during the scan)

Regards,

Brian.



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster + firefox/sqlite

[Matt Cowan]

On Fri, 6 Jan 2012, Harshavardhana wrote:

Users have mounted /home for years now, its nothing new.


I was dying to use it for home for it's replication capability, but
ran into issues with locking and sockets when using nfs clients.


I have 2 systems I've been able to do tests on; everything's
Scientific Linux 6.1:



client1 (nfs): 
firefox-3.6.24-3.el6_1.x86_64


homedir serverpair1 (replicate):
glusterfs-{core,fuse}-3.2.5-2.el6.x86_64 (from gluster.org)


client2 (nfs or gluster native):
same as client1 plus:
glusterfs-{,fuse}3.2.5-4.el6.x86_64 (from epel-testing)

homedir servercluster2 (distribute):
glusterfs-{core,fuse}-3.2.4-1.x86_64 (from gluster.org)



on client1, with nfs homedir on serverpair1, firefox takes *forever* to
startup.  An strace shows it's hanging up getting various locks
(sqlite stuff and others).  And when it finally does start, it pops up
a red banner saying:

   The bookmarks and history system will not be functional because
   one of Firefox's files is in use by another application.  Some
   security software can cause this problem

Interestingly, this does not happen on client 2 with nfs homedir on 
servercluster2,
which is running a slightly older version of gluster.


And sockets get created as pipes!?! and fail to function.

$ mksock testsock
$ ls -al testsock
prwxrwxr-x. 1 glustertest glustertest 0 Jan  6 14:13 testsock
^
This happens in both nfs homedir setups.


On client2, with a native gluster homedir from server2, everything
works fine.  I would go native everywhere, but it would be painful to
maintain on some legacy systems, I think I read 32bit systems aren't
really supported(?), and I'd like to avoid all the clients doubling
their homedir io.

-matt
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Transport endpoint is not connected

[Matt Weil]

All,

Is this normal?  Can this be corrected?

Thanks in advance for your responses.


[2011-12-06 17:56:59.48153] I 
[glusterd-rpc-ops.c:1243:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC 
from uuid: fc5e6659-a90a-4e25-a3a7-11de9a7de81d
[2011-12-06 17:56:59.48811] I 
[glusterd-rpc-ops.c:1243:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC 
from uuid: d1216f43-2ae6-42bd-a597-c0ab6a101d6b
[2011-12-06 17:56:59.49073] I 
[glusterd-rpc-ops.c:1243:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC 
from uuid: 4bf94e6e-69ca-4d51-9a85-c1d98a95325d
[2011-12-06 17:56:59.49137] I 
[glusterd-rpc-ops.c:1243:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC 
from uuid: 154cdbb2-6a53-449d-b6e3-bfd84091d90c
[2011-12-06 17:56:59.49567] I 
[glusterd-rpc-ops.c:1243:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC 
from uuid: 4c9d68d6-d573-43d0-aec5-07173c1699d0
[2011-12-06 17:56:59.49803] I 
[glusterd-rpc-ops.c:818:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received 
ACC from uuid: fc5e6659-a90a-4e25-a3a7-11de9a7de81d
[2011-12-06 17:56:59.49850] I 
[glusterd-rpc-ops.c:818:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received 
ACC from uuid: d1216f43-2ae6-42bd-a597-c0ab6a101d6b
[2011-12-06 17:56:59.50228] I 
[glusterd-rpc-ops.c:818:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received 
ACC from uuid: 4bf94e6e-69ca-4d51-9a85-c1d98a95325d
[2011-12-06 17:56:59.50285] I 
[glusterd-rpc-ops.c:818:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received 
ACC from uuid: 154cdbb2-6a53-449d-b6e3-bfd84091d90c
[2011-12-06 17:56:59.50346] I 
[glusterd-rpc-ops.c:818:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received 
ACC from uuid: 4c9d68d6-d573-43d0-aec5-07173c1699d0
[2011-12-06 17:56:59.50375] I [glusterd-op-sm.c:7250:glusterd_op_txn_complete] 
0-glusterd: Cleared local lock
[2011-12-06 17:56:59.52105] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (127.0.0.1:694)
[2011-12-06 17:56:59.168257] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.11:730)
[2011-12-06 17:56:59.168357] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.11:728)
[2011-12-06 17:56:59.168441] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.11:726)
[2011-12-06 17:56:59.168503] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.11:724)
[2011-12-06 17:56:59.168591] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.11:722)
[2011-12-06 17:56:59.168672] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.11:720)
[2011-12-06 17:56:59.169287] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.15:730)
[2011-12-06 17:56:59.169359] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.15:728)
[2011-12-06 17:56:59.169398] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.13:730)
[2011-12-06 17:56:59.169438] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.13:728)
[2011-12-06 17:56:59.169476] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.15:726)
[2011-12-06 17:56:59.169529] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.13:726)
[2011-12-06 17:56:59.169581] W [socket.c:1494:__socket_proto_state_machine] 
0-socket.management: reading from socket failed. Error (Transport endpoint is 
not connected), peer (10.0.30.13:724)


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] folders with no permissions.

[Matt Weil]

Just some simply iozone testing failed due to folder permissions.  The 
tmp folder is created by iozone.


six nodes with stripe 6 and underlying EXT4 file system.

The ext4 filesystems where not mounted with the -o acl option.

Any Ideas?

in both cases it created a folder with no permissions.



188K d- 2 rootroot  24K 2011-12-06 11:28 tmp/
iozone$ rm -rf tmp
rm: cannot remove directory `tmp': Permission denied
test$ cd iozone.broke/
iozone.broke$ ls
./  ../  tmp/
iozone.broke$ ls -lash
total 580K 288K drwxrwxrwx 6 rootroot  24K 2011-12-04 14:28 ../
4.0K d- 2 rootroot 4.0K 2011-12-06 11:32 tmp/





each node has error trying to set permissions on that folder.




[2011-12-06 12:23:36.593604] E [marker.c:2018:marker_setattr_cbk] 0-gluster1-marker: 
Operation not permitted occured during setattr of nul
[2011-12-06 12:23:36.593669] I [server3_1-fops.c:1526:server_setattr_cbk] 
0-gluster1-server: 433: SETATTR /test/iozone/tmp (-734804259) == -1 (Operation 
not permitted)
[2011-12-06 12:23:36.593669] I [server3_1-fops.c:1526:server_setattr_cbk] 
0-gluster1-server: 433: SETATTR /test/iozone/tmp (-734804259) == -1 (Operation 
not permitted)



188K d- 2 rootroot  24K 2011-12-06 11:28 tmp/
iozone$ chmod +rw tmp/
chmod: changing permissions of `tmp/': Operation not permitted
iozone$ ls -lash


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] folders with no permissions.

[Matt Weil]

On 12/7/11 12:53 PM, Matt Weil wrote:



Just some simply iozone testing failed due to folder permissions. The
tmp folder is created by iozone.

six nodes with stripe 6 and underlying EXT4 file system.

The ext4 filesystems where not mounted with out the -o acl option.



I meant with out the -o acl option.


Any Ideas?

in both cases it created a folder with no permissions.



188K d- 2 root root 24K 2011-12-06 11:28 tmp/
iozone$ rm -rf tmp
rm: cannot remove directory `tmp': Permission denied
test$ cd iozone.broke/
iozone.broke$ ls
./ ../ tmp/
iozone.broke$ ls -lash
total 580K 288K drwxrwxrwx 6 root root 24K 2011-12-04 14:28 ../
4.0K d- 2 root root 4.0K 2011-12-06 11:32 tmp/





each node has error trying to set permissions on that folder.




[2011-12-06 12:23:36.593604] E [marker.c:2018:marker_setattr_cbk]
0-gluster1-marker: Operation not permitted occured during setattr of
nul
[2011-12-06 12:23:36.593669] I
[server3_1-fops.c:1526:server_setattr_cbk] 0-gluster1-server: 433:
SETATTR /test/iozone/tmp (-734804259) == -1 (Operation not permitted)
[2011-12-06 12:23:36.593669] I
[server3_1-fops.c:1526:server_setattr_cbk] 0-gluster1-server: 433:
SETATTR /test/iozone/tmp (-734804259) == -1 (Operation not permitted)



188K d- 2 root root 24K 2011-12-06 11:28 tmp/
iozone$ chmod +rw tmp/
chmod: changing permissions of `tmp/': Operation not permitted
iozone$ ls -lash


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] syslog options for gluster

[Matt Weil]

are there any options to have glusterd use syslog?

Would like the logs to go to a central server.

Thanks
Matt
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] ESXi Gluster setup options

[Matt Temple]

Chris Haumesser ch@... writes:

 
 
 The native FUSE client has built-in failover if you mount using the server 
and volume name (e.g. `mount -t glusterfs server1:/foo /mnt/foo`). When
mounting this way, the gluster client downloads a list of all the bricks, and if
one goes down, automatically fails over to another available brick server.Note,
however,
that at mount-time, the client needs to be able to reach the brick server
specified in the mount command. You could use simple roundrobin DNS to 
make sure your clients always reach an available brick.As others stated
previously, NFS failover requires ucarp or an alternative ha 
framework.Cheers,-C-

 On Thu, May 26, 2011 at 2:37 PM, paul simpson paul at realisestudio.com
wrote:i'm also interested in this.  is there any pro/con to using native 
gluster FUSE client for xen images?  i would have thought that would mitigate
the use of ucarp (apart from initial connect).
 
 
 -p
 
 
 
 
 
 
 On 26 May 2011 21:39, Whit Blauvelt
whit.glus...@transpect.com wrote:
 
 On Thu, May 26, 2011 at 08:28:29PM +, Matt Temple wrote:
  We've been looking for a way to have HA for our VMWare datastore, too. (Our
  single server had a kernel panic last night and took down the VMs.)
  We're very much interested in a similar setup, using Gluster, but I have a
  question ... with Gluster NFS, don't you have to choose a specific 
 address of a
  server to connect to?   And if yes, if that node goes down, how does VMWare
  respond?
 Pretty much the standard thing to do is use a virtual IP address with ucarp
 handling reassignment if the primary node goes down. I've yet to run that
 through thorough tests to see if how transparent it is in a case of failure.
 It's simple to set up though.
 There are, of course, plenty of more complicated alternatives to ucarp in
 the HA world - heartbeat, pacemaker, corosync
 Whit
 
 ___
 Gluster-users mailing listGluster-users at
gluster.orghttp://gluster.org/cgi-bin/mailman/listinfo/gluster-users
 
 
 
 
 
 
 
 
 ___
 Gluster-users mailing
listGluster-users@gluster.orghttp://gluster.org/cgi-bin/mailman/listinfo/gluster-users
 
 
 
 
 
 ___
 Gluster-users mailing list
 Gluster-users@...
 http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
 
All,

   We just set up UCARP on our simple replicating 3.2 server, but when we 
tried to connect to the virtual address as an NFS store from VCenter, the
connection was rejected.  When we used the real ip address of the server, it
connected.
Is anyone successfully using UCARP/Gluster with VMWare for storage using
NFSconnections.   Or are we just doing something wrong?   HA on the VMWare
storagefor the VMDKs is pretty important and automatic NFS failover would really
be terrific.   Doesn't NFS have to much context to simply failover?

(If this were KVM, I'm sure we'd be able to use a native gluster client, but
that's another story.)

Any help would be appreciated.   We could document all the steps to a working
example.

Matt 



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] ESXi Gluster setup options

[Matt Temple]

Krishna Srinivas krishna@... writes:

 
 Troy,
 
 Yes it would work, you can setup 4 servers in a distributed replicated
 setup acting as NFS data store for VMWare. If any one of the storage
 node goes down it will not be seen by the ESX hosts. Many users use
 this type of config for storage HA for NFS datastores.
 
 You can use SATA or SAS.
 
 Since you are using 10gb performance should not be a problem. Backend
 disks would be the bottleneck and hence you should experiment with a
 good raid setup to get maximum backend disk throughput.
 
 On Thu, Apr 21, 2011 at 9:54 PM, Troy Swaine troy.swaine@... wrote:
  All,
 
 
 
     We are in the process of determining a virtualized infrastructure and
  wanted to hear from
 
  current users of Gluster and VMWare. What we were looking to setup was an HA
  ESXi cluster
 
  (2 heads) with gluster backend (4 bricks to start, replicated/distributed),
  all backend connectivity
  would be 10Gbe. Mainly the storage would be for VM images but may include
  NAS files later.
 
  Thanks,
 
  Troy


Hello,  

We've been looking for a way to have HA for our VMWare datastore, too. (Our
single server had a kernel panic last night and took down the VMs.)
We're very much interested in a similar setup, using Gluster, but I have a
question ... with Gluster NFS, don't you have to choose a specific address of a
server to connect to?   And if yes, if that node goes down, how does VMWare 
respond?


Matt Temple


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.1.1 64-bit only?

[Matt Keating]

FYI: its been working great so far... Havn't had any issues with it on a
self compiled i386 rpm.

On Tue, Dec 14, 2010 at 11:16 AM, Liam Slusser lslus...@gmail.com wrote:

 You can - its just not supported or very well tested.

 liam

 On Mon, Dec 13, 2010 at 10:07 AM, Matt Keating
 matt.keating.li...@gmail.com wrote:
  Will it not run at all if I compile on a 32bit system?
 
  ___
  Gluster-users mailing list
  Gluster-users@gluster.org
  http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
 
 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.1 bailout error

[Matt Keating]

 Hi Matt,

 Can you attach entire client and server log files?

 regards,


What part of the logs do you need exactly, as the logs are about 14mb. The
error comes up ALOT...

- Original Message -
 From: Matt Keating matt.keating.li...@gmail.com
 To: gluster-users@gluster.org
 Sent: Sunday, December 5, 2010 3:39:34 AM
 Subject: [Gluster-users] Gluster 3.1 bailout error

 Hi,

 I've got a GlusterFS share serving web pages and I'm finding that
 imagecache
 isn't always able to create new files on the mount.
 Since upgrading to GlusterFS 3.1, I'm having ALOT of these errors appearing
 in the logs:

 logs/EBS-drupal-shared-.log:[2010-11-29 10:29:18.141045] E
 [rpc-clnt.c:199:call_bail] drupal-client-0: bailing out frame
 type(GlusterFS
 3.1) op(FINODELK(30)) xid = 0xb69cc sent = 2010-11-29 09:59:10.112834.
 timeout = 1800
 logs/EBS-drupal-shared-.log:[2010-11-29 10:42:58.365735] E
 [rpc-clnt.c:199:call_bail] drupal-client-0: bailing out frame
 type(GlusterFS
 3.1) op(FINODELK(30)) xid = 0xb863e sent = 2010-11-29 10:12:54.584124.
 timeout = 1800
 logs/EBS-drupal-shared-.log:[2010-11-29 12:00:02.572679] E
 [rpc-clnt.c:199:call_bail] drupal-client-0: bailing out frame
 type(GlusterFS
 3.1) op(FINODELK(30)) xid = 0xbe36f sent = 2010-11-29 11:29:57.497653.
 timeout = 1800


 Could anyone shed any light on whats happening/wrong?

 Thanks,
 Matt


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Gluster 3.1.1 64-bit only?

[Matt Keating]

Will it not run at all if I compile on a 32bit system?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Gluster 3.1 bailout error

[Matt Keating]

Hi,

I've got a GlusterFS share serving web pages and I'm finding that imagecache
isn't always able to create new files on the mount.
Since upgrading to GlusterFS 3.1, I'm having ALOT of these errors appearing
in the logs:

logs/EBS-drupal-shared-.log:[2010-11-29 10:29:18.141045] E
[rpc-clnt.c:199:call_bail] drupal-client-0: bailing out frame type(GlusterFS
3.1) op(FINODELK(30)) xid = 0xb69cc sent = 2010-11-29 09:59:10.112834.
timeout = 1800
logs/EBS-drupal-shared-.log:[2010-11-29 10:42:58.365735] E
[rpc-clnt.c:199:call_bail] drupal-client-0: bailing out frame type(GlusterFS
3.1) op(FINODELK(30)) xid = 0xb863e sent = 2010-11-29 10:12:54.584124.
timeout = 1800
logs/EBS-drupal-shared-.log:[2010-11-29 12:00:02.572679] E
[rpc-clnt.c:199:call_bail] drupal-client-0: bailing out frame type(GlusterFS
3.1) op(FINODELK(30)) xid = 0xbe36f sent = 2010-11-29 11:29:57.497653.
timeout = 1800


Could anyone shed any light on whats happening/wrong?

Thanks,
Matt
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] filling gluster cluster with large file doesn't crash the system?!

[Matt Hodson]

craig,

A) stat on server that generated the file:

j...@riff:/mnt/gluster$ stat y.out
 File: `y.out'
 Size: 214574579712Blocks: 291441832  IO Block: 65536  regular file
Device: 16h/22dInode: 14213316644377695875  Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (0/root)   Gid: (0/root)
Access: 2010-11-09 10:32:16.0 -0800
Modify: 2010-11-09 13:57:05.0 -0800
Change: 2010-11-09 13:57:05.0 -0800

B) stat on gluster brick (brick2) that is housing the file:

[r...@brick2 ~]# stat /exp2/y.out
  File: `/exp2/y.out'
  Size: 214574579712Blocks: 291441832  IO Block: 4096   regular file
Device: fd00h/64768dInode: 655412  Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (0/root)   Gid: (0/root)
Access: 2010-11-09 10:32:16.0 -0800
Modify: 2010-11-09 13:57:05.0 -0800
Change: 2010-11-09 13:57:05.0 -0800

c) df on brick 2

[r...@brick2 ~]# df -h
FilesystemSize  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
  143G  143G 0 100% /
/dev/hda1  99M   13M   82M  13% /boot
tmpfs 470M 0  470M   0% /dev/shm
172.16.1.76:/gs-test  283G  158G  119G  58% /mnt/gluster


interesting that block size is different depending on who you ask.

[r...@brick2 ~]# dumpe2fs /dev/hda1 | grep -i 'Block size'
dumpe2fs 1.39 (29-May-2006)
Block size:   1024

-Matt


On Nov 11, 2010, at 9:44 PM, Craig Carl wrote:


Matt,
  Based on your Gluster servers configs that file is bigger than the  
available disk space, obviously that isn't right.


Can you send us the output of `stat y.out` taken from the Gluster  
mount point and from the back end of the server Gluster created the  
file on




  I'm also going to try and reproduce the problem here on 3.1 and  
3.1.1qa5.



Thanks,
Craig

--
Craig Carl
Gluster, Inc.
Cell - (408) 829-9953 (California, USA)
Gtalk - craig.c...@gmail.com


From: Matt Hodson ma...@geospiza.com
To: Craig Carl cr...@gluster.com
Cc: Jeff Kozlowski j...@genesifter.net, gluster-users@gluster.org
Sent: Wednesday, November 10, 2010 9:21:40 AM
Subject: Re: [Gluster-users] filling gluster cluster with large file  
doesn't crash the system?!


Craig,
inline...

On Nov 10, 2010, at 7:17 AM, Craig Carl wrote:

Matt -
   A couple of questions -

What is your volume config? (`gluster volume info all`)

gluster volume info all

Volume Name: gs-test
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 172.16.1.76:/exp1
Brick2: 172.16.2.117:/exp2

What is the hardware config for each storage server?

brick 1 = 141GB
brick 2 = 143GB

What command did you run to create the test data?

#perl -e 'print rand while 1'  y.out 

What process is still writing to the file?

same one as above.


Thanks,
Craig

--
Craig Carl
Gluster, Inc.
Cell - (408) 829-9953 (California, USA)
Gtalk - craig.c...@gmail.com


From: Matt Hodson ma...@geospiza.com
To: gluster-users@gluster.org
Cc: Jeff Kozlowski j...@genesifter.net
Sent: Tuesday, November 9, 2010 10:46:04 AM
Subject: Re: [Gluster-users] filling gluster cluster with large file  
doesn'tcrash the system?!


I should also note that on this non-production test rig the block size
on both bricks is 1KB (1024) so the theoretical file size limit is
16GB.  so how then did i get a file of 200GB?
-matt

On Nov 9, 2010, at 10:34 AM, Matt Hodson wrote:

 craig et al,

 I have a 2 brick distributed 283GB gluster cluster on CentoOS 5. we
 nfs mounted the cluster from a 3rd machine and wrote random junk to
 a file. i watched the file grow to 200GB on the cluster when it
 appeared to stop. however the machine writing to the file still
 lists the file as growing. it's now at over 320GB. what's going on?

 -matt

 ---
 Matt Hodson
 Scientific Customer Support, Geospiza
 (206) 633-4403, Ext. 111
 http://www.geospiza.com






___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users




___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] filling gluster cluster with large file doesn't crash the system?!

[Matt Hodson]

Craig,
inline...

On Nov 10, 2010, at 7:17 AM, Craig Carl wrote:


Matt -
   A couple of questions -

What is your volume config? (`gluster volume info all`)


gluster volume info all

Volume Name: gs-test
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 172.16.1.76:/exp1
Brick2: 172.16.2.117:/exp2


What is the hardware config for each storage server?


brick 1 = 141GB
brick 2 = 143GB


What command did you run to create the test data?


#perl -e 'print rand while 1'  y.out 


What process is still writing to the file?


same one as above.



Thanks,
Craig

--
Craig Carl
Gluster, Inc.
Cell - (408) 829-9953 (California, USA)
Gtalk - craig.c...@gmail.com


From: Matt Hodson ma...@geospiza.com
To: gluster-users@gluster.org
Cc: Jeff Kozlowski j...@genesifter.net
Sent: Tuesday, November 9, 2010 10:46:04 AM
Subject: Re: [Gluster-users] filling gluster cluster with large file  
doesn'tcrash the system?!


I should also note that on this non-production test rig the block size
on both bricks is 1KB (1024) so the theoretical file size limit is
16GB.  so how then did i get a file of 200GB?
-matt

On Nov 9, 2010, at 10:34 AM, Matt Hodson wrote:

 craig et al,

 I have a 2 brick distributed 283GB gluster cluster on CentoOS 5. we
 nfs mounted the cluster from a 3rd machine and wrote random junk to
 a file. i watched the file grow to 200GB on the cluster when it
 appeared to stop. however the machine writing to the file still
 lists the file as growing. it's now at over 320GB. what's going on?

 -matt

 ---
 Matt Hodson
 Scientific Customer Support, Geospiza
 (206) 633-4403, Ext. 111
 http://www.geospiza.com






___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] filling gluster cluster with large file doesn't crash the system?!

[Matt Hodson]

craig et al,

I have a 2 brick distributed 283GB gluster cluster on CentoOS 5. we  
nfs mounted the cluster from a 3rd machine and wrote random junk to a  
file. i watched the file grow to 200GB on the cluster when it appeared  
to stop. however the machine writing to the file still lists the file  
as growing. it's now at over 320GB. what's going on?


-matt

---
Matt Hodson
Scientific Customer Support, Geospiza
(206) 633-4403, Ext. 111
http://www.geospiza.com




___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] filling gluster cluster with large file doesn't crash the system?!

[Matt Hodson]

I should also note that on this non-production test rig the block size  
on both bricks is 1KB (1024) so the theoretical file size limit is  
16GB.  so how then did i get a file of 200GB?

-matt

On Nov 9, 2010, at 10:34 AM, Matt Hodson wrote:


craig et al,

I have a 2 brick distributed 283GB gluster cluster on CentoOS 5. we  
nfs mounted the cluster from a 3rd machine and wrote random junk to  
a file. i watched the file grow to 200GB on the cluster when it  
appeared to stop. however the machine writing to the file still  
lists the file as growing. it's now at over 320GB. what's going on?


-matt

---
Matt Hodson
Scientific Customer Support, Geospiza
(206) 633-4403, Ext. 111
http://www.geospiza.com






___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] filling gluster cluster with large file doesn't crash the system?!

[Matt Hodson]

approx 141GB+143GB=283 avail to the cluster.  see below.

Brick #1 (main gluster server)
ma...@meathouse|11:01:56
$ df -h
FilesystemSize  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
  141G   15G  119G  12% /


Brick #2 (slave)
[r...@localhost ~]# df -h
FilesystemSize  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
  143G  143G 0 100% /
/dev/hda1  99M   13M   82M  13% /boot
tmpfs 470M 0  470M   0% /dev/shm
172.16.1.76:/gs-test  283G  158G  119G  58% /mnt/gluster

-matt

---
Matt Hodson
Scientific Customer Support, Geospiza
(206) 633-4403, Ext. 111
http://www.geospiza.com




On Nov 9, 2010, at 10:57 AM, Luis E. Cerezo wrote:


what's the size of the storage bricks?


On Nov 9, 2010, at 12:46 PM, Matt Hodson wrote:

I should also note that on this non-production test rig the block  
size on both bricks is 1KB (1024) so the theoretical file size  
limit is 16GB.  so how then did i get a file of 200GB?

-matt

On Nov 9, 2010, at 10:34 AM, Matt Hodson wrote:


craig et al,

I have a 2 brick distributed 283GB gluster cluster on CentoOS 5.  
we nfs mounted the cluster from a 3rd machine and wrote random  
junk to a file. i watched the file grow to 200GB on the cluster  
when it appeared to stop. however the machine writing to the file  
still lists the file as growing. it's now at over 320GB. what's  
going on?


-matt

---
Matt Hodson
Scientific Customer Support, Geospiza
(206) 633-4403, Ext. 111
http://www.geospiza.com






___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


Luis E. Cerezo

blog: http://www.luiscerezo.org
fotofun: http://www.flickr.com/photos/luiscerezo/
twitter: http://twitter.com/luiscerezo/
Voice: +1 412 223 7396



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] monitor connections and activity?

[Matt Hodson]

Is there a way using gluster to monitor number of mounts to a gluster  
cluster and any current activity?



---
Matt Hodson
Scientific Customer Support, Geospiza
(206) 633-4403, Ext. 111
http://www.geospiza.com




___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] cannot nfs mount glusterFS

[Matt Hodson]

I just installed distributed gluster FS on 2 CentOS 5 boxes. install  
and configuration seemed to go fine. gluterd is running. firewalls/ 
iptables are off.  however for the life of me i cannot nfs mount the  
main gluster server from either a OSX or a CentOS 5 box. I use NFS  
often and have a fair amount of experience with it so i've reviewed  
most of the common pitfalls.


here's the command that fails from centos:
$ sudo mount -v -t nfs 172.16.1.76:/gs-test /mnt/gluster/
mount: trying 172.16.1.76 prog 13 vers 3 prot tcp port 2049
mount: trying 172.16.1.76 prog 15 vers 3 prot udp port 909
mount: 172.16.1.76:/gs-test failed, reason given by server: No such  
file or directory


and the same one from OSX 10.5
sudo mount -v -t nfs 172.16.1.76:/gs-test /gluster/
mount_nfs: can't access /gs-test: No such file or directory

what's weird is that i can mount actual dirs on the gluster server,  
just not the gluster VOLNMAE. in other words, this command works fine  
because it's mounting an actual dir.

$sudo mount -v -t nfs 172.16.1.76:/ /mnt/gluster/

help?

thanks,
-matt

---
Matt Hodson
Scientific Customer Support, Geospiza
(206) 633-4403, Ext. 111
http://www.geospiza.com




___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] cannot nfs mount glusterFS

[Matt Hodson]

HA! that was it. dolt! thank you. i was going crazy looking at other  
stuff.


-matt


---
Matt Hodson
Scientific Customer Support, Geospiza
(206) 633-4403, Ext. 111
http://www.geospiza.com




On Nov 3, 2010, at 11:26 AM, Vikas Gorur wrote:



On Nov 3, 2010, at 11:18 AM, Matt Hodson wrote:

I just installed distributed gluster FS on 2 CentOS 5 boxes.  
install and configuration seemed to go fine. gluterd is running.  
firewalls/iptables are off.  however for the life of me i cannot  
nfs mount the main gluster server from either a OSX or a CentOS 5  
box. I use NFS often and have a fair amount of experience with it  
so i've reviewed most of the common pitfalls.


here's the command that fails from centos:
$ sudo mount -v -t nfs 172.16.1.76:/gs-test /mnt/gluster/
mount: trying 172.16.1.76 prog 13 vers 3 prot tcp port 2049
mount: trying 172.16.1.76 prog 15 vers 3 prot udp port 909
mount: 172.16.1.76:/gs-test failed, reason given by server: No such  
file or directory


and the same one from OSX 10.5
sudo mount -v -t nfs 172.16.1.76:/gs-test /gluster/
mount_nfs: can't access /gs-test: No such file or directory

what's weird is that i can mount actual dirs on the gluster server,  
just not the gluster VOLNMAE. in other words, this command works  
fine because it's mounting an actual dir.

$sudo mount -v -t nfs 172.16.1.76:/ /mnt/gluster/


You have the kernel NFS service running. That is why you can mount  
regular directories on the gluster server.


When you try to mount Gluster the kernel NFS server is actually  
looking for a directory called /gs-test, which ofcourse does not  
exist. You need to stop the kernel NFS service and stop and start  
the gluster volume.


--
Vikas Gorur
Engineer - Gluster, Inc.
--










___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] GlusterFS behavior with 2 clients

[Matt Sorin]

Hello,

I recently discovered GlusterFS and followed the installation instructions and 
the relatively brief tutorials that describe setting up test-volume but the 
behavior I am seeing is odd.  I have 4 machines set up to test GlusterFS: 2 
servers and 2 clients.

All the machines are running RHEL5U4 x86_64. The servers were set up as so:

192.168.56.101   server1
192.168.56.102   server2
192.168.56.103   client1
192.168.56.104   client2

glusterfs-core-3.1.0-1 and glusterfs-fuse-3.1.0-1 were installed on all 4 
nodes. glusterd was started on all 4 nodes (though I expect only the 
bricks/peers need it).

The commands executed on server1 were:

gluster peer probe server2
(no problems here)
gluster peer status
(it returned the correct results but I don't have the output handy)
gluster volume create test-volume replica 2 transport tcp server1:/exp1 
server2:/exp2
(the command succeeded and it created an /exp1 directory on server1 and 
/exp2 on server2)
gluster volume start test-volume
(again, no problems)


Now, on the clients...

Mount test-volume on the clients:
[r...@client1 tmp]# mount -t glusterfs server1:/test-volume /mnt
[r...@client2 ~]# mount -t glusterfs server1:/test-volume /mnt

Make a file on test-volume using client1:
[r...@client1 tmp]# echo 1234 /mnt/foo
[r...@client1 tmp]# cat /mnt/foo
1234
[r...@client1 tmp]# ls -al /mnt/foo
-rw-r--r-- 1 root root 5 Oct 28 19:34 /mnt/foo

Make sure the file is seen on client2:
[r...@client2 ~]# cat /mnt/foo
1234
[r...@client2 ~]# ls -al /mnt/foo
-rw-r--r-- 1 root root 5 Oct 28 19:34 /mnt/foo

Append to the file using client2:
[r...@client2 ~]# echo 5678 /mnt/foo
[r...@client2 ~]# ls -al /mnt/foo
-rw-r--r-- 1 root root 10 Oct 28 19:35 /mnt/foo
[r...@client2 ~]# md5sum /mnt/foo
9053253e972cf40443a4083f452f24d4  /mnt/foo

Now check and see that client1 sees the changes:
[r...@client1 tmp]# cat /mnt/foo
1234
[r...@client1 tmp]# ls -al /mnt/foo
-rw-r--r-- 1 root root 10 Oct 28 19:35 /mnt/foo
[r...@client1 tmp]# md5sum /mnt/foo
e7df7cd2ca07f4f1ab415d457a6e1c13  /mnt/foo

If I check server1:/exp1/foo and server2:/exp2/foo, the files are in agreement 
with what client2 thinks the file should look like.

client1 sees the new file size according to ls but cat and md5sum still 
only think the file in its original 1234 state is there.

If I unmount and remount test-volume on client1, it sees the modified file as 
it exists on server1 and server2.

I've tried this same test on both RHEL5U4 and Fedora Core 13 with the same 
results so I must be doing something wrong.

Can someone please tell me what I have done incorrectly?

Thank you for your patience with a newbie.

-
Matt
  ___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Nufa deprecated?

[Matt]

Hi,

(i) a note at the top on the NUFA with single process page
http://www.gluster.com/community/documentation/index.php/NUFA_with_single_process
declares NUFA as deprecated.  
Does this mean any Nufa setup is scheduled to be unsupported of is it just the 
NUFA with single process as client and server processes have been merged. 

(ii) In setting up  Nufa/replicate with a single process I am right now 
starting the glusterfsd daemon by locally mounting my Nufa/replicate volume.  
This is a pretty strange way to start a daemon but first starting the daemon 
and then trying to mount locally does not work with my single .vol file. 

(iii) I am also wondering about my NUFA/replicate/distribute setup as 
described in http://gluster.org/pipermail/gluster-users/2010-July/004988.html
is this considered by the glusterFS developers an unsupported setup?

(iv) Would it be possible to re-introduce the unify translator to get a 
replicated NUFA setup with multiple local discs working? (I am quite willing
to not have the files distributed across local bricks.)

Thanks for the help!

Regards,
Matt
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] trouble combining nufa, distribute and replicate

[Matt]

On Jul 7, 2010, at 19:03 , Jeff Darcy wrote:

 On 07/07/2010 07:13 AM, Matthias Munnich wrote:
 thanks a lot for your comment! It sounds like I hit a bug here.  Would 
 you still feel queasy if I added option lookup-unhashed yes to avoid dht?
 
 I don't think that will really avoid dht, especially the
 extended-attribute parts that are the likely cause of conflict with nufa.
 
 What I reported here was some initial testing.  In the end I like to use 
 glusterfs to provide a uniform name space for our O(20) workstations
 with lokal storage of 4 to 16TB.  The data should be mirrored (once)
 for reliability and stored locally were possible for speed.   I also
 would prefer not to glue together local disk using LVM or software
 raid to keep it easy to add/remove disks without having to expand a
 filesystem. 
 
 Any hints how to  set this up  with glusterfs?
 
 I think you can get close enough with just nufa on top of replicate.
 If you have two disks per machine, no matter how many machines you have,
 assign one to be active and one passive.  Pair each active to a passive
 on another node however you like using cluster/replicate, then combine
 all of those using cluster/nufa with local-volume-name pointing to the
 replica pair where the active half is local.  If you had more than two
 disks per machine, this wouldn't work quite as well but the differences
 should be minor.  If that's not good enough, I just looked at the code
 and it doesn't seem like it would be hard to make nufa recognize
 multiple volumes (instead of just one) as local.  If you'd like, I could
 try generating and submitting a patch for that.

This would be REAL COOL!!!  Most of our machines hold 4 disk and a few
up to 8.

Thanks so much for the help!

... Matt

 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Matt
langel...@gmx.net



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] NUFA/replicate setup

[Matt]

Hi James,

did you ever figure out how to combine replicate with nufa. I always run into 
stale NFS handle errors when I try to do this.

Thanks for your help!

(I'll send a detailed description of my tries to the gluster-users list)


Matt
langel...@gmx.net



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] DHT and AFR question

[Matt M]

Hi,

I'm setting up gluster to share /usr/local among 24 compute nodes.  The 
basic goal is to be able to change files in /usr/local in one place, and 
have it replicate out to all the other nodes.


What I'd like to avoid is having a single point of failure where one (or 
several) nodes go down and the gluster volume becomes unavailable 
everywhere.  I've begun to set it up using this example 
http://www.gluster.org/docs/index.php/Mixing_DHT_and_AFR.


If all the nodes are both client and servers, how many replicate volumes 
should I have?  I don't want a ton of replication traffic, but 
redundancy is important.  Maximizing the size of the volume is less 
important to me since I'm just reclaiming otherwise unused local disk 
space on the nodes.


Thanks,
Matt

___
Gluster-users mailing list
Gluster-users@gluster.org
http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Problem with fs hang

[Matt M]

Stephan von Krawczynski wrote:

Hello all,

I am evaluating glusterfs currently and therefore use a testbed consisting of
four (real) boxes, two configured as fileservers, two as clients, see configs
below.
All I have to do to make the fs hang on both clients (sooner or later) is to
run bonnie (a simple fs check program) constantly on the mounted tree.


I'm seeing similar problems with 2.0.1 -- any log entries?  I get a lot 
of these in glusterfs.log when mine hangs:


[2009-06-11 05:10:51] E [client-protocol.c:292:call_bail] zircon: 
bailing out frame STATFS(15) frame sent = 2009-06-11 04:40:50. 
frame-timeout = 1800
[2009-06-11 05:10:52] W 
[client-protocol.c:5869:protocol_client_interpret] zircon: no frame for 
callid=1913585 type=4 op=29
[2009-06-11 05:10:52] W 
[client-protocol.c:5869:protocol_client_interpret] zircon: no frame for 
callid=2706745 type=4 op=40


My setup is very close to yours -- using replicate, iocache, readahead, 
and writebehind on the client and iothreads on the servers.  The thread 
below sounds like my problem, but I don't have autoscaling (explicitly) 
turned on:


http://www.mail-archive.com/gluster-de...@nongnu.org/msg06140.html

...so I'm wondering if it could be something else.

-Matt


___
Gluster-users mailing list
Gluster-users@gluster.org
http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] another NFS vs glusterfs performance question

[Matt M]

Hi All,

I'm new to gluster and have a basic test environment of three old PCs: 
two servers and one client.  I've currently got it configured to do AFR 
on the two servers and HA on the client, according to this example:

http://www.gluster.org/docs/index.php/High-availability_storage_using_server-side_AFR

I'm trying to figure out why NFS seems significantly faster in my 
(basic) tests.  My config files and results are below.  Any help is 
greatly appreciated!


server1 (garnet) is SUSE SLES 9(OES1), gluster 2.0.0rc7, FUSE 2.5.3, 
2.6.5-7.316-smp
server2 (or) is SUSE SLES 10, gluster 2.0.0rc7, FUSE 2.7.2, 
2.6.16.60-0.34-default
client1 (charon) is SUSE SLES 10, gluster 2.0.0rc7, FUSE 2.7.2, 
2.6.16.60-0.34-default



RESULTS
all tests performed on the client -- /gfs is my glusterfs mount and /nfs 
is the gluster FS shared from server1 via NFS:


GFS - no performance translators
time find /gfs/users/1 -type f
0.768u 1.460s 1:59.09 1.8%  0+0k 0+0io 0pf+0w

GFS - w/readahead and writeback:
0.784u 1.860s 1:59.62 2.2%  0+0k 0+0io 0pf+0w

NFS
time find /nfs/users/1 -type f
0.584u 3.796s 0:37.96 11.5% 0+0k 0+0io 0pf+0w

NFS - after an umount/mount
time find /nfs/users/1 -type f
0.556u 3.224s 0:40.57 9.2%  0+0k 0+0io 0pf+0w

GFS - dd
Directory: /gfs/users
[charon: users]# time sh -c dd if=/dev/zero of=ddfile bs=8k 
count=200  sync

200+0 records in
200+0 records out
1638400 bytes (16 GB) copied, 7065.52 seconds, 2.3 MB/s
1.488u 13.440s 1:57:45.64 0.2%  0+0k 0+0io 1pf+0w

NFS - dd
(unmount NFS volume, remount it)
Directory: /nfs/users
[charon: users]# time sh -c dd if=/dev/zero of=ddfile bs=8k 
count=200  sync

200+0 records in
200+0 records out
1638400 bytes (16 GB) copied, 1582.31 seconds, 10.4 MB/s
2.640u 125.299s 26:22.70 8.0%   0+0k 0+0io 5pf+0w


CONFIGS:
--
server1 (garnet)
[garnet: users]# cat /etc/gluster/glusterfsd-ha-afr.vol

# dataspace on garnet
volume gfs-ds
  type storage/posix
  option directory /export/home
end-volume

# posix locks
volume gfs-ds-locks
  type features/posix-locks
  subvolumes gfs-ds
end-volume

# dataspace on or
volume gfs-or-ds
  type protocol/client
  option transport-type tcp/client
  option remote-host 152.xx.xx.xx
  option remote-subvolume gfs-ds-locks
  option transport-timeout 10
end-volume

# automatic file replication translator for dataspace
volume gfs-ds-afr
  type cluster/afr
  subvolumes gfs-ds-locks gfs-or-ds# local and remote dataspaces
end-volume

# the actual volume to export
volume users
  type performance/io-threads
  option thread-count 8
  subvolumes gfs-ds-afr
end-volume

# make the home volume available as a server share
volume server
 type protocol/server
 option transport-type tcp
 subvolumes users
 option auth.addr.gfs-ds-locks.allow 152.xx.xx.*
 option auth.addr.users.allow 152.xx.xx.*
end-volume

--
server2 (or)
[or: gluster]# cat /etc/gluster/glusterfsd-ha-afr.vol

# dataspace on or
volume gfs-ds
  type storage/posix
  option directory /export/home
end-volume

# posix locks
volume gfs-ds-locks
  type features/posix-locks
  subvolumes gfs-ds
end-volume

# dataspace on garent
volume gfs-garnet-ds
  type protocol/client
  option transport-type tcp/client
  option remote-host 152.xx.xx.xx
  option remote-subvolume gfs-ds-locks
  option transport-timeout 10
end-volume

# automatic file replication translator for dataspace
volume gfs-ds-afr
  type cluster/afr
  subvolumes gfs-ds-locks gfs-garnet-ds# local and remote dataspaces
end-volume

# the actual volume to export
volume users
  type performance/io-threads
  option thread-count 8
  subvolumes gfs-ds-afr
end-volume

# make the users volume available as a server share
volume server
 type protocol/server
 option transport-type tcp
 subvolumes users
 option auth.addr.gfs-ds-locks.allow 152.xx.xx.*
 option auth.addr.users.allow 152.xx.xx.*
end-volume

--
client1 (charon)
[r...@charon:users]# cat /etc/gluster/glusterfs-ha.vol

# the exported volume to mount
volume gluster
  type protocol/client
  option transport-type tcp/client
  option remote-host gluster.example.com   # RRDNS
  option remote-subvolume users# exported volume
  option transport-timeout 10
end-volume

# performance block for gluster
volume writeback
  type performance/write-behind
  option aggregate-size 131072
  subvolumes gluster
end-volume

# performance block for gluster
volume readahead
  type performance/read-ahead
  option page-size 65536
  option page-count 16
  subvolumes writeback
end-volume

volume ioc
  type performance/io-cache
  option cache-size 128MB
  subvolumes readahead
end-volume

Thanks!
-Matt


___
Gluster-users mailing list
Gluster-users@gluster.org
http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users