Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[Anuradha Talur]

- Original Message -
> From: "David Gossage" 
> To: "Anuradha Talur" 
> Cc: "gluster-users@gluster.org List" , "Krutika 
> Dhananjay" 
> Sent: Monday, August 29, 2016 5:12:42 PM
> Subject: Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow
> 
> On Mon, Aug 29, 2016 at 5:39 AM, Anuradha Talur  wrote:
> 
> > Response inline.
> >
> > - Original Message -
> > > From: "Krutika Dhananjay" 
> > > To: "David Gossage" 
> > > Cc: "gluster-users@gluster.org List" 
> > > Sent: Monday, August 29, 2016 3:55:04 PM
> > > Subject: Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow
> > >
> > > Could you attach both client and brick logs? Meanwhile I will try these
> > steps
> > > out on my machines and see if it is easily recreatable.
> > >
> > > -Krutika
> > >
> > > On Mon, Aug 29, 2016 at 2:31 PM, David Gossage <
> > dgoss...@carouselchecks.com
> > > > wrote:
> > >
> > >
> > >
> > > Centos 7 Gluster 3.8.3
> > >
> > > Brick1: ccgl1.gl.local:/gluster1/BRICK1/1
> > > Brick2: ccgl2.gl.local:/gluster1/BRICK1/1
> > > Brick3: ccgl4.gl.local:/gluster1/BRICK1/1
> > > Options Reconfigured:
> > > cluster.data-self-heal-algorithm: full
> > > cluster.self-heal-daemon: on
> > > cluster.locking-scheme: granular
> > > features.shard-block-size: 64MB
> > > features.shard: on
> > > performance.readdir-ahead: on
> > > storage.owner-uid: 36
> > > storage.owner-gid: 36
> > > performance.quick-read: off
> > > performance.read-ahead: off
> > > performance.io-cache: off
> > > performance.stat-prefetch: on
> > > cluster.eager-lock: enable
> > > network.remote-dio: enable
> > > cluster.quorum-type: auto
> > > cluster.server-quorum-type: server
> > > server.allow-insecure: on
> > > cluster.self-heal-window-size: 1024
> > > cluster.background-self-heal-count: 16
> > > performance.strict-write-ordering: off
> > > nfs.disable: on
> > > nfs.addr-namelookup: off
> > > nfs.enable-ino32: off
> > > cluster.granular-entry-heal: on
> > >
> > > Friday did rolling upgrade from 3.8.3->3.8.3 no issues.
> > > Following steps detailed in previous recommendations began proces of
> > > replacing and healngbricks one node at a time.
> > >
> > > 1) kill pid of brick
> > > 2) reconfigure brick from raid6 to raid10
> > > 3) recreate directory of brick
> > > 4) gluster volume start <> force
> > > 5) gluster volume heal <> full
> > Hi,
> >
> > I'd suggest that full heal is not used. There are a few bugs in full heal.
> > Better safe than sorry ;)
> > Instead I'd suggest the following steps:
> >
> > Currently I brought the node down by systemctl stop glusterd as I was
> getting sporadic io issues and a few VM's paused so hoping that will help.
> I may wait to do this till around 4PM when most work is done in case it
> shoots load up.
> 
> 
> > 1) kill pid of brick
> > 2) to configuring of brick that you need
> > 3) recreate brick dir
> > 4) while the brick is still down, from the mount point:
> >a) create a dummy non existent dir under / of mount.
> >
> 
> so if noee 2 is down brick, pick node for example 3 and make a test dir
> under its brick directory that doesnt exist on 2 or should I be dong this
> over a gluster mount?
You should be doing this over gluster mount.
> 
> >b) set a non existent extended attribute on / of mount.
> >
> 
> Could you give me an example of an attribute to set?   I've read a tad on
> this, and looked up attributes but haven't set any yet myself.
> 
Sure. setfattr -n "user.some-name" -v "some-value" 
> Doing these steps will ensure that heal happens only from updated brick to
> > down brick.
> > 5) gluster v start <> force
> > 6) gluster v heal <>
> >
> 
> Will it matter if somewhere in gluster the full heal command was run other
> day?  Not sure if it eventually stops or times out.
> 
full heal will stop once the crawl is done. So if you want to trigger heal 
again,
run gluster v heal <>. Actually even brick up or volume start force should
trigger the heal.
> >
> > > 1st node worked as expected took 12 hours to heal 1TB data. Load was
> > little
> > > heavy but nothing shocking.
> > >
> > > Abo

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[Anuradha Talur]

Response inline.

- Original Message -
> From: "Krutika Dhananjay" 
> To: "David Gossage" 
> Cc: "gluster-users@gluster.org List" 
> Sent: Monday, August 29, 2016 3:55:04 PM
> Subject: Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow
> 
> Could you attach both client and brick logs? Meanwhile I will try these steps
> out on my machines and see if it is easily recreatable.
> 
> -Krutika
> 
> On Mon, Aug 29, 2016 at 2:31 PM, David Gossage < dgoss...@carouselchecks.com
> > wrote:
> 
> 
> 
> Centos 7 Gluster 3.8.3
> 
> Brick1: ccgl1.gl.local:/gluster1/BRICK1/1
> Brick2: ccgl2.gl.local:/gluster1/BRICK1/1
> Brick3: ccgl4.gl.local:/gluster1/BRICK1/1
> Options Reconfigured:
> cluster.data-self-heal-algorithm: full
> cluster.self-heal-daemon: on
> cluster.locking-scheme: granular
> features.shard-block-size: 64MB
> features.shard: on
> performance.readdir-ahead: on
> storage.owner-uid: 36
> storage.owner-gid: 36
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: on
> cluster.eager-lock: enable
> network.remote-dio: enable
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> server.allow-insecure: on
> cluster.self-heal-window-size: 1024
> cluster.background-self-heal-count: 16
> performance.strict-write-ordering: off
> nfs.disable: on
> nfs.addr-namelookup: off
> nfs.enable-ino32: off
> cluster.granular-entry-heal: on
> 
> Friday did rolling upgrade from 3.8.3->3.8.3 no issues.
> Following steps detailed in previous recommendations began proces of
> replacing and healngbricks one node at a time.
> 
> 1) kill pid of brick
> 2) reconfigure brick from raid6 to raid10
> 3) recreate directory of brick
> 4) gluster volume start <> force
> 5) gluster volume heal <> full
Hi,

I'd suggest that full heal is not used. There are a few bugs in full heal.
Better safe than sorry ;)
Instead I'd suggest the following steps:

1) kill pid of brick
2) to configuring of brick that you need
3) recreate brick dir
4) while the brick is still down, from the mount point:
   a) create a dummy non existent dir under / of mount.
   b) set a non existent extended attribute on / of mount.
Doing these steps will ensure that heal happens only from updated brick to down 
brick.
5) gluster v start <> force
6) gluster v heal <>
> 
> 1st node worked as expected took 12 hours to heal 1TB data. Load was little
> heavy but nothing shocking.
> 
> About an hour after node 1 finished I began same process on node2. Heal
> proces kicked in as before and the files in directories visible from mount
> and .glusterfs healed in short time. Then it began crawl of .shard adding
> those files to heal count at which point the entire proces ground to a halt
> basically. After 48 hours out of 19k shards it has added 5900 to heal list.
> Load on all 3 machnes is negligible. It was suggested to change this value
> to full cluster.data-self-heal-algorithm and restart volume which I did. No
> efffect. Tried relaunching heal no effect, despite any node picked. I
> started each VM and performed a stat of all files from within it, or a full
> virus scan and that seemed to cause short small spikes in shards added, but
> not by much. Logs are showing no real messages indicating anything is going
> on. I get hits to brick log on occasion of null lookups making me think its
> not really crawling shards directory but waiting for a shard lookup to add
> it. I'll get following in brick log but not constant and sometime multiple
> for same shard.
> 
> [2016-08-29 08:31:57.478125] W [MSGID: 115009]
> [server-resolve.c:569:server_resolve] 0-GLUSTER1-server: no resolution type
> for (null) (LOOKUP)
> [2016-08-29 08:31:57.478170] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-GLUSTER1-server: 12591783:
> LOOKUP (null) (---00
> 00-/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.221) ==> (Invalid
> argument) [Invalid argument]
> 
> This one repeated about 30 times in row then nothing for 10 minutes then one
> hit for one different shard by itself.
> 
> How can I determine if Heal is actually running? How can I kill it or force
> restart? Does node I start it from determine which directory gets crawled to
> determine heals?
> 
> David Gossage
> Carousel Checks Inc. | System Administrator
> Office 708.613.2284
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] FreeBSD: I can't replace-bricks - Distributed-Replicate

[Anuradha Talur]

- Original Message -
> From: "Pranith Kumar Karampuri" 
> To: "Jan Michael Martirez" , "Anuradha Talur" 
> 
> Cc: "gluster-users" 
> Sent: Tuesday, August 23, 2016 9:23:54 PM
> Subject: Re: [Gluster-users] FreeBSD: I can't replace-bricks - 
> Distributed-Replicate
> 
> hi Jan,
>  Are you doing this as part of erasing the underlying disk(which we
> call as reset brick) or replacing the complete brick with a new brick?
> If it is replacing and not resetting, considering you are using 3.7.6
> version you can use the CLI directly without all these steps.
> 
> Just use single command "gluster volume replace-brick  
>  commit force"
> 
> We never tested it on FreeBSD though, may be it is a good idea to try this
> out in a test environment and do it on your production setup. This code
> does get executed on NetBSD regressions, not sure if that is good enough
> for FreeBSD.
> 
> 
> +Anuradha,
>Could you update readthedocs documentation with the release details
> after which just executing replace-brick is good enough? As per git log
> v3.7.3 has your patch.
Yes, will do.
> 
> On Tue, Aug 23, 2016 at 6:18 AM, Jan Michael Martirez 
> wrote:
> 
> > I can't use replace-bricks.
> >
> > I followed this tutorial: https://gluster.readthedocs.io/en/latest/
> > Administrator%20Guide/Managing%20Volumes/#replace-brick
> >
> > Volume Name: dr
> > Type: Distributed-Replicate
> > Volume ID: 0ce3038c-55c6-4a4e-9b97-22269bce9d11
> > Status: Started
> > Number of Bricks: 2 x 2 = 4
> > Transport-type: tcp
> > Bricks:
> > Brick1: gluster01:/glu1
> > Brick2: gluster02:/glu2
> > Brick3: gluster03:/glu3
> > Brick4: gluster04:/glu4
> > Options Reconfigured:
> > features.shard-block-size: 4MB
> > features.shard: on
> > performance.readdir-ahead: on
> >
> > I'm stuck with setfattr. I'm using FreeBSD, so I use setextattr instead.
> >
> > root@gluster01:/mnt/fuse # setextattr system wheel abc /mnt/fuse
> > setextattr: /mnt/fuse: failed: Operation not supported
> >
> > root@gluster01:/mnt/fuse # glusterd --version
> > glusterfs 3.7.6 built on Jul 13 2016 20:32:46
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> >
> 
> 
> 
> --
> Pranith
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Move brick to new Node ( Distributed - Replicated Mode )

[Anuradha Talur]

- Original Message -
> From: "tecforte jason" 
> To: "Anuradha Talur" 
> Cc: gluster-users@gluster.org
> Sent: Friday, August 12, 2016 1:30:21 PM
> Subject: Re: [Gluster-users] Move brick to new Node ( Distributed - 
> Replicated Mode )
> 
> Hi Anuradha,
> 
> Thanks for the advice, the version i use is 3.8.
> 
> You are correct with my requirement with original 2 nodes ( Distributed -
> Replicated Mode ) expanded to 3 nodes  ( Distributed - Replicated Mode ).May
> i know the additional work for heal to be triggered based on the glusterfs
> version 3.8 ?
> 
If you are in version 3.8 then there is no additional work to be done.

> My another question is : IF i only have 2 bricks in each nodes,
> any possibility add the 4th node ( also only have 2 bricks ) or 5th node
> etc.. to the cluster ? With the condition cannot add more brick to existing
> node.
> 
> 
I'm not sure I get what you are trying to ask.
You can add 4th and 5th node the same way you would be adding the third one.
The main thing that you would need to plan is how you want to pair the replica
bricks.
If you are going to add 4th and 5th node together, you can peer probe them into
the cluster first.

Assume the bricks on them as follows:
node4:/exp4/brick1 and node4:/exp4/brick2
node5:/exp5/brick1 and node5:/exp5/brick2

Then run the following command:
gluster v add-brick  node4:/exp4/brick1 node5:/exp5/brick1 
node4:/exp4/brick2
node5:/exp5/brick2

This should add all new four bricks into the volume.

> Thank you for the guide.
> 
> Jason
> 
> 
> On Fri, Aug 12, 2016 at 3:19 PM, Anuradha Talur  wrote:
> 
> >
> >
> > - Original Message -
> > > From: "tecforte jason" 
> > > To: gluster-users@gluster.org
> > > Sent: Friday, August 12, 2016 5:41:07 AM
> > > Subject: [Gluster-users] Move brick to new Node ( Distributed -
> > ReplicatedMode )
> > >
> > > Hi,
> > >
> > > If i have existing 2 nodes with Distributed - Replicated setup like
> > below :
> > >
> > > gluster volume create test-volume replica 2
> > > node1:/exp1/brick1 node2:/exp2/brick2
> > > node1:/exp1/brick3 node2:/exp2/brick4
> > > node1:/exp1/brick5 node2:/exp2/brick6
> > >
> > > And now i want to add another new node to the gluster and
> > > node2:/exp2/brick2 replicated with the new brick node3"/exp3/brick7
> > >
> > > with the condition cannot add brick to existing node 1 and node 2.
> >
> > What version of glusterfs are you using?
> >
> > If I understand correctly, the current volume configuration is as follows:
> >
> > node1:/exp1/brick1 node2:/exp2/brick2 (replica of each other)
> > node1:/exp1/brick3 node2:/exp2/brick4
> > node1:/exp1/brick5 node2:/exp2/brick6
> >
> > And you want to change it to:
> >
> > node2:/exp2/brick2 node3:/exp3/brick7 (replica of each other)
> > node1:/exp1/brick3 node2:/exp2/brick4
> > node1:/exp1/brick5 node2:/exp2/brick6
> >
> > You can do the following:
> > gluster peer probe node 3 (run this from one of the nodes in your cluster)
> > gluster v replace-brick  node1:/exp1/brick1 node3:/exp3/brick7
> > commit force
> >
> > This should now make node2:/exp2/brick2 and node3:/exp3/brick7 as replica
> > of each other.
> >
> > You might have to do some additional work for heal to be triggered based
> > on the glusterfs version
> > you are using. So do mention the gluster version being used.
> >
> > Let me know if my understanding of the requirement given was incorrect.
> >
> > Hope this helps.
> > >
> > > May i know how to do this ?
> > >
> > > Appreciate for the advice.
> > >
> > > Thanks
> > > Jason
> > >
> > > ___
> > > Gluster-users mailing list
> > > Gluster-users@gluster.org
> > > http://www.gluster.org/mailman/listinfo/gluster-users
> >
> > --
> > Thanks,
> > Anuradha.
> >
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Move brick to new Node ( Distributed - Replicated Mode )

[Anuradha Talur]

- Original Message -
> From: "tecforte jason" 
> To: gluster-users@gluster.org
> Sent: Friday, August 12, 2016 5:41:07 AM
> Subject: [Gluster-users] Move brick to new Node ( Distributed - Replicated
> Mode )
> 
> Hi,
> 
> If i have existing 2 nodes with Distributed - Replicated setup like below :
> 
> gluster volume create test-volume replica 2
> node1:/exp1/brick1 node2:/exp2/brick2
> node1:/exp1/brick3 node2:/exp2/brick4
> node1:/exp1/brick5 node2:/exp2/brick6
> 
> And now i want to add another new node to the gluster and
> node2:/exp2/brick2 replicated with the new brick node3"/exp3/brick7
> 
> with the condition cannot add brick to existing node 1 and node 2.

What version of glusterfs are you using?

If I understand correctly, the current volume configuration is as follows:

node1:/exp1/brick1 node2:/exp2/brick2 (replica of each other)
node1:/exp1/brick3 node2:/exp2/brick4
node1:/exp1/brick5 node2:/exp2/brick6

And you want to change it to:

node2:/exp2/brick2 node3:/exp3/brick7 (replica of each other)
node1:/exp1/brick3 node2:/exp2/brick4
node1:/exp1/brick5 node2:/exp2/brick6

You can do the following:
gluster peer probe node 3 (run this from one of the nodes in your cluster)
gluster v replace-brick  node1:/exp1/brick1 node3:/exp3/brick7 commit 
force

This should now make node2:/exp2/brick2 and node3:/exp3/brick7 as replica of 
each other.

You might have to do some additional work for heal to be triggered based on the 
glusterfs version
you are using. So do mention the gluster version being used.

Let me know if my understanding of the requirement given was incorrect.

Hope this helps.
> 
> May i know how to do this ?
> 
> Appreciate for the advice.
> 
> Thanks
> Jason
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Clarification on common tasks

[Anuradha Talur]

- Original Message -
> From: "Gandalf Corvotempesta" 
> To: "Lindsay Mathieson" 
> Cc: "Anuradha Talur" , "gluster-users" 
> 
> Sent: Friday, August 12, 2016 2:48:37 AM
> Subject: Re: [Gluster-users] Clarification on common tasks
> 
> 2016-08-11 16:13 GMT+02:00 Gandalf Corvotempesta
> :
> > # gluster volume heal gv0 info heal-failed
> > Gathering list of heal failed entries on volume gv0 has been
> > unsuccessful on bricks that are down. Please check if all brick
> > processes are running.
> 
> Healing is now complete:
> 
> # gluster volume heal gv0 info
> Brick 1.2.3.112:/export/brick1/brick
> Status: Connected
> Number of entries: 0
> 
> Brick 1.2.3.113:/export/sdb1/brick
> Status: Connected
> Number of entries: 0
> 
> Brick 1.2.3.114:/export/sdb1/brick
> Status: Connected
> Number of entries: 0
> 
> I remember that there was some errors during healing, so i've tried to run
> this:
> 
> # gluster volume heal gv0 info heal-failed
> Gathering list of heal failed entries on volume gv0 has been
> unsuccessful on bricks that are down. Please check if all brick
> processes are running.
Actually info healed and heal-failed are deprecated. Recently some changes
were made due to which wrong error message is being given to the users.
A bug has been raised for the same and will be worked on for the next releases.
> 
> but all bricks are up and running:
> 
> # gluster volume status
> Status of volume: gv0
> Gluster process TCP Port  RDMA Port  Online  Pid
> --
> Brick 1.2.3.112:/export/brick1/brick49153 0  Y   11222
> Brick 1.2.3.113:/export/sdb1/brick  49152 0  Y   5786
> Brick 1.2.3.114:/export/sdb1/brick  49152 0  Y   920
> Self-heal Daemon on localhost   N/A   N/AY   11227
> Self-heal Daemon on 1.2.3.113   N/A   N/AY   31559
> Self-heal Daemon on 1.2.3.114   N/A   N/AY   26173
> 
> Task Status of Volume gv0
> --
> There are no active volume tasks
> 
> 
> 
> Why i'm unable to see what happened during healing ? I can't use any
> of "healed" or "heal-failed" arguments.
> "split-brain" works.

> 
> Any suggestion? I'm curious to see why some files triggered an heal failure.
You can get glustershd logs to see why heals failed. If you need help with it,
please give us the log files.
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Clarification on common tasks

[Anuradha Talur]

You are correct. Needs to be changed. Will edit it in the next few days.

- Original Message -
> From: "Gandalf Corvotempesta" 
> To: "Anuradha Talur" 
> Cc: "gluster-users" 
> Sent: Thursday, August 11, 2016 7:58:40 PM
> Subject: Re: [Gluster-users] Clarification on common tasks
> 
> 2016-08-11 16:25 GMT+02:00 Anuradha Talur :
> > The replace-brick you did, mentioned in the previous mails was correct and
> > fine.
> > You said you have different names for old and new brick, so it works.
> > setfattr is *not* required in this case.
> >
> > In the above case that you have quoted, I'm talking when brick names have
> > to be the same (if you have that
> > requirement). There were 2 setfattrs that were needed. Anyway, sorry about
> > the confusion.
> > As brick names are different, it doesn't affect your testcase.
> 
> Anyway, the suggested docs page is wrong. It has many bad commands, in
> example:
> 
> "gluster volume heal info failed"
> 
> this doesn't exist. The right command should be
> "gluster volume  heal info heal-failed
> 
> Also, the replace-brick in a replicated volumes, makes use of
> "setfattr" and also ask for a directory creation/removal (step 4):
> 
> mkdir /mnt/r2/
> rmdir /mnt/r2/
> setfattr -n trusted.non-existent-key -v abc /mnt/r2
> setfattr -x trusted.non-existent-key  /mnt/r2
> 
> i didn't this and seems to work fine.
> 
> I think that this doc page should be rewrote, there is much confusion
> and unneeded steps.
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Clarification on common tasks

[Anuradha Talur]

- Original Message -
> From: "Gandalf Corvotempesta" 
> To: "Anuradha Talur" 
> Cc: "gluster-users" 
> Sent: Thursday, August 11, 2016 7:46:37 PM
> Subject: Re: [Gluster-users] Clarification on common tasks
> 
> 2016-08-11 16:13 GMT+02:00 Anuradha Talur :
> > There was a document that I'm not able to locate right now.
> > The first step after mounting the brick was to set volume ID using
> > setfattr -n trusted.glusterfs.volume-id -v  .
> > I think there were more steps, I will update once I find the doc.
> > Once all the required xattrs are set, gluster v start force was supposed to
> > be done.
> 
> This is a test environment so I can break everything everytime.
> I did (only once) "replace-brick" by using a new mount point, without
> setting any fattr and
> brick process was brought up immediatly and healing is going on.
> 
> Is setfattr mandatory ?

The replace-brick you did, mentioned in the previous mails was correct and fine.
You said you have different names for old and new brick, so it works.
setfattr is *not* required in this case.

In the above case that you have quoted, I'm talking when brick names have to be 
the same (if you have that
requirement). There were 2 setfattrs that were needed. Anyway, sorry about the 
confusion.
As brick names are different, it doesn't affect your testcase.
>

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Clarification on common tasks

[Anuradha Talur]

- Original Message -
> From: "Anuradha Talur" 
> To: "Gandalf Corvotempesta" 
> Cc: "gluster-users" 
> Sent: Thursday, August 11, 2016 5:47:12 PM
> Subject: Re: [Gluster-users] Clarification on common tasks
> 
> 
> 
> - Original Message -
> > From: "Gandalf Corvotempesta" 
> > To: "gluster-users" 
> > Sent: Thursday, August 11, 2016 2:43:34 PM
> > Subject: [Gluster-users] Clarification on common tasks
> > 
> > I would like to make some clarification on common tasks needed by
> > gluster administrators.
> > 
> > A) Let's assume a disk/brick is failed (or is going to fail) and I
> > would like to replace.
> > Which is the proper way to do so with no data loss or downtime ?
> > 
> > Looking on mailing list, seems to be the following:
> > 
> > 1) kill the brick process (how can I ensure which is the brick process
> > to kill)? I have the following on a test cluster (with just one
> > brick):
> > # ps ax -o command | grep gluster
> > /usr/sbin/glusterfsd -s 1.2.3.112 --volfile-id
> > gv0.1.2.3.112.export-sdb1-brick -p
> > /var/lib/glusterd/vols/gv0/run/1.2.3.112-export-sdb1-brick.pid -S
> > /var/run/gluster/27555a68c738d9841879991c725e92e0.socket --brick-name
> > /export/sdb1/brick -l /var/log/glusterfs/bricks/export-sdb1-brick.log
> > --xlator-option
> > *-posix.glusterd-uuid=c97606ac-f6b7-4fdc-a401-6c2d04dd73a8
> > --brick-port 49152 --xlator-option gv0-server.listen-port=49152
> > /usr/sbin/glusterd -p /var/run/glusterd.pid
> > /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p
> > /var/lib/glusterd/glustershd/run/glustershd.pid -l
> > /var/log/glusterfs/glustershd.log -S
> > /var/run/gluster/5f3713389b19487b6c7d6efca6102987.socket
> > --xlator-option
> > *replicate*.node-uuid=c97606ac-f6b7-4fdc-a401-6c2d04dd73a8
> > 
> > which is the "brick process" ?
> > 
> As clarified by Lindsay, you can find the correct brick to kill
> by mapping output of gluster v status with the brick that has failed.
> > 2) unmount the brick, in example:
> > unmount /dev/sdc
> > 
> > 3) remove the failed disk
> > 
> > 4) insert the new disk
> > 5) create an XFS filesystem on the new disk
> > 6) mount the new disk where the previous one was
> > 7) add the new brick to the gluster. How ?
> > 8) run "gluster v start force".
> 
> If this is a replicate volume then only these steps are not enough.
> 
> If you are okay with the mount of new and previous brick to be
> different-
> 
> After you mount the new-brick, you will have to run
> gluster v replace-brick  old_brick new_brick commit force.
> 
> By doing this you would be adding new brick to the gluster cluster
> and also letting the replicate translator know that
> the brick has been replaced and that it needs to be healed.
> 
> Once this is done, self-heal-daemon will start the healing process
> automatically.
> 
> If this step is done, you wouldn't have to run step 8 - gluster v start
> force.
> As replace-brick command takes care of bringing the new brick up.
> 
> In case you want to mount the new brick to the same path as the previous one,
> then after step 6, I'd suggest you:
> a) Create a dummy-non-existent-dir under '/' of the volume's mount point.
> b) create a dummy-non-existent-xattr on '/' of the volume's mount point.
> The above steps are basically again letting the replicate translator know
> that some healing has to be done on the brick that is down. replace-brick
> command would do this for you but as it doesn't support same path for old
> and new brick, this is a work-around. (Support for replacing bricks with
> same path will be provided in upcoming releases. It is being worked on.)
> 
Sorry, there was a mistake in this mail.
As I said, replace-brick can't be used when old and new path are the same.
And I by mistake suggested replace-brick after all the steps again!

There was a document that I'm not able to locate right now.
The first step after mounting the brick was to set volume ID using
setfattr -n trusted.glusterfs.volume-id -v  .
I think there were more steps, I will update once I find the doc.
Once all the required xattrs are set, gluster v start force was supposed to be 
done.

start force needs to be done here as volume is already in start state but the
management daemon, glusterd, is not aware that the failed brick has been fixed
with new disks. start force is a way of letting glusterd know that there is
a brick that is down but needs to be started. This will be done without
affecting the existi

Re: [Gluster-users] Clarification on common tasks

[Anuradha Talur]

- Original Message -
> From: "Gandalf Corvotempesta" 
> To: "gluster-users" 
> Sent: Thursday, August 11, 2016 2:43:34 PM
> Subject: [Gluster-users] Clarification on common tasks
> 
> I would like to make some clarification on common tasks needed by
> gluster administrators.
> 
> A) Let's assume a disk/brick is failed (or is going to fail) and I
> would like to replace.
> Which is the proper way to do so with no data loss or downtime ?
> 
> Looking on mailing list, seems to be the following:
> 
> 1) kill the brick process (how can I ensure which is the brick process
> to kill)? I have the following on a test cluster (with just one
> brick):
> # ps ax -o command | grep gluster
> /usr/sbin/glusterfsd -s 1.2.3.112 --volfile-id
> gv0.1.2.3.112.export-sdb1-brick -p
> /var/lib/glusterd/vols/gv0/run/1.2.3.112-export-sdb1-brick.pid -S
> /var/run/gluster/27555a68c738d9841879991c725e92e0.socket --brick-name
> /export/sdb1/brick -l /var/log/glusterfs/bricks/export-sdb1-brick.log
> --xlator-option
> *-posix.glusterd-uuid=c97606ac-f6b7-4fdc-a401-6c2d04dd73a8
> --brick-port 49152 --xlator-option gv0-server.listen-port=49152
> /usr/sbin/glusterd -p /var/run/glusterd.pid
> /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p
> /var/lib/glusterd/glustershd/run/glustershd.pid -l
> /var/log/glusterfs/glustershd.log -S
> /var/run/gluster/5f3713389b19487b6c7d6efca6102987.socket
> --xlator-option
> *replicate*.node-uuid=c97606ac-f6b7-4fdc-a401-6c2d04dd73a8
> 
> which is the "brick process" ?
> 
As clarified by Lindsay, you can find the correct brick to kill
by mapping output of gluster v status with the brick that has failed.
> 2) unmount the brick, in example:
> unmount /dev/sdc
> 
> 3) remove the failed disk
> 
> 4) insert the new disk
> 5) create an XFS filesystem on the new disk
> 6) mount the new disk where the previous one was
> 7) add the new brick to the gluster. How ?
> 8) run "gluster v start force".

If this is a replicate volume then only these steps are not enough.

If you are okay with the mount of new and previous brick to be
different-

After you mount the new-brick, you will have to run
gluster v replace-brick  old_brick new_brick commit force.

By doing this you would be adding new brick to the gluster cluster
and also letting the replicate translator know that
the brick has been replaced and that it needs to be healed.

Once this is done, self-heal-daemon will start the healing process
automatically.

If this step is done, you wouldn't have to run step 8 - gluster v start force.
As replace-brick command takes care of bringing the new brick up.

In case you want to mount the new brick to the same path as the previous one,
then after step 6, I'd suggest you:
a) Create a dummy-non-existent-dir under '/' of the volume's mount point.
b) create a dummy-non-existent-xattr on '/' of the volume's mount point.
The above steps are basically again letting the replicate translator know
that some healing has to be done on the brick that is down. replace-brick
command would do this for you but as it doesn't support same path for old
and new brick, this is a work-around. (Support for replacing bricks with
same path will be provided in upcoming releases. It is being worked on.)

Once this is done, run the replace-brick command mentioned above.
This should add some volume uuids to the brick, start the brick and then
trigger heal to new brick.
> 
> Why should I need the step 8? If the volume is already started and
> working (remember that I would like to change disk with no downtime,
> thus i can't stop the volume), why should I "start" it again ?
> 
> 
> 
> 
> B) let's assume I would like to add a bounch of new bricks on existing
> servers. Which is the proper procedure to do so?

Do you mean increase the capacity of the volume by adding new bricks?
You can use gluster v add-brick new-brick(s)

The options provided to add-brick are going to vary based on how you plan to
add these bricks (whether you want to increase replica-count or add a new
replica set etc).
> 
> 
> Ceph has a good documentation page where some common tasks are explained:
> http://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/
> i've not found anything similiar in gluster.

I found this for glusterFS :
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-brick

Hope this helps.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Failed file system

[Anuradha Talur]

Replace brick commit force command can be used. If you are on glusterfs 3.7.3 
and above, self-heal will be
automatically triggered from good bricks to the newly added brick.
But you can't replace a brick on the same path as before, your new brick path 
will
have to be different that the existing ones in the volume.

- Original Message -
> From: "Mahdi Adnan" 
> To: "Andres E. Moya" , "gluster-users" 
> 
> Sent: Thursday, August 4, 2016 1:25:59 AM
> Subject: Re: [Gluster-users] Failed file system
> 
> Hi,
> 
> I'm not expert in Gluster but, i think it would be better to replace the
> downed brick with a new one.
> Maybe start from here;
> 
> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-brick
> 
> 
> --
> 
> Respectfully
> Mahdi A. Mahdi
> 
> 
> 
> 
> Date: Wed, 3 Aug 2016 15:39:35 -0400
> From: am...@moyasolutions.com
> To: gluster-users@gluster.org
> Subject: Re: [Gluster-users] Failed file system
> 
> Does anyone else have input?
> 
> we are currently only running off 1 node and one node is offline in replicate
> brick.
> 
> we are not experiencing any downtime because the 1 node is up.
> 
> I do not understand which is the best way to bring up a second node.
> 
> Do we just re create a file system on the node that is down and the mount
> points and allow gluster to heal( my concern with this is whether the node
> that is down will some how take precedence and wipe out the data on the
> healthy node instead of vice versa)
> 
> Or do we fully wipe out the config on the node that is down, re create the
> file system and re add the node that is down into gluster using the add
> brick command replica 3, and then wait for it to heal then run the remove
> brick command for the failed brick
> 
> which would be the safest and easiest to accomplish
> 
> thanks for any input
> 
> 
> 
> 
> From: "Leno Vo" 
> To: "Andres E. Moya" 
> Cc: "gluster-users" 
> Sent: Tuesday, August 2, 2016 6:45:27 PM
> Subject: Re: [Gluster-users] Failed file system
> 
> if you don't want any downtime (in the case that your node 2 really die), you
> have to create a new gluster san (if you have the resources of course, 3
> nodes as much as possible this time), and then just migrate your vms (or
> files), therefore no downtime but you have to cross your finger that the
> only node will not die too... also without sharding the vm migration
> especially an rdp one, will be slow access from users till it migrated.
> 
> you have to start testing sharding, it's fast and cool...
> 
> 
> 
> 
> On Tuesday, August 2, 2016 2:51 PM, Andres E. Moya 
> wrote:
> 
> 
> couldnt we just add a new server by
> 
> gluster peer probe
> gluster volume add-brick replica 3 (will this command succeed with 1 current
> failed brick?)
> 
> let it heal, then
> 
> gluster volume remove remove-brick
> 
> From: "Leno Vo" 
> To: "Andres E. Moya" , "gluster-users"
> 
> Sent: Tuesday, August 2, 2016 1:26:42 PM
> Subject: Re: [Gluster-users] Failed file system
> 
> you need to have a downtime to recreate the second node, two nodes is
> actually not good for production and you should have put raid 1 or raid 5 as
> your gluster storage, when you recreate the second node you might try
> running some VMs that need to be up and rest of vm need to be down but stop
> all backup and if you have replication, stop it too. if you have 1G nic,
> 2cpu and less 8Gram, then i suggest all turn off the VMs during recreation
> of second node. someone said if you have sharding with 3.7.x, maybe some vip
> vm can be up...
> 
> if it just a filesystem, then just turn off the backup service until you
> recreate the second node. depending on your resources and how big is your
> storage, it might be hours to recreate it and even days...
> 
> here's my process on recreating the second or third node (copied and modifed
> from the net),
> 
> #make sure partition is already added
> This procedure is for replacing a failed server, IF your newly installed
> server has the same hostname as the failed one:
> 
> (If your new server will have a different hostname, see this article
> instead.)
> 
> For purposes of this example, the server that crashed will be server3 and the
> other servers will be server1 and server2
> 
> On both server1 and server2, make sure hostname server3 resolves to the
> correct IP address of the new replacement server.
> #On either server1 or server2, do
> grep server3 /var/lib/glusterd/peers/*
> 
> This will return a uuid followed by ":hostname1=server3"
> 
> #On server3, make sure glusterd is stopped, then do
> echo UUID={uuid from previous step}>/var/lib/glusterd/glusterd.info
> 
> #actual testing below,
> [root@node1 ~]# cat /var/lib/glusterd/glusterd.info
> UUID=4b9d153c-5958-4dbe-8f91-7b5002882aac
> operating-version=30710
> #the second line is new. maybe not needed...
> 
> On server3:
> make sure that all brick directories are created/mounted
> start glusterd
> peer probe one of 

Re: [Gluster-users] 3.7.13, index healing broken?

[Anuradha Talur]

- Original Message -
> From: "Dmitry Melekhov" 
> To: "Pranith Kumar Karampuri" 
> Cc: "gluster-users" 
> Sent: Tuesday, July 12, 2016 9:27:17 PM
> Subject: Re: [Gluster-users] 3.7.13, index healing broken?
> 
> 
> 
> 12.07.2016 17:39, Pranith Kumar Karampuri пишет:
> 
> 
> 
> Wow, what are the steps to recreate the problem?
> 
> just set file length to zero, always reproducible.
> 
If you are setting the file length to 0 on one of the bricks (looks like
that is the case), it is not a bug.

Index heal relies on failures seen from the mount point(s)
to identify the files that need heal. It won't be able to recognize any file
modification done directly on bricks. Same goes for heal info command which
is the reason heal info also shows 0 entries.

Heal full on the other hand will individually compare certain aspects of all
files/dir to identify files to be healed. This is why heal full works in this 
case
but index heal doesn't.
> 
> 
> 
> 
> On Tue, Jul 12, 2016 at 3:09 PM, Dmitry Melekhov < d...@belkam.com > wrote:
> 
> 
> 
> 12.07.2016 13:33, Pranith Kumar Karampuri пишет:
> 
> 
> 
> What was "gluster volume heal  info" showing when you saw this
> issue?
> 
> just reproduced :
> 
> 
> [root@father brick]# > gstatus-0.64-3.el7.x86_64.rpm
> 
> [root@father brick]# gluster volume heal pool
> Launching heal operation to perform index self heal on volume pool has been
> successful
> Use heal info commands to check status
> [root@father brick]# gluster volume heal pool info
> Brick father:/wall/pool/brick
> Status: Connected
> Number of entries: 0
> 
> Brick son:/wall/pool/brick
> Status: Connected
> Number of entries: 0
> 
> Brick spirit:/wall/pool/brick
> Status: Connected
> Number of entries: 0
> 
> [root@father brick]#
> 
> 
> 
> 
> 
> 
> 
> On Mon, Jul 11, 2016 at 3:28 PM, Dmitry Melekhov < d...@belkam.com > wrote:
> 
> 
> Hello!
> 
> 3.7.13, 3 bricks volume.
> 
> inside one of bricks:
> 
> [root@father brick]# ls -l gstatus-0.64-3.el7.x86_64.rpm
> -rw-r--r-- 2 root root 52268 июл 11 13:00 gstatus-0.64-3.el7.x86_64.rpm
> [root@father brick]#
> 
> 
> [root@father brick]# > gstatus-0.64-3.el7.x86_64.rpm
> [root@father brick]# ls -l gstatus-0.64-3.el7.x86_64.rpm
> -rw-r--r-- 2 root root 0 июл 11 13:54 gstatus-0.64-3.el7.x86_64.rpm
> [root@father brick]#
> 
> so now file has 0 length.
> 
> try to heal:
> 
> 
> 
> [root@father brick]# gluster volume heal pool
> Launching heal operation to perform index self heal on volume pool has been
> successful
> Use heal info commands to check status
> [root@father brick]# ls -l gstatus-0.64-3.el7.x86_64.rpm
> -rw-r--r-- 2 root root 0 июл 11 13:54 gstatus-0.64-3.el7.x86_64.rpm
> [root@father brick]#
> 
> 
> nothing!
> 
> [root@father brick]# gluster volume heal pool full
> Launching heal operation to perform full self heal on volume pool has been
> successful
> Use heal info commands to check status
> [root@father brick]# ls -l gstatus-0.64-3.el7.x86_64.rpm
> -rw-r--r-- 2 root root 52268 июл 11 13:00 gstatus-0.64-3.el7.x86_64.rpm
> [root@father brick]#
> 
> 
> full heal is OK.
> 
> But, self-heal is doing index heal according to
> 
> http://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Developer-guide/afr-self-heal-daemon/
> 
> Is this bug?
> 
> 
> As far as I remember it worked in 3.7.10
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> --
> Pranith
> 
> 
> 
> 
> --
> Pranith
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Expand distributed replicated volume

[Anuradha Talur]

- Original Message -
> From: "Gandalf Corvotempesta" 
> To: "Pranith Kumar Karampuri" 
> Cc: "Anuradha Talur" , "gluster-users" 
> 
> Sent: Tuesday, July 12, 2016 4:32:57 PM
> Subject: Re: [Gluster-users] Expand distributed replicated volume
> 
> 2016-07-12 12:56 GMT+02:00 Pranith Kumar Karampuri :
> > For adding new bricks into a replica set you need each brick in the replica
> > set to be from different machine. So you can't add all bricks directly from
> > just one machine. So how do you get the extra bricks that can be combined
> > with the bricks on new machine? From old servers. How do you get those?
> > Using replace-brick. I hope with this info if you can go through the
> > example
> > once again, it probably may clarify your doubts. Please feel free to ask
> > any
> > questions you may have.
> 
> I'm new to gluster, so let me try to explain and sorry for my dump question.
> 
> If I have a replica made with:
> 
> (s1b1, s2b1, s3b1)
> (s1b2, s2b2, s3b2)
> (s1b3, s2b3, s3b3)
> 
> Why I cant add a new server getting this result:
> 
> (s1b1, s2b1, s3b1, s4b1)
> (s1b2, s2b2, s3b2, s4b2)
> (s1b3, s2b3, s3b3, s4b3)
> 
> ?? In this case , I've added a new server with 3 bricks.
> 

Some clarification here :

In Pranith's example, each replica set is represented as comma separated values.
Meaning, (s1b1, s2b1, s3b1) will all contain the same data replicated thrice.

If you add new server with 3 bricks like you've represented :
(s1b1, s2b1, s3b1, s4b1) as per Pranith's representation it means 
there are 4 bricks which are replica of each other. Meaning they all have the 
same data,
so there is 4 fold replication here.

>From my understanding, you say that your job should only be to add a node with 
>3
bricks and gluster should internally take care of distributing the data. This 
is a
perfectly valid requirement, but right now, gluster makes its replica set based 
on the
order of bricks specified in the cli. When you perform add-brick, and mention 3 
new
bricks, they will become replica of each other automatically. Which is why 
steps mentioned
by Pranith will help in solving your issue.

I'm repeating the same example that Pranith gave. Skip the next section if you 
have already
understood.
What Pranith is trying to say is, at the end you would have technically added 3 
bricks
to your cluster (all residing in the same node) but, the distribution will not 
happen automatically.
It will be your responsibility (until heketi is usable to do the same) to make 
sure that the
newly added bricks are arranged such that you don't lose redundancy in case of 
node failures.

Taking your example here:

Step 1) You have :
(s1b1, s2b1, s3b1)
(s1b2, s2b2, s3b2)
(s1b3, s2b3, s3b3)

Step 2) You want to add 3 more disks/bricks to increase volume size.
Condition is that you want to add only one new node. As you don't want to lose
redundancy in case of node failure, you have to ensure that no 2 bricks which
contain the same data reside in the same node. As this arrangement is not taken 
care by
gluster yet, using the same resources, there is only one way to do it:

Empty one brick from each existing node so that we have gained distribution of 
our
bricks to all nodes. And now substitute each of these bricks with the 3 new 
bricks
from the latest node. This is done using replace-brick option.

At this point, your cluster will look like :
(s4b1, s2b1, s3b1)
(s1b2, s4b2, s3b2)
(s1b3, s2b3, s4b3)

So instead of s4b1, s4b2 and s4b3 being free, you have s1b1, s2b2 and s3b3 free.

Step 3)
You can now add the three free bricks to form a new 3 way replica set. Do this,
using add-brick command.

This way, you have essentially added only one node containing 3 bricks, and also
achieved redundancy.

I realize I just repeated the same example in different words but I hope this 
answers
your questions. So do let us know if something is not clear.

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Expand distributed replicated volume

[Anuradha Talur]

- Original Message -
> From: "Anuradha Talur" 
> To: "Gandalf Corvotempesta" 
> Cc: "gluster-users" 
> Sent: Tuesday, July 12, 2016 1:16:00 PM
> Subject: Re: [Gluster-users] Expand distributed replicated volume
> 
> 
> 
> - Original Message -----
> > From: "Gandalf Corvotempesta" 
> > To: "Anuradha Talur" 
> > Cc: "gluster-users" 
> > Sent: Thursday, July 7, 2016 10:14:46 PM
> > Subject: Re: [Gluster-users] Expand distributed replicated volume
> > 
> > Il 07 lug 2016 13:18, "Anuradha Talur"  ha scritto:
> > > Node to the cluster? You can add one per time.
> > >
> > > If you are asking about adding the number of bricks to a volume,
> > > in case of replica 3, if you want to keep the replica count same (3)
> > > then you have to add 3 bricks. These three bricks will be taken as a
> > > new replica group added to the volume.
> > >
> > > If you want to increase the replica count to 4, then you will have to
> > > add one brick each to all the existing replica groups. So that the
> > > replica groups now have 4 bricks each.
> > >
> > > Hope this resolves your query.
> > 
> > Lets assume a 3 nodes cluster,  replica 3, 6 1tb disks per node, 1 brick
> > per disk.
> > 
> > i would like to increase space and thus add a new node.
> > Can i add a single node with 3 disks/bricks and automatically rebalance?
> 
> Yes you can add single node with 3 bricks. But, given that you are keeping
> the replica count
> same, these three bricks will be replica of each other. It is not so useful
> in case of node
> failures/shutdown.
> 
> > In this case, gluster moves data around to use the maximum number or disks
> > for performance reasons?
> > 
> There are 2 ways to trigger rebalance after adding brick,
> 1) without force option : This will not move data. But will add linkto files.
> Think of them as symbolic links. Files will reside in the bricks where they
> already exist,
> but will have links in the new bricks.
> 2) With force option : This will actually move the data from old bricks to
> new bricks.
> 
> Both these operations will be done only for those files that need to
> be migrated.
> 
> The new files will either
> 1) be created in new bricks if they hash to new bricks.
> or
> 2) be created in new bricks even though they hash to old bricks and old
> bricks will
> have linkto files.
> 
> If there is no space in old bricks to create linkto files too, the operation
> will
> fail with ENOSPC (no space available).
> 
> I'm cc'ing developers from DHT team in case you have more questions with
> respect to
> rebalance.
Sorry for spam. Forgot to CC.
> > We start with 18tb raw, 18 disks, 6tb usable, 100% used (6tb on 6 usable
> > disks)
> > After adding the new node with 3 disks we will end with 7tb usable.
> > all disks would be used as 6tb/7=0.85% right?
> > 
> 
> --
> Thanks,
> Anuradha.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Expand distributed replicated volume

[Anuradha Talur]

- Original Message -
> From: "Gandalf Corvotempesta" 
> To: "Anuradha Talur" 
> Cc: "gluster-users" 
> Sent: Thursday, July 7, 2016 10:14:46 PM
> Subject: Re: [Gluster-users] Expand distributed replicated volume
> 
> Il 07 lug 2016 13:18, "Anuradha Talur"  ha scritto:
> > Node to the cluster? You can add one per time.
> >
> > If you are asking about adding the number of bricks to a volume,
> > in case of replica 3, if you want to keep the replica count same (3)
> > then you have to add 3 bricks. These three bricks will be taken as a
> > new replica group added to the volume.
> >
> > If you want to increase the replica count to 4, then you will have to
> > add one brick each to all the existing replica groups. So that the
> > replica groups now have 4 bricks each.
> >
> > Hope this resolves your query.
> 
> Lets assume a 3 nodes cluster,  replica 3, 6 1tb disks per node, 1 brick
> per disk.
> 
> i would like to increase space and thus add a new node.
> Can i add a single node with 3 disks/bricks and automatically rebalance?

Yes you can add single node with 3 bricks. But, given that you are keeping the 
replica count
same, these three bricks will be replica of each other. It is not so useful in 
case of node
failures/shutdown.

> In this case, gluster moves data around to use the maximum number or disks
> for performance reasons?
> 
There are 2 ways to trigger rebalance after adding brick,
1) without force option : This will not move data. But will add linkto files.
Think of them as symbolic links. Files will reside in the bricks where they 
already exist,
but will have links in the new bricks.
2) With force option : This will actually move the data from old bricks to new 
bricks.

Both these operations will be done only for those files that need to
be migrated.

The new files will either
1) be created in new bricks if they hash to new bricks.
or
2) be created in new bricks even though they hash to old bricks and old bricks 
will
have linkto files.

If there is no space in old bricks to create linkto files too, the operation 
will
fail with ENOSPC (no space available).

I'm cc'ing developers from DHT team in case you have more questions with 
respect to
rebalance.
> We start with 18tb raw, 18 disks, 6tb usable, 100% used (6tb on 6 usable
> disks)
> After adding the new node with 3 disks we will end with 7tb usable.
> all disks would be used as 6tb/7=0.85% right?
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Expand distributed replicated volume

[Anuradha Talur]

- Original Message -
> From: "Gandalf Corvotempesta" 
> To: "gluster-users" 
> Sent: Wednesday, July 6, 2016 10:43:05 PM
> Subject: [Gluster-users] Expand distributed replicated volume
> 
> Let's assume a distributed replicated (replica 3) volume, with sharding
> enabled.
> Can I add 1 node per time or I have to add 3 nodes every time ?

Node to the cluster? You can add one per time.

If you are asking about adding the number of bricks to a volume,
in case of replica 3, if you want to keep the replica count same (3)
then you have to add 3 bricks. These three bricks will be taken as a
new replica group added to the volume.

If you want to increase the replica count to 4, then you will have to
add one brick each to all the existing replica groups. So that the
replica groups now have 4 bricks each.

Hope this resolves your query.

> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.7.12 disaster

[Anuradha Talur]

Lindsay,

Did you see any problems in the setup before you set those options?

Also, could you please share glusterd and glfsheal logs before you revert to 
3.7.11,
so that it can be analyzed?

- Original Message -
> From: "Lindsay Mathieson" 
> To: "gluster-users" 
> Sent: Wednesday, June 29, 2016 2:07:30 PM
> Subject: Re: [Gluster-users] 3.7.12 disaster
> 
> On 29 June 2016 at 18:30, Lindsay Mathieson 
> wrote:
> > Same problem again. VM froze and heal info timed out with "Not able to
> > fetch volfile from glusterd". I'm going to have to revert to 3.7.11
> 
> 
> Heal process seems to be stuck at the following:
> 
> gluster v heal datastore4 info
> Brick vnb.proxmox.softlog:/tank/vmdata/datastore4
> Status: Connected
> Number of entries: 0
> 
> Brick vng.proxmox.softlog:/tank/vmdata/datastore4
>  - Possibly undergoing heal
> 
> Status: Connected
> Number of entries: 1
> 
> Brick vna.proxmox.softlog:/tank/vmdata/datastore4
>  - Possibly undergoing heal
> 
> Status: Connected
> Number of entries: 1
> 
> I'm on my home now, will be offline for a couple of hours. But for now
> my cluster is offline.
> 
> --
> Lindsay
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Data Replication

[Anuradha Talur]

- Original Message -
> From: "TomK" 
> To: gluster-users@gluster.org
> Sent: Thursday, June 9, 2016 11:13:36 AM
> Subject: [Gluster-users] Data Replication
> 
> Hey All,
> 
> Still new and learning GlusterFS.  Thanks for bearing with me.
> 
> I cannot writes files from nodes to the glusterfs and see them
> replicated.  I can only write from the master node opennebula01 and see
> the file distributed and replicated.  How do I replicate from the nodes?
> Or am I just asking something that isn't part of the design:
> 

Hi,

Looks like you are doing something that is not part of the design.

This is what I understood from your mail :

glusterv01 and glusterv02 are the nodes that constitute your glusterfs volume.
opennebula01 is where you have "mounted" this gluster volume. You can 
write/create
files only from this mount such that the files get replicate to both
glusterv01 and glusterv02 nodes. If you directly write into these nodes,
the files won't get replicated.

Does this answer your question? Let me know if it does not.

> [root@mdskvm-p01 glusterv01]# echo "from mdskvm-p01" > from.txt
> [root@mdskvm-p01 glusterv01]# ls -altri
> total 6792232
> 146 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19
> ouch-10.iso
> 145 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19
> ouch-09.iso
> 144 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19
> ouch-08.iso
> 143 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19
> ouch-07.iso
> 142 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19
> ouch-06.iso
> 141 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19
> ouch-05.iso
> 140 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19
> ouch-04.iso
> 136 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19
> CentOS-7-x86_64-Minimal-1511.iso
> 137 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:20
> ouch-01.iso
> 139 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:21
> ouch-03.iso
> 138 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:21
> ouch-02.iso
> 2147483776 drwxr-xr-x. 3 root root24 Jun  9 01:29 .trashcan
> 128 drwxr-xr-x. 3 oneadmin oneadmin23 Jun  9 01:30 ..
>   536871104 drw---. 9 root root  4096 Jun  9 01:35
> .glusterfs
>   402653377 -rw-r--r--. 2 root root 3 Jun  9 01:35 test.txt
>   402653378 -rw-r--r--. 1 root root16 Jun  9 01:36 from.txt
>   402653376 drwxr-xr-x. 4 oneadmin oneadmin  4096 Jun  9 01:36 .
> [root@mdskvm-p01 glusterv01]#
> 
> 
> [root@mdskvm-p02 glusterv02]# ls -altri
> total 4
>  64 drwxr-xr-x. 3 root root  23 Jun  8 19:38 ..
>  67 drw---. 9 root root 188 Jun  8 19:45 .glusterfs
> 3221225600 drwxr-xr-x. 4 oneadmin oneadmin  54 Jun  8 19:45 .
> 3221225604 -rw-r--r--. 2 root root   3 Jun  8 19:45 test.txt
>  74 drwxr-xr-x. 3 root root  24 Jun  9  2016 .trashcan
> [root@mdskvm-p02 glusterv02]#
> 
> 
> 
> [root@opennebula01 0]# ls -altri
> total 8
> 68736015 drwxr-x---. 7 oneadmin oneadmin   60 May  9 00:19 ..
> 1 drwxr-xr-x. 4 oneadmin oneadmin 4096 Jun  9 01:29 .
> 5 drwxr-xr-x. 3 root root 4096 Jun  9 01:29 .trashcan
> [root@opennebula01 0]#
> [root@opennebula01 0]#
> [root@opennebula01 0]#
> [root@opennebula01 0]#
> [root@opennebula01 0]# echo "yo" > test.txt
> [root@opennebula01 0]# ls -altri
> total 9
>  68736015 drwxr-x---. 7 oneadmin oneadmin   60 May  9 00:19 ..
> 1 drwxr-xr-x. 4 oneadmin oneadmin 4096 Jun  8 20:45 .
> 10502401602053818307 -rw-r--r--. 1 root root3 Jun  8 20:45
> test.txt
> 5 drwxr-xr-x. 3 root root 4096 Jun  9 01:29
> .trashcan
> [root@opennebula01 0]#
> [root@opennebula01 0]#
> 
> 
> 
> Cheers,
> Tom K.
> -
> 
> Living on earth is expensive, but it includes a free trip around the sun.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] CentOS 7.2 + Gluster 3.6.3 volume stuck in heal

[Anuradha Talur]

Kingsley,

Looking at the statedumps provided, I can see blocked locks.
Could you try restarting self-heal daemon and then getting heal info again?

To restart self-heal-daemon you will have to run:
gluster volume start  force

- Original Message -
> From: "Kingsley" 
> To: gluster-users@gluster.org
> Sent: Friday, May 27, 2016 3:41:15 PM
> Subject: Re: [Gluster-users] CentOS 7.2 + Gluster 3.6.3 volume stuck in heal
> 
> Can nobody help with this?
> 
> Cheers,
> Kingsley.
> 
> On Fri, 2016-05-20 at 17:36 +0100, Kingsley wrote:
> > Hi,
> > 
> > We've got a volume that has been stuck in "Possibly undergoing heal"
> > status for several months. I would like to upgrade gluster to a newer
> > version but I would feel safer if we could get the volume fixed first.
> > 
> > Some info:
> > 
> > # gluster volume info voicemail
> > 
> > Volume Name: voicemail
> > Type: Replicate
> > Volume ID: 8f628d09-60d0-4cb8-8d1a-b7a272c42a23
> > Status: Started
> > Number of Bricks: 1 x 4 = 4
> > Transport-type: tcp
> > Bricks:
> > Brick1: gluster1a-1:/data/brick/voicemail
> > Brick2: gluster1b-1:/data/brick/voicemail
> > Brick3: gluster2a-1:/data/brick/voicemail
> > Brick4: gluster2b-1:/data/brick/voicemail
> > 
> > 
> > 
> > # gluster volume heal voicemail info
> > Brick gluster1a-1.dns99.co.uk:/data/brick/voicemail/
> > /borked/Old - Possibly undergoing heal
> > 
> > Number of entries: 1
> > 
> > Brick gluster1b-1.dns99.co.uk:/data/brick/voicemail/
> > Number of entries: 0
> > 
> > Brick gluster2a-1.dns99.co.uk:/data/brick/voicemail/
> > /borked/Old - Possibly undergoing heal
> > 
> > Number of entries: 1
> > 
> > Brick gluster2b-1.dns99.co.uk:/data/brick/voicemail/
> > /borked/Old - Possibly undergoing heal
> > 
> > Number of entries: 1
> > 
> > 
> > There's a statedump here; I've appended the server name to each file (I
> > grabbed a copy from each server, in case it helped):
> > 
> > http://gluster.dogwind.com/files/statedump.voicemail.tar.gz
> > 
> > 
> > What should I do?
> > 
> > Cheers,
> > Kingsley.
> > 
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> > 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance issue in gluster volume

[Anuradha Talur]

- Original Message -
> From: "Ramavtar" 
> To: gluster-users@gluster.org
> Sent: Friday, May 20, 2016 11:12:43 PM
> Subject: [Gluster-users] performance issue in gluster volume
> 
> Hi Ravi,
> 
> I am using gluster volume and it's having 2.7 TB data ( mp4 and jpeg
> files)  with nginx webserver
> 
> I am facing performance related issue with gluster volume please help me
Hi,

Could you elaborate what kind of performance issue you are talking about?

> please find  the gluster details :
> 
> 
> [root@webnode3 ~]# gluster --version
> glusterfs 3.7.11 built on Apr 27 2016 14:09:22
> Repository revision: git://git.gluster.com/glusterfs.git
> Copyright (c) 2006-2011 Gluster Inc. 
> GlusterFS comes with ABSOLUTELY NO WARRANTY.
> You may redistribute copies of GlusterFS under the terms of the GNU
> General Publ
> ic License.
> 
> 
> [root@webnode3 ~]# gluster volume info
> 
> Volume Name: DATA-STORE
> Type: Distributed-Replicate
> Volume ID: b64c1fea-1500-4014-b36a-0487818fa893
> Status: Started
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: webhost75:/home/DATABRINK
> Brick2: webhost90:/home/DATABRINK
> Brick3: mysqlhost1:/home/DATABRINK
> Brick4: mysqlhost2:/home/DATABRINK
> Options Reconfigured:
> performance.readdir-ahead: on
> 
> I mounted this volume on same server with fuse.
> 
> please help me.
> 
> Thanks,
> Ram
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 2 nodes volume with existing data

[Anuradha Talur]

- Original Message -
> From: "CHEVALIER Pierre" 
> To: "Anuradha Talur" 
> Cc: gluster-users@gluster.org, "Pranith Kumar Karampuri" 
> , "Ravishankar N"
> , "Atin Mukherjee" 
> Sent: Thursday, May 19, 2016 6:17:50 PM
> Subject: Re: [Gluster-users] 2 nodes volume with existing data
> 
> 
> Le 19/05/2016 12:06, Anuradha Talur a écrit :
> >
> > - Original Message -
> >> From: "Atin Mukherjee" 
> >> To: "CHEVALIER Pierre" , gluster-users@gluster.org
> >> Cc: "Pranith Kumar Karampuri" , "Ravishankar N"
> >> , "Anuradha Talur"
> >> 
> >> Sent: Tuesday, May 17, 2016 10:02:55 PM
> >> Subject: Re: [Gluster-users] 2 nodes volume with existing data
> >>
> >>
> >>
> >> On 05/17/2016 01:59 PM, CHEVALIER Pierre wrote:
> >>> Hi everyone,
> >>>
> >>> I have found a similar question in the ML but didn't find the answer I
> >>> wanted.
> >>> My situation is  the following :
> >>> We have 2 existing data storage unit with 72TB each, with 10TB left.
> >>> Currently these storage unit are synced using rsync periodically. We
> >>> want to tansform these 2 nodes into glusterfs bricks without loosing any
> >>> data or deleting a node.
> >>>
> >>> I was thinking setting up the 2 nodes with glusterfs in replica this way
> >>> :
> >>>
> >>> 1) Rsync server1/brick1 and server2/brick2 content
> >>> 2) Create the gluster replica and start :
> >>> gluster volume create volume1 replica 2 transport tcp server1:/brick1
> >>> server2:/brick1
> >>> gluster volume start volume1
> >> Gluster heavily depends on extended attributes for its business logic
> >> and hence configuring a brick with existing data means that the extended
> >> attributes would not be set for these files and hence this may not work
> >> as per the expectation.
> >>
> >> Given that you have a restriction using another brick, I'd suggest you
> >> to do the following:
> >>
> >> 1. Delete all the contents of server2/brick as this is a secondary copy
> >> 2. Create a plain distributed volume in server2 with server2/brick1
> >> 2. Start the volume
> >> 3. Mount the volume and then do copy the content from server1/brick1 to
> >> the mount point.
> >> 4. Probe server2 into the cluster
> >> 5. Perform an add-brick with server1/brick by gluster volume add-brick
> >>  replica 2  (Again make sure the content is deleted
> >> first)
> >> 6. This should trigger the self heal
> > As of now converting a plain distribute volume to distribute replicate is
> > not going to
> > automatically heal the files. (We are working on a solution for this.)
> >
> > If you really want to go ahead with this method then I'd recommend you to
> > do the following :
> >
> > 1) Stop I/O on your volume.
> > 2) Add the new brick.
> > 3) Stop and start your gluster volume
> > 4) Trigger lookup on the files from mount point by doing find . | xargs
> > stat on the mount point.
> >
> > This should trigger heals on the new brick.
> Hi,
> 
> OK this is good news,  I wanted to avoid to have no backup of the content.
> 
> So after that I mount the new volume as gluster fs and can start I/O on
> it  ?
> 
> Thanks for these informations.

Once you have completed the above mentioned steps, yes, you can mount the 
volume and start io on it.
> 
> >
> >> However since the data volume is very high, self heal can take a longer
> >> time to sync and finish.
> >>
> >> Also with a two node setup prevention of split brains is not guaranteed.
> >> An arbiter volume or a 3 way replica would be the solution to tackle
> >> this problem.
> >>
> >> Since I am not an AFR expert, I'd like to get a sign off from AFR team
> >> (In cc) before you try this out.
> >>
> >> ~Atin
> >>
> >>> Then, should I run a healing process, or stat each files in order to
> >>> self heal ?
> >>>
> >>> What happens if there is a slight difference between the 2 nodes,
> >>> mathematically if :
> >>> server1/brick1= server2/brick1+delta
> >>>
> >>> Will the delta part impact server1, or will it be lost  ?
> >>>
> >>> Maybe I worry for nothing, but as these 2 nodes host important data, I
> >>> wanted to make sure that eveything is OK.
> >>>
> >>> Thanks for your help !
> >>>
> >>> --
> >>> Pierre CHEVALIER
> >>>
> >>>
> >>>
> >>> ___
> >>> Gluster-users mailing list
> >>> Gluster-users@gluster.org
> >>> http://www.gluster.org/mailman/listinfo/gluster-users
> >>>
> --
> Pierre CHEVALIER
> 
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] heal info report a gfid

[Anuradha Talur]

- Original Message -
> From: "Jesper Led Lauridsen TS Infra server" 
> To: gluster-users@gluster.org
> Sent: Thursday, May 19, 2016 2:49:33 PM
> Subject: [Gluster-users] heal info report a gfid
> 
> Hi,
> 
> I have a replicated volume where "gluster volume heal  info" reports
> a GFID only on one of the bricks.
> 
> The GFID referees to this file, but I can't locate the file on the brick
> located on  glustertst01 or on a mounted volume
> File =
> /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/master/tasks/ad75ad79-d90f-483d-8061-0ca640ad93d8/ad75ad79-d90f-483d-8061-0ca640ad93d8.task
> 
> How do I solve this?
> 
> # gluster volume info  glu_rhevtst_dr2_data_01
> Brick5: glustoretst01.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01
> Brick6: glustoretst02.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01
> 
> # gluster volume heal glu_rhevtst_dr2_data_01 info split-brain
> Brick glustoretst01.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01
> Number of entries: 0
> Brick glustoretst02.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01
> Number of entries: 0
> 
> # gluster volume heal glu_rhevtst_dr2_data_01 info
> Brick glustertst01.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01/
> Number of entries: 0
> Brick glustertst02.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01/
> 
> Number of entries: 1
> 
Self-heal daemon (if it is in on state) will heal this file from glustertst02 
to glustertst01. I'm not sure why you are trying to locate it.

If you want to locate it, this is how you do it :
1) ls -i 
/bricks/brick1/glu_rhevtst_dr2_data_01/.glusterfs/32/5c/325ccd9f-a7f1-4ad0-bfc8-6d4b73930b9f
 on glustoretst02.net.dr.dk
2) find the inode number that you will get associated with this file on the 
same brick (you can use -inum option of find).

You should be able to locate the file. I hope this answers your question.

> # stat
> /var/run/gluster/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/master/tasks/ad75ad79-d90f-483d-8061-0ca640ad93d8/ad75ad79-d90f-483d-8061-0ca640ad93d8.task
> stat: cannot stat
> `/var/run/gluster/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/master/tasks/ad75ad79-d90f-483d-8061-0ca640ad93d8/ad75ad79-d90f-483d-8061-0ca640ad93d8.task':
> No such file or directory
> 
> glustertst02 ~]# getfattr -d -m . -e hex
> /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/master/tasks/ad75ad79-d90f-483d-8061-0ca640ad93d8/ad75ad79-d90f-483d-8061-0ca640ad93d8.task
> getfattr: Removing leading '/' from absolute path names
> # file:
> bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/master/tasks/ad75ad79-d90f-483d-8061-0ca640ad93d8/ad75ad79-d90f-483d-8061-0ca640ad93d8.task
> security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
> trusted.afr.glu_rhevtst_dr2_data_01-client-4=0x00010002
> trusted.afr.glu_rhevtst_dr2_data_01-client-5=0x
> trusted.gfid=0x325ccd9fa7f14ad0bfc86d4b73930b9f
> trusted.glusterfs.dht.linkto=0x676c755f726865767473745f6472325f646174615f30312d7265706c69636174652d3300
> trusted.glusterfs.quota.bf0a8e25-e918-4ae3-a947-7971b7b8a372.contri=0x
> 
> glustertst01 ~]# getfattr -n "trusted.gfid" -e hex
> /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/master/tasks/ad75ad79-d90f-483d-8061-0ca640ad93d8/ad75ad79-d90f-483d-8061-0ca640ad93d8.task
> getfattr:
> /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/master/tasks/ad75ad79-d90f-483d-8061-0ca640ad93d8/ad75ad79-d90f-483d-8061-0ca640ad93d8.task:
> No such file or directory
> 
> Any help appreciated
> 
> Thanks
> Jesper
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 2 nodes volume with existing data

[Anuradha Talur]

- Original Message -
> From: "Atin Mukherjee" 
> To: "CHEVALIER Pierre" , gluster-users@gluster.org
> Cc: "Pranith Kumar Karampuri" , "Ravishankar N" 
> , "Anuradha Talur"
> 
> Sent: Tuesday, May 17, 2016 10:02:55 PM
> Subject: Re: [Gluster-users] 2 nodes volume with existing data
> 
> 
> 
> On 05/17/2016 01:59 PM, CHEVALIER Pierre wrote:
> > Hi everyone,
> > 
> > I have found a similar question in the ML but didn't find the answer I
> > wanted.
> > My situation is  the following :
> > We have 2 existing data storage unit with 72TB each, with 10TB left.
> > Currently these storage unit are synced using rsync periodically. We
> > want to tansform these 2 nodes into glusterfs bricks without loosing any
> > data or deleting a node.
> > 
> > I was thinking setting up the 2 nodes with glusterfs in replica this way :
> > 
> > 1) Rsync server1/brick1 and server2/brick2 content
> > 2) Create the gluster replica and start :
> > gluster volume create volume1 replica 2 transport tcp server1:/brick1
> > server2:/brick1
> > gluster volume start volume1
> Gluster heavily depends on extended attributes for its business logic
> and hence configuring a brick with existing data means that the extended
> attributes would not be set for these files and hence this may not work
> as per the expectation.
> 
> Given that you have a restriction using another brick, I'd suggest you
> to do the following:
> 
> 1. Delete all the contents of server2/brick as this is a secondary copy
> 2. Create a plain distributed volume in server2 with server2/brick1
> 2. Start the volume
> 3. Mount the volume and then do copy the content from server1/brick1 to
> the mount point.
> 4. Probe server2 into the cluster
> 5. Perform an add-brick with server1/brick by gluster volume add-brick
>  replica 2  (Again make sure the content is deleted
> first)
> 6. This should trigger the self heal

As of now converting a plain distribute volume to distribute replicate is not 
going to
automatically heal the files. (We are working on a solution for this.)

If you really want to go ahead with this method then I'd recommend you to do 
the following :

1) Stop I/O on your volume.
2) Add the new brick.
3) Stop and start your gluster volume
4) Trigger lookup on the files from mount point by doing find . | xargs stat on 
the mount point.

This should trigger heals on the new brick.

> 
> However since the data volume is very high, self heal can take a longer
> time to sync and finish.
> 
> Also with a two node setup prevention of split brains is not guaranteed.
> An arbiter volume or a 3 way replica would be the solution to tackle
> this problem.
> 
> Since I am not an AFR expert, I'd like to get a sign off from AFR team
> (In cc) before you try this out.
> 
> ~Atin
>   
> > 
> > Then, should I run a healing process, or stat each files in order to
> > self heal ?
> > 
> > What happens if there is a slight difference between the 2 nodes,
> > mathematically if :
> > server1/brick1= server2/brick1+delta
> > 
> > Will the delta part impact server1, or will it be lost  ?
> > 
> > Maybe I worry for nothing, but as these 2 nodes host important data, I
> > wanted to make sure that eveything is OK.
> > 
> > Thanks for your help !
> > 
> > --
> > Pierre CHEVALIER
> > 
> > 
> > 
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> > 
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] replace brick with brick names the same

[Anuradha Talur]

Response inline.

- Original Message -
> From: "Atin Mukherjee" 
> To: "Lindsay Mathieson" 
> Cc: "gluster-users" , "Anuradha Talur" 
> 
> Sent: Saturday, May 14, 2016 8:35:38 AM
> Subject: Re: [Gluster-users] replace brick with brick names the same
> 
> -Atin
> Sent from one plus one
> On 14-May-2016 4:45 AM, "Lindsay Mathieson" 
> wrote:
> >
> > I'd like to replace a brick on a replica 3 setup (3.7.11 ). And for ease
> of admin and scripting reasons I'd like it to have the same location/name.
> Would the following work?
> 
> IIRC, as of now it won't work. However this is on our plate and Anuradha
> had done some work on this, not sure whether the patch is yet available or
> not for review.
> 
> @Anuradha - could you shed some light on this?

Like Atin said this will not work as of now.
A patch was up for review but new requirements were added to it. I need to 
rework on
it and put it up again.

> >
> > 1. Down vng.proxmox.softlog:/tank/vmdata/data
> >
> > 2. Erase /tank/vmdata/data
> >
> > 3. recreate /tank/vmdata/data
> >
> > 4. volume replace-brick datastore4 vng.proxmox.softlog:/tank/vmdata/data
> vng.proxmox.softlog:/tank/vmdata/data commit
> >
> >
> >
> >
> > --
> > Lindsay Mathieson
> >
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster volume heal info split brain command not showing files in split-brain

[Anuradha Talur]

- Original Message -
> From: "ABHISHEK PALIWAL" 
> To: "Anuradha Talur" 
> Cc: gluster-users@gluster.org, gluster-de...@gluster.org
> Sent: Monday, March 21, 2016 10:44:53 AM
> Subject: Re: [Gluster-users] gluster volume heal info split brain command not 
> showing files in split-brain
> 
> Hi Anuradha,
> 
> Have you got any pointer from the above scenarios.
> 
Hi Abhishek,

I went through all the logs that you have given.
There is only one brick's log in the info you provided,
and for only one day. Where is the other brick's logfile?

In the same log file I see a lot of connects and disconnects in quick 
succession.
Which could be the cause of gfid mismatch if I/O was going on during the time.
The other logs that have been provided also do not have enough information to
determine how your setup could have ended up with no pending markers.

I understand that output from heal info split-brain is more easy to get info
for files in split-brain. But without pending markers, this info cannot be 
obtained.

For second scenario, is your self-heal-daemon on?
> Regards,
> Abhishek
> 
> On Fri, Mar 18, 2016 at 11:18 AM, ABHISHEK PALIWAL 
> wrote:
> 
> >
> >
> > On Fri, Mar 18, 2016 at 1:41 AM, Anuradha Talur  wrote:
> >
> >>
> >>
> >> - Original Message -
> >> > From: "ABHISHEK PALIWAL" 
> >> > To: "Anuradha Talur" 
> >> > Cc: gluster-users@gluster.org, gluster-de...@gluster.org
> >> > Sent: Thursday, March 17, 2016 4:00:58 PM
> >> > Subject: Re: [Gluster-users] gluster volume heal info split brain
> >> command not showing files in split-brain
> >> >
> >> > Hi Anuradha,
> >> >
> >> > Please confirm me, this is bug in glusterfs or we need to do something
> >> at
> >> > our end.
> >> >
> >> > Because this problem is stopping our development.
> >> Hi Abhishek,
> >>
> >> When you say file is not getting sync, do you mean that the files are not
> >> in sync after healing or that the existing GFID mismatch that you tried to
> >> heal failed?
> >> In one of the previous mails, you said that the GFID mismatch problem is
> >> resolved, is it not so?
> >>
> >
> > As I mentioned I have two scenario:
> > 1. First scenario is where files are in split-brain but not recognized by
> > the the split-brain and heal info commands. So we are identifying those
> > file when I/O errors occurred on those files (the same method mentioned in
> > the link which you shared earlier) but this method is not reliable in our
> > case because other modules have the dependencies on this file and those
> > modules can't wait until heal in progress. In this case we required manual
> > identification of the file those are falling in I/O error which is somehow
> > not the correct way. It is better if the split-brain or heal info command
> > identify the files and based on the output we will perform the self healing
> > on those files only.
> >
> > 2. Second scenario in which we have one log file which have the fixed size
> > and wrapping of data properties and continuously written by the system even
> > when the other brick is down or rebooting. In this case, we have two brick
> > in replica mode and when one goes down and comes up but this file remains
> > out of sync. We are not getting any of the following on this file:
> > A. Not recognized by the split-brain and heal info command.
> > B. Not getting any I/O error
> > C. Do not have the GFID mismatch
> >
> > Here, are the getfattr output of this file
> >
> > Brick B which rebooted and have the file out of sync
> >
> > getfattr -d -m . -e hex
> > opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml
> >
> > # file:
> > opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml
> > trusted.afr.c_glusterfs-client-1=0x
> > trusted.afr.dirty=0x
> > trusted.bit-rot.version=0x000b56d6dd1d000ec7a9
> > trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae
> >
> >
> > Brick A where file was getting updated when Brick B was rebooting
> >
> > getfattr -d -m . -e hex
> > opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml
> > trusted.afr.c_glusterfs-client-0=0x0008
> > trusted.afr.c_glusterfs-client-2=0x0002
> > trusted.afr.c_glusterfs-client-4=0x0002
> > trusted.afr.c_glusterfs-cli

Re: [Gluster-users] gluster volume heal info split brain command not showing files in split-brain

[Anuradha Talur]

- Original Message -
> From: "Anuradha Talur" 
> To: "ABHISHEK PALIWAL" 
> Cc: gluster-users@gluster.org, gluster-de...@gluster.org
> Sent: Wednesday, March 16, 2016 5:32:26 PM
> Subject: Re: [Gluster-users] gluster volume heal info split brain command not 
> showing files in split-brain
> 
> 
> 
> - Original Message -----
> > From: "ABHISHEK PALIWAL" 
> > To: "Anuradha Talur" 
> > Cc: gluster-users@gluster.org, gluster-de...@gluster.org
> > Sent: Wednesday, March 16, 2016 4:39:26 PM
> > Subject: Re: [Gluster-users] gluster volume heal info split brain command
> > not showing files in split-brain
> > 
> > Hi Anuradha,
> > 
> > I am doing the same which is mentioned in the link you shared and It has
> > been resolved the issue.
> > 
> > But my question is if it is the split-brain scenario then why the command
> > "gluster volume heal info split-brain"
> >  not showing these files in the output even not the parent directory is
> > present in split-brain.
> > 
> > Please find the requested logs

Abhishek,

Yes, ideally it should show. I will look into it. The only reason I can think 
of, is when parent directory did not have any pending markers to indicate 
split-brain; which is why I asked getfattr output for the parent directory too. 
But if the issue is resolved, there isn't much info we can get out of it. 
Thanks for sharing the logs. Will see what could have caused this.

> Abhishek,
> 
> Yes, ideally it should show. I will look into it.
> I saw another case with this issue. The parent directory did not have any pe
> > 
> > On Wed, Mar 16, 2016 at 4:20 PM, Anuradha Talur  wrote:
> > 
> > > Hi Abhishek,
> > >
> > > The files that are reporting i/o error have gfid-mismatch. This situation
> > > is called directory or entry split-brain. You can find steps to resolve
> > > this kind of split brain here :
> > > https://gluster.readthedocs.org/en/latest/Troubleshooting/split-brain/ .
> > >
> > > Ideally, the parent directories of these files have to be listed in heal
> > > info split-brain output. Can you please get extended attributes of parent
> > > directories of the files that show i/o error (Same getfattr command that
> > > you previously used.) ?
> > >
> > > - Original Message -
> > > > From: "ABHISHEK PALIWAL" 
> > > > To: "Anuradha Talur" 
> > > > Cc: gluster-users@gluster.org, gluster-de...@gluster.org
> > > > Sent: Thursday, March 10, 2016 11:22:35 AM
> > > > Subject: Re: [Gluster-users] gluster volume heal info split brain
> > > command not showing files in split-brain
> > > >
> > > > Hi Anuradha,
> > > >
> > > > Please find the glusterfs and glusterd logs directory as an attachment.
> > > >
> > > > Regards,
> > > > Abhishek
> > > >
> > > >
> > > >
> > > > On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL <
> > > abhishpali...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Anuradha,
> > > > >
> > > > > Sorry for late reply.
> > > > >
> > > > > Please find the requested logs below:
> > > > >
> > > > > Remote: 10.32.0.48
> > > > > Local : 10.32.1.144
> > > > >
> > > > > Local:
> > > > > #gluster volume heal c_glusterfs info split-brain
> > > > > Brick 10.32.1.144:/opt/lvmdir/c2/brick
> > > > > Number of entries in split-brain: 0
> > > > >
> > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick
> > > > > Number of entries in split-brain: 0
> > > > >
> > > > > Remote:
> > > > > #gluster volume heal c_glusterfs info split-brain
> > > > > Brick 10.32.1.144:/opt/lvmdir/c2/brick
> > > > > Number of entries in split-brain: 0
> > > > >
> > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick
> > > > > Number of entries in split-brain: 0
> > > > >
> > > > > auto-sync.sh.
> > > > > Here you can see that i/o error is detected. Below is the required
> > > > > meta
> > > > > data from both the bricks.
> > > > >
> > > > > 1)
> > > > > stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior':
> > > > > 

Re: [Gluster-users] gluster volume heal info split brain command not showing files in split-brain

[Anuradha Talur]

- Original Message -
> From: "ABHISHEK PALIWAL" 
> To: "Anuradha Talur" 
> Cc: gluster-users@gluster.org, gluster-de...@gluster.org
> Sent: Thursday, March 17, 2016 4:00:58 PM
> Subject: Re: [Gluster-users] gluster volume heal info split brain command not 
> showing files in split-brain
> 
> Hi Anuradha,
> 
> Please confirm me, this is bug in glusterfs or we need to do something at
> our end.
> 
> Because this problem is stopping our development.
Hi Abhishek,

When you say file is not getting sync, do you mean that the files are not in 
sync after healing or that the existing GFID mismatch that you tried to heal 
failed?
In one of the previous mails, you said that the GFID mismatch problem is 
resolved, is it not so?

To your question about finding the files in split-brain, can you try running 
gluster volume heal  info? Heal info is also supposed to show
the files in split-brain.

If the GFID mismatch is not resolved yet, it would really help understand the 
underlying problem if you give the output of getfattr -m. -de hex 
.
> 
> Regards,
> Abhishek
> 
> On Thu, Mar 17, 2016 at 1:54 PM, ABHISHEK PALIWAL 
> wrote:
> 
> > Hi Anuradha,
> >
> > But in this case I need to do tail on each file which is time taking
> > process and other end I can't pause my module until these file is getting
> > healed.
> >
> > Any how I need the output of the split-brain to resolve this problem.
> >
> > Regards,
> > Abhishek
> >
> > On Wed, Mar 16, 2016 at 6:21 PM, ABHISHEK PALIWAL  > > wrote:
> >
> >> Hi Anuradha,
> >>
> >> The issue is resolved but we have one more issue something similar to
> >> this one in which the file is not getting sync after the steps followed,
> >> mentioned in the link which you shared in the previous mail.
> >>
> >> And problem is that why split-brain command is not showing split-brain
> >> entries.
> >>
> >> Regards,
> >> Abhishek
> >>
> >> On Wed, Mar 16, 2016 at 6:06 PM, Anuradha Talur 
> >> wrote:
> >>
> >>>
> >>>
> >>> - Original Message -
> >>> > From: "Anuradha Talur" 
> >>> > To: "ABHISHEK PALIWAL" 
> >>> > Cc: gluster-users@gluster.org, gluster-de...@gluster.org
> >>> > Sent: Wednesday, March 16, 2016 5:32:26 PM
> >>> > Subject: Re: [Gluster-users] gluster volume heal info split brain
> >>> command not showing files in split-brain
> >>> >
> >>> >
> >>> >
> >>> > - Original Message -
> >>> > > From: "ABHISHEK PALIWAL" 
> >>> > > To: "Anuradha Talur" 
> >>> > > Cc: gluster-users@gluster.org, gluster-de...@gluster.org
> >>> > > Sent: Wednesday, March 16, 2016 4:39:26 PM
> >>> > > Subject: Re: [Gluster-users] gluster volume heal info split brain
> >>> command
> >>> > > not showing files in split-brain
> >>> > >
> >>> > > Hi Anuradha,
> >>> > >
> >>> > > I am doing the same which is mentioned in the link you shared and It
> >>> has
> >>> > > been resolved the issue.
> >>> > >
> >>> > > But my question is if it is the split-brain scenario then why the
> >>> command
> >>> > > "gluster volume heal info split-brain"
> >>> > >  not showing these files in the output even not the parent directory
> >>> is
> >>> > > present in split-brain.
> >>> > >
> >>> > > Please find the requested logs
> >>>
> >>> Abhishek,
> >>>
> >>> Yes, ideally it should show. I will look into it. The only reason I can
> >>> think of, is when parent directory did not have any pending markers to
> >>> indicate split-brain; which is why I asked getfattr output for the parent
> >>> directory too. But if the issue is resolved, there isn't much info we can
> >>> get out of it. Thanks for sharing the logs. Will see what could have
> >>> caused
> >>> this.
> >>>
> >>> > Abhishek,
> >>> >
> >>> > Yes, ideally it should show. I will look into it.
> >>> > I saw another case with this issue. The parent directory did not have
> >>> any pe
> >>> > >
> >>> > > On Wed, Mar 16, 2016 at

Re: [Gluster-users] gluster volume heal info split brain command not showing files in split-brain

[Anuradha Talur]

Hi Abhishek,

The files that are reporting i/o error have gfid-mismatch. This situation is 
called directory or entry split-brain. You can find steps to resolve this kind 
of split brain here : 
https://gluster.readthedocs.org/en/latest/Troubleshooting/split-brain/ .

Ideally, the parent directories of these files have to be listed in heal info 
split-brain output. Can you please get extended attributes of parent 
directories of the files that show i/o error (Same getfattr command that you 
previously used.) ?

- Original Message -
> From: "ABHISHEK PALIWAL" 
> To: "Anuradha Talur" 
> Cc: gluster-users@gluster.org, gluster-de...@gluster.org
> Sent: Thursday, March 10, 2016 11:22:35 AM
> Subject: Re: [Gluster-users] gluster volume heal info split brain command not 
> showing files in split-brain
> 
> Hi Anuradha,
> 
> Please find the glusterfs and glusterd logs directory as an attachment.
> 
> Regards,
> Abhishek
> 
> 
> 
> On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL 
> wrote:
> 
> > Hi Anuradha,
> >
> > Sorry for late reply.
> >
> > Please find the requested logs below:
> >
> > Remote: 10.32.0.48
> > Local : 10.32.1.144
> >
> > Local:
> > #gluster volume heal c_glusterfs info split-brain
> > Brick 10.32.1.144:/opt/lvmdir/c2/brick
> > Number of entries in split-brain: 0
> >
> > Brick 10.32.0.48:/opt/lvmdir/c2/brick
> > Number of entries in split-brain: 0
> >
> > Remote:
> > #gluster volume heal c_glusterfs info split-brain
> > Brick 10.32.1.144:/opt/lvmdir/c2/brick
> > Number of entries in split-brain: 0
> >
> > Brick 10.32.0.48:/opt/lvmdir/c2/brick
> > Number of entries in split-brain: 0
> >
> > auto-sync.sh.
> > Here you can see that i/o error is detected. Below is the required meta
> > data from both the bricks.
> >
> > 1)
> > stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior':
> > Input/output error
> > Remote:
> >
> > getfattr -d -m . -e hex
> > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > trusted.afr.dirty=0x
> > trusted.bit-rot.version=0x000256ded2f6000ad80f
> > trusted.gfid=0x771221a7bb3c4f1aade40ce9e38a95ee
> >
> > Local:
> >
> > getfattr -d -m . -e hex
> > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > trusted.bit-rot.version=0x000256ded38f000e3a51
> > trusted.gfid=0x8ea33f46703c4e2d95c09153c1b858fd
> >
> >
> > 2)
> > stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error
> > Remote:
> >
> > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity
> > # file: opt/lvmdir/c2/brick/security/corbasecurity
> > trusted.afr.dirty=0x
> > trusted.bit-rot.version=0x000256ded2f6000ad80f
> > trusted.gfid=0xd298b7a0c8834f3e99abb39741363013
> >
> > Local:
> >
> > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity
> > # file: opt/lvmdir/c2/brick/security/corbasecurity
> > trusted.bit-rot.version=0x000256ded38f000e3a51
> > trusted.gfid=0x890df0f706184b52803fac3242a2f15b
> >
> > I observed that getfattr command output doesn't show all the fields all
> > the times.
> >
> > Here you can check that gluster split-brain command hasn't reported any
> > split-brains but resulted in IO errors when accessed few files.
> >
> > Could you please tell me if "split-brain" command doesn't reported any
> > entry as output, then is there any way through which we can find out that
> > the files are in split-brain if we are getting the IO error on those file.
> >
> >
> > Regards,
> > Abhishek
> >
> >
> >
> > On Thu, Mar 3, 2016 at 5:32 PM, Anuradha Talur  wrote:
> >
> >>
> >>
> >> - Original Message -
> >> > From: "ABHISHEK PALIWAL" 
> >> > To: gluster-users@gluster.org, gluster-de...@gluster.org
> >> > Sent: Thursday, March 3, 2016 12:10:42 PM
> >> > Subject: [Gluster-users] gluster volume heal info split brain command
> >> not showing files in split-brain
> >> >
> >> >
> >> > Hello,
> >> >
> >> > In gluster, we use the command "gluster volume heal c_glusterfs info
> >> > split-brain&

Re: [Gluster-users] gluster volume heal info split brain command not showing files in split-brain

[Anuradha Talur]

Hi Abhishek,

Apologies for the delay. I was not in office from March 10th till 14th, still 
catching up with the mails.
I'm going through the logs. Will respond to this once done.

- Original Message -
> From: "ABHISHEK PALIWAL" 
> To: "Anuradha Talur" 
> Cc: gluster-users@gluster.org, gluster-de...@gluster.org
> Sent: Monday, March 14, 2016 10:31:02 AM
> Subject: Re: [Gluster-users] gluster volume heal info split brain command not 
> showing files in split-brain
> 
> Hi Anuradha,
> 
> Please reply on this issue.
> 
> Regards,
> Abhishek
> 
> On Thu, Mar 10, 2016 at 11:22 AM, ABHISHEK PALIWAL 
> wrote:
> 
> > Hi Anuradha,
> >
> > Please find the glusterfs and glusterd logs directory as an attachment.
> >
> > Regards,
> > Abhishek
> >
> >
> >
> >
> > On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL 
> > wrote:
> >
> >> Hi Anuradha,
> >>
> >> Sorry for late reply.
> >>
> >> Please find the requested logs below:
> >>
> >> Remote: 10.32.0.48
> >> Local : 10.32.1.144
> >>
> >> Local:
> >> #gluster volume heal c_glusterfs info split-brain
> >> Brick 10.32.1.144:/opt/lvmdir/c2/brick
> >> Number of entries in split-brain: 0
> >>
> >> Brick 10.32.0.48:/opt/lvmdir/c2/brick
> >> Number of entries in split-brain: 0
> >>
> >> Remote:
> >> #gluster volume heal c_glusterfs info split-brain
> >> Brick 10.32.1.144:/opt/lvmdir/c2/brick
> >> Number of entries in split-brain: 0
> >>
> >> Brick 10.32.0.48:/opt/lvmdir/c2/brick
> >> Number of entries in split-brain: 0
> >>
> >> auto-sync.sh.
> >> Here you can see that i/o error is detected. Below is the required meta
> >> data from both the bricks.
> >>
> >> 1)
> >> stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior':
> >> Input/output error
> >> Remote:
> >>
> >> getfattr -d -m . -e hex
> >> opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> >> # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> >> trusted.afr.dirty=0x
> >> trusted.bit-rot.version=0x000256ded2f6000ad80f
> >> trusted.gfid=0x771221a7bb3c4f1aade40ce9e38a95ee
> >>
> >> Local:
> >>
> >> getfattr -d -m . -e hex
> >> opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> >> # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> >> trusted.bit-rot.version=0x000256ded38f000e3a51
> >> trusted.gfid=0x8ea33f46703c4e2d95c09153c1b858fd
> >>
> >>
> >> 2)
> >> stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error
> >> Remote:
> >>
> >> getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity
> >> # file: opt/lvmdir/c2/brick/security/corbasecurity
> >> trusted.afr.dirty=0x
> >> trusted.bit-rot.version=0x000256ded2f6000ad80f
> >> trusted.gfid=0xd298b7a0c8834f3e99abb39741363013
> >>
> >> Local:
> >>
> >> getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity
> >> # file: opt/lvmdir/c2/brick/security/corbasecurity
> >> trusted.bit-rot.version=0x000256ded38f000e3a51
> >> trusted.gfid=0x890df0f706184b52803fac3242a2f15b
> >>
> >> I observed that getfattr command output doesn't show all the fields all
> >> the times.
> >>
> >> Here you can check that gluster split-brain command hasn't reported any
> >> split-brains but resulted in IO errors when accessed few files.
> >>
> >> Could you please tell me if "split-brain" command doesn't reported any
> >> entry as output, then is there any way through which we can find out that
> >> the files are in split-brain if we are getting the IO error on those file.
> >>
> >>
> >> Regards,
> >> Abhishek
> >>
> >>
> >>
> >> On Thu, Mar 3, 2016 at 5:32 PM, Anuradha Talur  wrote:
> >>
> >>>
> >>>
> >>> - Original Message -
> >>> > From: "ABHISHEK PALIWAL" 
> >>> > To: gluster-users@gluster.org, gluster-de...@gluster.org
> >>> > Sent: Thursday, March 3, 2016 12:10:42 PM
> >>> > Subject: [Gluster-users] glust

Re: [Gluster-users] gluster volume heal info split brain command not showing files in split-brain

[Anuradha Talur]

- Original Message -
> From: "ABHISHEK PALIWAL" 
> To: gluster-users@gluster.org, gluster-de...@gluster.org
> Sent: Thursday, March 3, 2016 12:10:42 PM
> Subject: [Gluster-users] gluster volume heal info split brain command not 
> showing files in split-brain
> 
> 
> Hello,
> 
> In gluster, we use the command "gluster volume heal c_glusterfs info
> split-brain" to find the files that are in split-brain scenario.
> We run heal script (developed by Windriver prime team) on the files reported
> by above command to resolve split-brain issue.
> 
> But we observed that the above command is not showing all files that are in
> split-brain,
> even though split brain scenario actually exists on the node.
> 
> Now a days this issue is seen more often and IO errors are reported when
> tried to access these files under split-brain.
> 
> Can you please check why this gluster command is not showing files under
> split-brain?
> We can provide you required logs and support to resolve this issue.
Hi,

Could you paste the output of getfattr -m. -de hex 
 from all the bricks that the files lie in?

> 
> Please reply on this because I am not getting any reply from the community.
> 
> --
> 
> Regards
> Abhishek Paliwal
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] about tail command

[Anuradha Talur]

- Original Message -
> From: "Anuradha Talur" 
> To: "songxin" 
> Cc: "gluster-user" 
> Sent: Thursday, March 3, 2016 12:31:41 PM
> Subject: Re: [Gluster-users] about tail command
> 
> 
> 
> ----- Original Message -
> > From: "songxin" 
> > To: "Anuradha Talur" 
> > Cc: "gluster-user" 
> > Sent: Wednesday, March 2, 2016 4:09:01 PM
> > Subject: Re:Re: [Gluster-users] about tail command
> > 
> > 
> > 
> > Thank you for your reply.I have two more questions as below
> > 
> > 
> > 1. the command "gluster v replace-brick " is async or sync?  The replace is
> > complete when the command quit ?
> It is a sync command, replacing the brick finishes as the command returns.
> 
> In one of the earlier mails I gave incomplete command for replace brick,
> sorry about that.
> The only replace-brick operation allowed from glusterfs 3.7.9 onwards is
> 'gluster v replace-brick   
> commit force'.
Sorry for spamming, but there is a typo here, I meant glusterfs 3.7.0 onwards,
not 3.7.9.
> > 2.I run "tail -n 0" on mount point.Does it trigger the heal?
> > 
> > 
> > Thanks,
> > Xin
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > At 2016-03-02 15:22:35, "Anuradha Talur"  wrote:
> > >
> > >
> > >- Original Message -
> > >> From: "songxin" 
> > >> To: "gluster-user" 
> > >> Sent: Tuesday, March 1, 2016 7:19:23 PM
> > >> Subject: [Gluster-users] about tail command
> > >> 
> > >> Hi,
> > >> 
> > >> recondition:
> > >> A node:128.224.95.140
> > >> B node:128.224.162.255
> > >> 
> > >> brick on A node:/data/brick/gv0
> > >> brick on B node:/data/brick/gv0
> > >> 
> > >> 
> > >> reproduce steps:
> > >> 1.gluster peer probe 128.224.162.255 (on A node)
> > >> 2. gluster volume create gv0 replica 2 128.224.95.140:/data/brick/gv0
> > >> 128.224.162.255:/data/brick/gv0 force (on A node)
> > >> 3.gluster volume start gv0 (on A node)
> > >> 4. mount -t glusterfs 128.224.95.140:/gv0 gluster (on A node)
> > >> 5.create some files (a,b,c) in dir gluster (on A node)
> > >> 6.shutdown the B node
> > >> 7.change the files (a,b,c) in dir gluster (on A node)
> > >> 8.reboot B node
> > >> 9.start glusterd on B node but glusterfsd is offline (on B node)
> > >> 10. gluster volume remove-brick gv0 replica 1
> > >> 128.224.162.255:/data/brick/gv0
> > >> force (on A node)
> > >> 11. gluster volume add-brick gv0 replica 2
> > >> 128.224.162.255:/data/brick/gv0
> > >> force (on A node)
> > >> 
> > >> Now the files are not same between two brick
> > >> 
> > >> 12." gluster volume heal gv0 info " show entry num is 0 (on A node)
> > >> 
> > >> Now What I should do if I want to sync file(a,b,c) on two brick?
> > >> 
> > >Currently, once you add a brick to a cluster, files won't sync
> > >automatically.
> > >Patch has been sent to handle this requirement. Auto-heal will be
> > >available
> > >soon.
> > >
> > >You could kill the newly added brick and perform the following operations
> > >from mount
> > >for the sync to start :
> > >1) create a directory 
> > >2) setfattr -n "user.dirname" -v "value" 
> > >3) delete the directory 
> > >
> > >Once these steps are done, start the killed brick. self-heal-daemon will
> > >heal the files.
> > >
> > >But, for the case you have mentioned, why are you removing brick and using
> > >add-brick again?
> > >Is it because you don't want to change the brick-path?
> > >
> > >You could use "replace-brick" command.
> > >gluster v replace-brick  
> > >
> > >Note that source and destination should be different for this command to
> > >work.
> > >
> > >> I know the "heal full" can work , but I think the command take too long
> > >> time.
> > >> 
> > >> So I run "tail -n 1 file" to all file on A node, but some files are sync
> > >> but
> > >> some files are not.
> > >> 
> > >> My question is below:
> > >> 1.Why the tail can't sync all files?
> > >Did you run the tail command on mount point or from the backend (bricks)?
> > >If you run from bricks, sync won't happen. Was client-side healing on?
> > >To check if they were on or off, run `gluster v get volname all | grep
> > >self-heal`, cluster.metadata-self-heal, cluster.data-self-heal,
> > >cluster.entry-self-heal should be on.
> > >
> > >> 2.Can the command "tail -n 1 filename" trigger selfheal, just like "ls
> > >> -l
> > >> filename"?
> > >> 
> > >> Thanks,
> > >> Xin
> > >> 
> > >> 
> > >> 
> > >> 
> > >> 
> > >> 
> > >> 
> > >> 
> > >> 
> > >> 
> > >> 
> > >> ___
> > >> Gluster-users mailing list
> > >> Gluster-users@gluster.org
> > >> http://www.gluster.org/mailman/listinfo/gluster-users
> > >
> > >--
> > >Thanks,
> > >Anuradha.
> > 
> 
> --
> Thanks,
> Anuradha.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] about tail command

[Anuradha Talur]

- Original Message -
> From: "songxin" 
> To: "Anuradha Talur" 
> Cc: "gluster-user" 
> Sent: Wednesday, March 2, 2016 4:09:01 PM
> Subject: Re:Re: [Gluster-users] about tail command
> 
> 
> 
> Thank you for your reply.I have two more questions as below
> 
> 
> 1. the command "gluster v replace-brick " is async or sync?  The replace is
> complete when the command quit ?
It is a sync command, replacing the brick finishes as the command returns.

In one of the earlier mails I gave incomplete command for replace brick, sorry 
about that.
The only replace-brick operation allowed from glusterfs 3.7.9 onwards is
'gluster v replace-brick
commit force'.
> 2.I run "tail -n 0" on mount point.Does it trigger the heal?
> 
> 
> Thanks,
> Xin
> 
> 
> 
> 
> 
> 
> 
> At 2016-03-02 15:22:35, "Anuradha Talur"  wrote:
> >
> >
> >- Original Message -
> >> From: "songxin" 
> >> To: "gluster-user" 
> >> Sent: Tuesday, March 1, 2016 7:19:23 PM
> >> Subject: [Gluster-users] about tail command
> >> 
> >> Hi,
> >> 
> >> recondition:
> >> A node:128.224.95.140
> >> B node:128.224.162.255
> >> 
> >> brick on A node:/data/brick/gv0
> >> brick on B node:/data/brick/gv0
> >> 
> >> 
> >> reproduce steps:
> >> 1.gluster peer probe 128.224.162.255 (on A node)
> >> 2. gluster volume create gv0 replica 2 128.224.95.140:/data/brick/gv0
> >> 128.224.162.255:/data/brick/gv0 force (on A node)
> >> 3.gluster volume start gv0 (on A node)
> >> 4. mount -t glusterfs 128.224.95.140:/gv0 gluster (on A node)
> >> 5.create some files (a,b,c) in dir gluster (on A node)
> >> 6.shutdown the B node
> >> 7.change the files (a,b,c) in dir gluster (on A node)
> >> 8.reboot B node
> >> 9.start glusterd on B node but glusterfsd is offline (on B node)
> >> 10. gluster volume remove-brick gv0 replica 1
> >> 128.224.162.255:/data/brick/gv0
> >> force (on A node)
> >> 11. gluster volume add-brick gv0 replica 2 128.224.162.255:/data/brick/gv0
> >> force (on A node)
> >> 
> >> Now the files are not same between two brick
> >> 
> >> 12." gluster volume heal gv0 info " show entry num is 0 (on A node)
> >> 
> >> Now What I should do if I want to sync file(a,b,c) on two brick?
> >> 
> >Currently, once you add a brick to a cluster, files won't sync
> >automatically.
> >Patch has been sent to handle this requirement. Auto-heal will be available
> >soon.
> >
> >You could kill the newly added brick and perform the following operations
> >from mount
> >for the sync to start :
> >1) create a directory 
> >2) setfattr -n "user.dirname" -v "value" 
> >3) delete the directory 
> >
> >Once these steps are done, start the killed brick. self-heal-daemon will
> >heal the files.
> >
> >But, for the case you have mentioned, why are you removing brick and using
> >add-brick again?
> >Is it because you don't want to change the brick-path?
> >
> >You could use "replace-brick" command.
> >gluster v replace-brick  
> >
> >Note that source and destination should be different for this command to
> >work.
> >
> >> I know the "heal full" can work , but I think the command take too long
> >> time.
> >> 
> >> So I run "tail -n 1 file" to all file on A node, but some files are sync
> >> but
> >> some files are not.
> >> 
> >> My question is below:
> >> 1.Why the tail can't sync all files?
> >Did you run the tail command on mount point or from the backend (bricks)?
> >If you run from bricks, sync won't happen. Was client-side healing on?
> >To check if they were on or off, run `gluster v get volname all | grep
> >self-heal`, cluster.metadata-self-heal, cluster.data-self-heal,
> >cluster.entry-self-heal should be on.
> >
> >> 2.Can the command "tail -n 1 filename" trigger selfheal, just like "ls -l
> >> filename"?
> >> 
> >> Thanks,
> >> Xin
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> ___
> >> Gluster-users mailing list
> >> Gluster-users@gluster.org
> >> http://www.gluster.org/mailman/listinfo/gluster-users
> >
> >--
> >Thanks,
> >Anuradha.
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Broken after 3.7.8 upgrade from 3.7.6

[Anuradha Talur]

- Original Message -
> From: "Alan Millar" 
> To: "Anuradha Talur" 
> Cc: gluster-users@gluster.org
> Sent: Wednesday, March 2, 2016 2:00:49 AM
> Subject: Re: [Gluster-users] Broken after 3.7.8 upgrade from 3.7.6
> 
> >> I’m still trying to figure out why the self-heal-daemon doesn’t seem to be
> >> working, and what “unable to get index-dir” means. Any advice on what to
> >> look at would be appreciated. Thanks!
> 
> 
> >At any point did you have one node with 3.7.6 and another in 3.7.8 version?
> 
> Yes.  I upgraded each server in turn by stopping all gluster server and
> client processes on a machine, installing the update, and restarting
> everything on that machine.  Then I waited for "heal info" on all volumes to
> say it had nothing to work on before proceeding to the next server.
> 
Thanks for the information.
The "unable to get index-dir on .." messages you saw in log are not harmful in 
this scenario.
A simple explanation : when you have 1 new node and 2 old nodes, the 
self-heal-deamon and
heal commands run on the new node are expecting that the index-dir 
"/.glusterfs/xattrop/dirty" exists, to process entries from it.
But the mentioned directory doesn't exist on the old bricks. (it was introduced 
as part of
3.7.7). As a result you see these logs.

But this doesn't explain why heal didn't happen on replacing the brick. :-/
After replacing, did you check volume heal info to know the status of heal?
> 
> >I couldn't find information about the setup (number of nodes, vol info etc).
> 
> 
> I wasn't sure how much to spam the list :-)
> 
> There are 3 servers in the gluster pool.
> 
> 
> amillar@chunxy:~$ sudo gluster pool list
> UUIDHostnameState
> c1531143-229e-44a8-9fbb-089769ec999dpve4Connected
> e82b0dc0-a490-47e4-bd11-6dcfcd36fd62pve3Connected
> 0422e779-7219-4392-ad8d-6263af4372falocalhost   Connected
> 
> 
> 
> Example volume:
> 
> 
> 
> amillar@chunxy:~$ sudo gluster vol info public
> 
> Volume Name: public
> Type: Replicate
> Volume ID: f75bbc7b-5db7-4497-b14c-c0433a84bcd9
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: chunxy:/data/glfs23/public
> Brick2: pve4:/data/glfs240/public
> Brick3: pve3:/data/glfs830/public
> Options Reconfigured:
> cluster.metadata-self-heal: on
> cluster.data-self-heal: on
> cluster.entry-self-heal: on
> cluster.self-heal-daemon: enable
> performance.readdir-ahead: on
> 
> 
> 
> amillar@chunxy:~$ sudo gluster vol status public
> Status of volume: public
> Gluster process TCP Port  RDMA Port  Online  Pid
> --
> Brick chunxy:/data/glfs23/public49184 0  Y
> 13119
> Brick pve4:/data/glfs240/public 49168 0  Y   837
> Brick pve3:/data/glfs830/public 49167 0  Y
> 21074
> NFS Server on localhost 2049  0  Y
> 10237
> Self-heal Daemon on localhost   N/A   N/AY
> 10243
> NFS Server on pve3  2049  0  Y
> 30177
> Self-heal Daemon on pve3N/A   N/AY
> 30183
> NFS Server on pve4  2049  0  Y
> 13070
> Self-heal Daemon on pve4N/A   N/AY
> 13069
> 
> Task Status of Volume public
> --
> There are no active volume tasks
> 
> 
> 
> amillar@chunxy:~$ sudo gluster vol heal public info
> Brick chunxy:/data/glfs23/public
> Number of entries: 0
> 
> Brick pve4:/data/glfs240/public
> Number of entries: 0
> 
> Brick pve3:/data/glfs830/public
> Number of entries: 0
> 
> 
> 
> amillar@chunxy:~$ sudo grep public /var/log/glusterfs/glustershd.log |tail
> -10
> [2016-03-01 18:38:41.006543] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
> [2016-03-01 18:48:42.001580] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
> [2016-03-01 18:58:43.001631] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
> [2016-03-01 19:08:43.002149] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-clie

Re: [Gluster-users] about tail command

[Anuradha Talur]

- Original Message -
> From: "songxin" 
> To: "gluster-user" 
> Sent: Tuesday, March 1, 2016 7:19:23 PM
> Subject: [Gluster-users] about tail command
> 
> Hi,
> 
> recondition:
> A node:128.224.95.140
> B node:128.224.162.255
> 
> brick on A node:/data/brick/gv0
> brick on B node:/data/brick/gv0
> 
> 
> reproduce steps:
> 1.gluster peer probe 128.224.162.255 (on A node)
> 2. gluster volume create gv0 replica 2 128.224.95.140:/data/brick/gv0
> 128.224.162.255:/data/brick/gv0 force (on A node)
> 3.gluster volume start gv0 (on A node)
> 4. mount -t glusterfs 128.224.95.140:/gv0 gluster (on A node)
> 5.create some files (a,b,c) in dir gluster (on A node)
> 6.shutdown the B node
> 7.change the files (a,b,c) in dir gluster (on A node)
> 8.reboot B node
> 9.start glusterd on B node but glusterfsd is offline (on B node)
> 10. gluster volume remove-brick gv0 replica 1 128.224.162.255:/data/brick/gv0
> force (on A node)
> 11. gluster volume add-brick gv0 replica 2 128.224.162.255:/data/brick/gv0
> force (on A node)
> 
> Now the files are not same between two brick
> 
> 12." gluster volume heal gv0 info " show entry num is 0 (on A node)
> 
> Now What I should do if I want to sync file(a,b,c) on two brick?
> 
Currently, once you add a brick to a cluster, files won't sync automatically.
Patch has been sent to handle this requirement. Auto-heal will be available 
soon.

You could kill the newly added brick and perform the following operations from 
mount
for the sync to start :
1) create a directory 
2) setfattr -n "user.dirname" -v "value" 
3) delete the directory 

Once these steps are done, start the killed brick. self-heal-daemon will heal 
the files.

But, for the case you have mentioned, why are you removing brick and using 
add-brick again?
Is it because you don't want to change the brick-path?

You could use "replace-brick" command.
gluster v replace-brick   

Note that source and destination should be different for this command to work.

> I know the "heal full" can work , but I think the command take too long time.
> 
> So I run "tail -n 1 file" to all file on A node, but some files are sync but
> some files are not.
> 
> My question is below:
> 1.Why the tail can't sync all files?
Did you run the tail command on mount point or from the backend (bricks)?
If you run from bricks, sync won't happen. Was client-side healing on?
To check if they were on or off, run `gluster v get volname all | grep 
self-heal`, cluster.metadata-self-heal, cluster.data-self-heal, 
cluster.entry-self-heal should be on.

> 2.Can the command "tail -n 1 filename" trigger selfheal, just like "ls -l
> filename"?
> 
> Thanks,
> Xin
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Broken after 3.7.8 upgrade from 3.7.6

[Anuradha Talur]

- Original Message -
> From: "Alan Millar" 
> To: "Alan Millar" 
> Cc: gluster-users@gluster.org
> Sent: Tuesday, March 1, 2016 12:46:56 PM
> Subject: Re: [Gluster-users] Broken after 3.7.8 upgrade from 3.7.6
> 
> 
> On Feb 29, 2016, at 5:43 PM, Alan Millar < grunthos...@yahoo.com > wrote:
> 
> 
> 
> 
> > I have tried with entry-self-heal/metdata-self-heal/data-self-heal set both
> > on and off; neither seems to make a difference.
> 
> Correction: setting these to ON does fix the actual replicated data. I
> checked with md5sum on various files on both bricks, and it matches.
> 
> But it does not fix the duplicated directory listing entries.
> 
> 
> I found that the duplicated directory entries are isolated to a few
> directories; it is not everywhere. Now I’m thinking it was a problem that
> was there before I upgraded, and I only just now noticed it. Perhaps left
> over from a glitch during a rebalance. I deleted the zero-length files from
> the bricks, and their .glusterfs GFID hard-links. Now the directory listing
> and file access through the Fuse mount are correct.
> 
> I’m still trying to figure out why the self-heal-daemon doesn’t seem to be
> working, and what “unable to get index-dir” means. Any advice on what to
> look at would be appreciated. Thanks!
> 
Hi,

I couldn't find information about the setup (number of nodes, vol info etc).
At any point did you have one node with 3.7.6 and another in 3.7.8 version?
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume heal won't complete while I/O is running on v3.7.8

[Anuradha Talur]

- Original Message -
> From: "彭繼霆" 
> To: gluster-users@gluster.org
> Sent: Wednesday, February 24, 2016 2:20:16 PM
> Subject: [Gluster-users] Volume heal won't complete while I/O is running on   
> v3.7.8
> 
> 
> 
> Hi all,
> 
> 
> 
> After replacing a brick in a disperse volume(2+1) with I/O running, I/O works
> well but healing won't complete.
> 
> 
> 
> 
> 'gluster volume heal info' always shows that file with I/O running is still
> healing.
> 
Currently in EC there isn't a mechanism to detect the difference whether
files truly need heal or they *appear* to need heal due to the on-going I/O.
This is the reason why a file with I/O going on keeps showing up on heal info.

EC will provide a way to distinguish between these two scenario in future.
> 
> 
> 
> 
> *Example:
> 
> 
> ***
> 
> Brick N9:/export/IFT_lvol_VOIVAqzlWd/fs
> 
> /_R@W_000
> 
> Number of entries: 1
> 
> ***.
> 
> 
> 
> 
> When I/O is stopped on the file after hours, healing completes after a few
> seconds.
> 
> Is the 'file changing' cause the healing endless?
> 
> 
> 
> 
> thanks
> 
> 
> 
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] question about sync replicate volume after rebooting one node

[Anuradha Talur]

- Original Message -
> From: "songxin" 
> To: "Atin Mukherjee" 
> Cc: "Anuradha Talur" , gluster-users@gluster.org
> Sent: Wednesday, February 17, 2016 11:44:14 AM
> Subject: Re:Re: [Gluster-users] question about sync replicate volume after 
> rebooting one node
> 
> Hi，
> The version of glusterfs on  A node and B node are both 3.7.6.
> The time on B node is same after rebooting because B node hasn't RTC. Does it
> cause the problem?
> 
> 
> If I run " gluster volume start gv0 force " the glusterfsd can be started but
> "gluster volume start gv0" don't work.
> 
Yes, there is a difference between volume start and volume start force.
When a volume is in "Started" state already, gluster volume start gv0 won't do
anything (meaning it doesn't bring up the dead bricks). When you say start 
force,
status of glusterfsd's is checked and the glusterfsd's not running are spawned.
Which is the case here in the setup you have.
> 
> The file  /var/lib/glusterd/vols/gv0/info on B node as below.
> ...
> type=2
> count=2
> status=1
> sub_count=2
> stripe_count=1
> replica_count=2
> disperse_count=0
> redundancy_count=0
> version=2
> transport-type=0
> volume-id=c4197371-6d01-4477-8cb2-384cda569c27
> username=62e009ea-47c4-46b4-8e74-47cd9c199d94
> password=ef600dcd-42c5-48fc-8004-d13a3102616b
> op-version=3
> client-op-version=3
> quota-version=0
> parent_volname=N/A
> restored_from_snap=----
> snap-max-hard-limit=256
> performance.readdir-ahead=on
> brick-0=128.224.162.255:-data-brick-gv0
> brick-1=128.224.162.163:-home-wrsadmin-work-tmp-data-brick-gv0
> 
> 
> The file  /var/lib/glusterd/vols/gv0/info on A node as below.
> 
> 
> wrsadmin@pek-song1-d1:~/work/tmp$ sudo cat /var/lib/glusterd/vols/gv0/info
> type=2
> count=2
> status=1
> sub_count=2
> stripe_count=1
> replica_count=2
> disperse_count=0
> redundancy_count=0
> version=2
> transport-type=0
> volume-id=c4197371-6d01-4477-8cb2-384cda569c27
> username=62e009ea-47c4-46b4-8e74-47cd9c199d94
> password=ef600dcd-42c5-48fc-8004-d13a3102616b
> op-version=3
> client-op-version=3
> quota-version=0
> parent_volname=N/A
> restored_from_snap=----
> snap-max-hard-limit=256
> performance.readdir-ahead=on
> brick-0=128.224.162.255:-data-brick-gv0
> brick-1=128.224.162.163:-home-wrsadmin-work-tmp-data-brick-gv0
> 
> 
> Thanks,
> Xin
> 
> At 2016-02-17 12:01:37, "Atin Mukherjee"  wrote:
> >
> >
> >On 02/17/2016 08:23 AM, songxin wrote:
> >> Hi,
> >> Thank you for your immediate and detailed reply.And I have a few more
> >> question about glusterfs.
> >> A node IP is 128.224.162.163.
> >> B node IP is 128.224.162.250.
> >> 1.After reboot B node and start the glusterd service the glusterd log is
> >> as blow.
> >> ...
> >> [2015-12-07 07:54:55.743966] I [MSGID: 101190]
> >> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread
> >> with index 2
> >> [2015-12-07 07:54:55.744026] I [MSGID: 101190]
> >> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread
> >> with index 1
> >> [2015-12-07 07:54:55.744280] I [MSGID: 106163]
> >> [glusterd-handshake.c:1193:__glusterd_mgmt_hndsk_versions_ack]
> >> 0-management: using the op-version 30706
> >> [2015-12-07 07:54:55.773606] I [MSGID: 106490]
> >> [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req]
> >> 0-glusterd: Received probe from uuid: b6efd8fc-5eab-49d4-a537-2750de644a44
> >> [2015-12-07 07:54:55.777994] E [MSGID: 101076]
> >> [common-utils.c:2954:gf_get_hostname_from_ip] 0-common-utils: Could not
> >> lookup hostname of 128.224.162.163 : Temporary failure in name resolution
> >> [2015-12-07 07:54:55.778290] E [MSGID: 106010]
> >> [glusterd-utils.c:2717:glusterd_compare_friend_volume] 0-management:
> >> Version of Cksums gv0 differ. local cksum = 2492237955, remote cksum =
> >> 4087388312 on peer 128.224.162.163
> >The above log entry is the reason of the rejection of the peer, most
> >probably its due to the compatibility issue. I believe the gluster
> >versions are different (share gluster versions from both the nodes) in
> >two nodes and you might have hit a bug.
> >
> >Can you share the delta of /var/lib/glusterd/vols/gv0/info file from
> >both the nodes?
> >
> >
> >~Atin
> >> [2015-12-07 07:54:55.778384] I [MSGID: 106493]
> >> [glusterd-handler.c:37

Re: [Gluster-users] question about sync replicate volume after rebooting one node

[Anuradha Talur]

- Original Message -
> From: "songxin" 
> To: gluster-users@gluster.org
> Sent: Tuesday, February 16, 2016 3:59:50 PM
> Subject: [Gluster-users] question about sync replicate volume after   
> rebooting one node
> 
> Hi,
> I have a question about how to sync volume between two bricks after one node
> is reboot.
> 
> There are two node, A node and B node.A node ip is 128.124.10.1 and B node ip
> is 128.124.10.2.
> 
> operation steps on A node as below
> 1. gluster peer probe 128.124.10.2
> 2. mkdir -p /data/brick/gv0
> 3.gluster volume create gv0 replica 2 128.124.10.1 :/data/brick/gv0
> 128.124.10.2 :/data/brick/gv1 force
> 4. gluster volume start gv0
> 5.mount -t glusterfs 128.124.10.1 :/gv0 gluster
> 
> operation steps on B node as below
> 1 . mkdir -p /data/brick/gv0
> 2.mount -t glusterfs 128.124.10.1 :/gv0 gluster
> 
> After all steps above , there a some gluster service process, including
> glusterd, glusterfs and glusterfsd, running on both A and B node.
> I can see these servic by command ps aux | grep gluster and command gluster
> volume status.
> 
> Now reboot the B node.After B reboot , there are no gluster service running
> on B node.
> After I systemctl start glusterd , there is just glusterd service but not
> glusterfs and glusterfsd on B node.
> Because glusterfs and glusterfsd are not running so I can't gluster volume
> heal gv0 full.
> 
> I want to know why glusterd don't start glusterfs and glusterfsd.

On starting glusterd, glusterfsd should have started by itself.
Could you share glusterd and brick log (on node B) so that we know why 
glusterfsd
didn't start?

Do you still see glusterfsd service running on node A? You can try running 
"gluster v start  force"
on one of the nodes and check if all the brick processes started.

gluster volume status  should be able to provide you with gluster 
process status.

On restarting the node, glusterfs process for mount won't start by itself. You 
will have to run
step 2 on node B again for it.

> How do I restart these services on B node?
> How do I sync the replicate volume after one node reboot?

Once the glusterfsd process starts on node B too, glustershd -- 
self-heal-daemon -- for replicate volume
should start healing/syncing files that need to be synced. This deamon does 
periodic syncing of files.

If you want to trigger a heal explicitly, you can run gluster volume heal 
 on one of the servers.
> 
> Thanks,
> Xin
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Unexpected behaviour adding a third server

[Anuradha Talur]

- Original Message -
> From: "Pranith Kumar Karampuri" 
> To: "Steve Spence" , gluster-users@gluster.org
> Cc: "Anuradha Talur" 
> Sent: Monday, January 25, 2016 8:24:41 PM
> Subject: Re: [Gluster-users] Unexpected behaviour adding a third server
> 
> 
> On 01/23/2016 02:17 PM, Steve Spence wrote:
> > We've a simple two-server one volume arrangement, replicating ~340k
> > files (15GB) between our web servers.
> >
> > The servers are in AWS, sat in different availability zones. One of
> > the operations for this weekend is to add another pair of machines,
> > one in each AZ.
> >
> > I've deployed the same OS image of the gluster server (3.6) and was
> > under the impression I could add a brick to the existing replica
> > simply by issuing the below:
> >
> > gluster volume add-brick volume1 replica 3 pd-wfe3:/gluster-store
> >
> > And then presumably would add the fourth server by repeating the above
> > with "replica 4" and the fourth server name.
> >
> > The operation appeared to succeed, the brick appears alongside the others:
> >
> > Status: Started
> > Number of Bricks: 1 x 3 = 3
> > Transport-type: tcp
> > Bricks:
> > Brick1: pd-wfe1:/gluster-store
> > Brick2: pd-wfe2:/gluster-store
> > Brick3: pd-wfe3:/gluster-store
> >
> > but almost immediately pd-wfe1 crept up to 100% CPU with the gluster
> > processes, and nginx began timing out serving content from the volume.
> 
> Could you disable client-side healing?
> "gluster volume set  cluster.entry-self-heal off
> "gluster volume set  cluster.data-self-heal off
> "gluster volume set  cluster.metadata-self-heal off
> 
> We are in the process of making this experience smooth for 3.8. by
> introducing throttling of self-heal traffic, automatic healing.
> +Anuradha,
> Could you give him the steps he need to perform after doing
> add-brick until the patch you sent is merged?
Hi Steve,

Once you add-bricks to a replicate volume such that replica count is increased 
(2 to 3 and then 3 to 4 in the case you mentioned), files need to be healed 
from the pre-existing bricks to newly added ones.

The process of triggering heals from old to new bricks is not automatic yet. We 
have a patch for it undergoing review.
Meanwhile, you can follow below given steps to trigger heals from 
self-heal-daemon:

I'm assuming you have added the third brick and yet to add the 4th one.

1) Turn off client side self-healing by using steps given by Pranith.
2) Kill the 3rd brick (newly added one).
3) On the new brick, i.e., pd-wfe3:/gluster-store : setfattr -n 
trusted.afr.dirty -v 0x0001 
4) Say the mount point used for this volume is in /mnt, perform :
  a) touch /mnt/
  b) setfattr -n  -v <1> /mnt
  c) rm /mnt/
  d) setfattr -x  /mnt
These operations will set pending xattrs on the newly added brick such that 
heal is triggered.
5) Bring the brick back up by gluster volume start force
6) Run gluster volume heal  from one of the servers

You can monitor if the files are bring healed or not from gluster volume heal 
 info.

Let me know if there is any clarification required.

> 
> Pranith
> >
> > The glusterfs-glusterd-vol log is filled with this error at pd-wfe1:
> >
> > [2016-01-23 08:43:28.459215] W [socket.c:620:__socket_rwv]
> > 0-management: readv on
> > /var/run/c8bc2f99e7584cb9cf077c4f98d1db2e.socket failed (Invalid argument)
> >
> > while I see this error for the log named by the mount point:
> >
> > [2016-01-23 08:43:28.986379] W
> > [client-rpc-fops.c:306:client3_3_mkdir_cbk] 2-volume1-client-2: remote
> > operation failed: Permission denied. Path: (null)
> >
> > Does anyone have any suggestions how to proceed? I would appreciate
> > any input on this one.
> >
> > Steve
> >
> >
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> 
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] More Peculiar heal behaviour after removing brick

[Anuradha Talur]

- Original Message -
> From: "Lindsay Mathieson" 
> To: "gluster-users" 
> Sent: Saturday, January 23, 2016 9:20:52 AM
> Subject: [Gluster-users] More Peculiar heal behaviour after removing brick
> 
> Maybe I'm doing some wrong here but I'm not sure what, or maybe this is
> normal behaviour?
> 
> 
> All of the following is performed from my vna node which has the highest
> numbered uuid. Indenting applied by me for readability.
> 
> Spoiler, because it happens at the end: removing the vng brick followed by a
> full heal gives this error:
> " Commit failed on vng.proxmox.softlog. Please check log file for details."
> 
> 
> Sets to recreate:
> 1. Create a test volume:
> 
> 
> vna$ gluster volume create test3 rep 3 transport tcp
> vnb.proxmox.softlog:/vmdata/test3 vng.proxmox.softlog:/vmdata/test3
> vna.proxmox.softlog:/vmdata/test3
> vna$ gluster volume set test3 group softlog
> vna$ gluster volume info test3
> 
> 
> Volume Name: test3
> Type: Replicate
> Volume ID: 0be89d63-775c-4eb5-9d98-0a4a87f30fbf
> Status: Created
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: vnb.proxmox.softlog:/vmdata/test3
> Brick2: vng.proxmox.softlog:/vmdata/test3
> Brick3: vna.proxmox.softlog:/vmdata/test3
> Options Reconfigured:
> cluster.data-self-heal-algorithm: full
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> performance.stat-prefetch: off
> performance.strict-write-ordering: on
> performance.write-behind: off
> nfs.enable-ino32: off
> nfs.addr-namelookup: off
> nfs.disable: on
> performance.cache-refresh-timeout: 4
> performance.io-thread-count: 32
> performance.low-prio-threads: 32
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> client.event-threads: 4
> server.event-threads: 4
> cluster.self-heal-window-size: 256
> features.shard-block-size: 512MB
> features.shard: on
> performance.readdir-ahead: off
> vna$ gluster volume start test3
> 
> 2. Immediately remove the vng brick:
> 
> 
> vna$ gluster volume remove-brick test3 replica 2
> vng.proxmox.softlog:/vmdata/test3 force
> vna$ gluster volume info test3
> 
> 
> Volume Name: test3
> Type: Replicate
> Volume ID: 36421a23-68c4-455d-8d4c-e21d9428e1da
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: vnb.proxmox.softlog:/vmdata/test3
> Brick2: vna.proxmox.softlog:/vmdata/test3
> Options Reconfigured:
> cluster.data-self-heal-algorithm: full
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> performance.stat-prefetch: off
> performance.strict-write-ordering: on
> performance.write-behind: off
> nfs.enable-ino32: off
> nfs.addr-namelookup: off
> nfs.disable: on
> performance.cache-refresh-timeout: 4
> performance.io-thread-count: 32
> performance.low-prio-threads: 32
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> client.event-threads: 4
> server.event-threads: 4
> cluster.self-heal-window-size: 256
> features.shard-block-size: 512MB
> features.shard: on
> performance.readdir-ahead: off
> 
> 3. Then run a full heal:
> 
> 
> 
> vna$ gluster volume heal test3 full
> Commit failed on vng.proxmox.softlog. Please check log file for details.
> 
> 
Hi,

Could you provide glusterd logs from vna? It will be in the same directory as 
glustershd logs.

> Weird, because of cause the vng brick has been removed. This happens every
> time.
> 
> I have preserved the glustershd logs from vna & vng if needed. There were no
> heal logs.
> 
> 
> 
> 
> 
> 
> 
> 
> --
> Lindsay Mathieson
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] are they no longer syncing?

[Anuradha Talur]

- Original Message -
> From: "Mark Chaney" 
> To: gluster-users@gluster.org
> Sent: Monday, January 18, 2016 11:21:18 AM
> Subject: [Gluster-users] are they no longer syncing?
> 
> I have a two node cluster setup with iscsi using the image files that
> are stored on the gluster cluster as LUNs. They do appear to be syncing,
> but I have a few questions and I appreciate any help you can give me.
> Thanks for your time!
> 
> 1) Why does the second brick show as N for online?

'N' means that the second brick is not online. Running 'gluster volume start 
 force'
should bring the brick up.

> 2) Why is the healer daemon shown as NA? How can I correct that if it
> needs to be corrected?
Self-heal daemon status on both gluster1 and gluster2 is shown online ( Y ).
It doesn't need to be corrected.

> 3) Should i really be mounting the gluster volumes on each gluster node
> for iscsi access or should i be accessing /var/gluster-storage directly?
> 4) If i only have about 72GB of files stored in gluster, why is each
> gluster host about 155GB? Are their duplicates stored somewhere and why?
> 
> root@gluster1:~# gluster volume status volume1
> Status of volume: volume1
> Gluster process TCP Port  RDMA Port  Online
> Pid
> --
> Brick gluster1:/var/gluster-storage 49152 0  Y
> 3043
> Brick gluster2:/var/gluster-storage N/A   N/AN
> N/A
> NFS Server on localhost 2049  0  Y
> 3026
> Self-heal Daemon on localhost   N/A   N/AY
> 3034
> NFS Server on gluster2  2049  0  Y
> 2738
> Self-heal Daemon on gluster2N/A   N/AY
> 2743
> 
> Task Status of Volume volume1
> --
> There are no active volume tasks
> 
> root@gluster1:~# gluster peer status
> Number of Peers: 1
> 
> Hostname: gluster2
> Uuid: abe7ee21-bea9-424f-ac5c-694bdd989d6b
> State: Peer in Cluster (Connected)
> root@gluster1:~#
> root@gluster1:~# mount | grep gluster
> gluster1:/volume1 on /mnt/glusterfs type fuse.glusterfs
> (rw,default_permissions,allow_other,max_read=131072)
> 
> 
> root@gluster2:~# gluster volume status volume1
> Status of volume: volume1
> Gluster process TCP Port  RDMA Port  Online
> Pid
> --
> Brick gluster1:/var/gluster-storage 49152 0  Y
> 3043
> Brick gluster2:/var/gluster-storage N/A   N/AN
> N/A
> NFS Server on localhost 2049  0  Y
> 2738
> Self-heal Daemon on localhost   N/A   N/AY
> 2743
> NFS Server on gluster1.mgr.example.com  2049  0  Y
> 3026
> Self-heal Daemon on gluster1.mgr.example.co
> m   N/A   N/AY
> 3034
> 
> Task Status of Volume volume1
> --
> There are no active volume tasks
> 
> root@gluster2:~# gluster peer status
> Number of Peers: 1
> 
> Hostname: gluster1.mgr.example.com
> Uuid: dff9118b-a2bd-4cd8-b562-0dfdbd2ea8a3
> State: Peer in Cluster (Connected)
> root@gluster2:~#
> root@gluster2:~# mount | grep gluster
> gluster1:/volume1 on /mnt/glusterfs type fuse.glusterfs
> (rw,default_permissions,allow_other,max_read=131072)
> root@gluster2:~#
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster - Performance issue while copying bulk files/folders

[Anuradha Talur]

- Original Message -
> From: "Srikanth Mampilakal" 
> To: "Anuradha Talur" 
> Cc: "gluster-users" 
> Sent: Thursday, December 31, 2015 1:43:11 AM
> Subject: Re: [Gluster-users] Gluster - Performance issue while copying bulk 
> files/folders
> 
> Hi All,
> 
> I have shared the Gluster Volume Profile for your reference. I am facing
> performance issue with my Gluster setup while copying multiples
> files/folders from client to the mounted gluster volume.
> 
> Any suggestion to improve the copy speed to the Gluster volume is much
> appreciated.

Hi Srikanth,

Thanks for providing the profile info.
Currently in a replicate volume, you will find performance issue while
copying small files. This is because a create operation takes 2 network calls 
and write 
takes 3; this is done to acquire locks and store some glusterfs metadata 
so that data consistency is ensured.

There is work being done to improve this performance.
1) Compound fops - This translator will reduce the network overhead hence 
reducing
   the time taken.
2) Eager locking - This translator will anticipate the locks required,
   and will be helpful in the bulk copy/write cases.
Both of these are improvements are targeted for gluster 3.8.
I hope this info helps.
> 
> Thanks
> Srikanth
> 
> On Tue, Dec 15, 2015 at 10:47 PM, Srikanth Mampilakal <
> shrikanth1...@gmail.com> wrote:
> 
> > Hi Anuradha,
> >
> > Please find the Gluster Volume Profile details
> >
> > time cp -RPp drupal\ code/ /mnt/testmount/copytogluster
> >
> >
> >
> >
> > *Profile info of the volume when you copy dirs/files into glusterfs.*
> >
> >
> > *Time taken to copy (70 MB files/Folder)*
> >
> > [root@GFSCLIENT01 temp]# time cp -RPp /mnt/testmount/
> > /mnt/testmount/copytogluster
> >
> > real29m40.985s
> > user0m0.172s
> > sys 0m1.688s
> >
> >
> >
> > [root@GFSNODE01 ~]# gluster volume profile gv1 info
> > Brick: GFSNODE01:/mnt/perfDisk/gv1
> >
> > --
> > Cumulative Stats:
> >Block Size: 16b+  32b+
> >  64b+
> >  No. of Reads:0 0
> > 0
> > No. of Writes:   1911
> >75
> >
> >Block Size:128b+ 256b+
> > 512b+
> >  No. of Reads:0 0
> > 0
> > No. of Writes:   77   221
> >   297
> >
> >Block Size:   1024b+2048b+
> >  4096b+
> >  No. of Reads:0 0
> > 0
> > No. of Writes:  344   305
> >   336
> >
> >Block Size:   8192b+   16384b+
> > 32768b+
> >  No. of Reads:0 0
> > 0
> > No. of Writes:  160   200
> >87
> >
> >Block Size:  65536b+  131072b+
> >  No. of Reads:0 0
> > No. of Writes:   5938
> >  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
> >   Fop
> >  -   ---   ---   ---   
> >  
> >   0.00   0.00 us   0.00 us   0.00 us   2198
> > RELEASE
> >   0.00   0.00 us   0.00 us   0.00 us 18
> >  RELEASEDIR
> >   0.00  39.75 us  22.00 us  59.00 us  4
> > READDIR
> >   0.01  63.12 us   3.00 us 143.00 us  8
> > OPENDIR
> >   0.01 108.83 us  27.00 us 194.00 us  6
> >  GETXATTR
> >   0.11  58.07 us  28.00 us 124.00 us170
> >  STAT
> >   0.54 113.57 us  46.00 us 258.00 us440
> >  SETXATTR
> >   0.79  97.28 us  23.00 us 224.00 us745
> >  STATFS
> >   1.37  57.40 us  12.00 us 428.00 us   2198
> > FLUSH
> >   3.70  77.12 us  15.00 us 322.00 us   4420
> >  FINODELK
> >   3.94  68.70 us  14.00 us 259.00 us   5278
> > ENTRYLK
> >   4.98 205.68 us  70.00 us2874.00 us   2229
> > WRITE
> >   5.151077.38 us 202.00 us  112584.00 u

Re: [Gluster-users] Gluster - Performance issue while copying bulk files/folders

[Anuradha Talur]

Response inline.

- Original Message -
> From: "Srikanth Mampilakal" 
> To: gluster-users@gluster.org
> Sent: Thursday, December 10, 2015 7:59:04 PM
> Subject: Re: [Gluster-users] Gluster - Performance issue while copying bulk   
> files/folders
> 
> 
> 
> Hi members,
> 
> Really appreciate if you can share your thoughts or any feedback for
> resolving the slow copy issue
> 
> Regards
> Srikanth
> On 10-Dec-2015 2:12 AM, "Srikanth Mampilakal" < srikanth.mampila...@gmail.com
> > wrote:
> 
> 
> 
> Hi,
> 
> 
> I have production gluster file service used as a shared storage where the
> content management system uses it as document root. I have run in to a
> performance issue with the gluster/fuse client.
> 
> Looking for your thoughts and experience in resolving Gluster performance
> issues:
> 
> Gluster Infrastructure
> 
> Gluster version :GlusterFS 3.7.6
> 
> 2 gluster nodes of the same config below
> 
> Redhat EL7.0-64
> Memory : 4GB
> Processor : 2 x 2.0 Ghz
> Network : 100 Mbps
> File Storage Volume : NETAPP Storage LUN with 2.0 IOPS/GB
> 
> Gluster Volume information:
> 
> [root@GlusterFileServe1 ~]# gluster volume info
> 
> Volume Name: prodcmsroot
> Type: Replicate
> Volume ID: f1284bf0-1939-46f9-a672-a7716e362947
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: Server1:/glusterfs/brick1/prodcmsroot
> Brick2: Server2:/glusterfs/brick1/prodcmsroot
> Options Reconfigured:
> performance.io-thread-count: 64
> performance.cache-size: 1073741824
> performance.readdir-ahead: on
> performance.write-behind-window-size: 524288
> 
> [root@GlusterFileServe1 ~]#
> 
> The replication between Gluster node are quick and consistent.
> 
> The apache webservers are accessing the Gluster volume using native gluster
> fuse client and located in the same VLAN as the Gluster Server.
> 
> GlusterFileServe1:/prodcmsroot /mnt/glusterfs glusterfs
> direct-io-mode=disable,defaults,_netdev 0 0
> 
> The server utilization (memory,cpu,network and disk 1/0) is relatively low
> 
> I am experiencing very slow performance while copying multiple file/folders
> (approx 75 MB) and it takes atleast approx 35 min. Even copy a folder (with
> multiple files/subfolders) within the Gluster volume take the same time.
> 
> However, if I do dd to check the copy speed, I get the below result.
> 
> [root@ClientServer ~]# time sh -c "dd if=/dev/zero of=/mnt/testmount/test.tmp
> bs=4k count=2 && sync"
> 2+0 records in
> 2+0 records out
> 8192 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s
> 
> real 0m17.337s
> user 0m0.031s
> sys 0m0.317s
> 
> 
> Anyone experience the same kind of performance issue, please let me know your
> thoughts.
> 
Hi Srikanth,
 
Could you please provide the following information so that the reason behind
slow copy can be deduced?

1) Profile info of the volume when you copy dirs/files into glusterfs.
2) Profile info of the volume when you copy dirs/files within glusterfs.

The following steps should help you with profile info:
1) gluster volume profile  start
2) Perform copy operations
3) gluster volume profile  info (you will get stats of the FOPs at 
this point)
4) gluster volume profile  stop

Please follow steps 1 through 4 twice. Once for copy into glusterfs and once 
for copy
within.

> Cheers
> Srikanth
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Shard Volume testing (3.7.5)

[Anuradha Talur]

- Original Message -
> From: "Lindsay Mathieson" 
> To: "Krutika Dhananjay" 
> Cc: "gluster-users" 
> Sent: Wednesday, October 28, 2015 1:08:33 PM
> Subject: Re: [Gluster-users] Shard Volume testing (3.7.5)
> 
> 
> On 28 October 2015 at 17:03, Krutika Dhananjay < kdhan...@redhat.com > wrote:
> 
> 
> 
> So sharding also helps with better disk utilization in distributed-replicated
> volumes for large files (like VM images).
> ..
> 
> There are other long-term benefits one could reap from using sharding: for
> instance, for someone who might want to use tiering in VM store use-case,
> having sharding will be beneficial in terms of only migrating the shards
> between hot and cold tiers, as opposed to moving large files in full, even
> if only a small portion of the file is changed/accessed. :)
> 
> 
> Interesting points, thanks.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Yes. So Paul Cuzner and Satheesaran who have been testing sharding here have
> reported better write performance with 512M shards. I'd be interested to
> know what you feel about performance with relatively larger shards (think
> 512M).
> 
> Seq Read speeds basically tripled, and seq writes improved to the limit of
> the network connection.
> 
> OK. And what about the data heal performance with 512M shards? Satisfactory?
> 
> 
> Easily satisfactory, a bit slower than the 4MB shard but still way faster
> than a full multi GB file heal :)
> 
> 
> Something I have noticed, is that the heal info (gluster volume heal
>  info) can be very slow to return, as in many 10's of seconds -
> is there a way to speed that up?
> 
Yes, there is a way to speed it up. Basically the process of finding out
whether a file needs heal or not takes some time, leading to slow heal info.
This decision making can be done in a faster way. I'm working on the approach
and will send a patch in the coming days.
> It would be every useful if there was a command that quickly gave
> summary/progress status, e.g "There are  shards to be healed"
> 
> 
> --
> Lindsay
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster 3.7.3 - volume heal info hangs - unknown heal status

[Anuradha Talur]

- Original Message -
> From: "Andreas Mather" 
> To: "Anuradha Talur" 
> Cc: "Gluster-users@gluster.org List" 
> Sent: Thursday, September 24, 2015 6:59:38 PM
> Subject: Re: [Gluster-users] gluster 3.7.3 - volume heal info hangs - unknown 
> heal status
> 
> Hi Anuradha!
> 
> Thanks for your reply! Attached you can find the dump files. As I'm not
> sure if they make their way through as attachments, here're links to them
> as well:
> 
> brick1 - http://pastebin.com/3ivkhuRH
> brick2 - http://pastebin.com/77sT1mut
Hi,

I see some blocked locks from the statedump.
Could you let me know what kind of workload you had when you observed the hang?
> 
> - Andreas
> 
> 
> 
> 
> On Thu, Sep 24, 2015 at 3:18 PM, Anuradha Talur  wrote:
> 
> >
> >
> > - Original Message -
> > > From: "Andreas Mather" 
> > > To: "Gluster-users@gluster.org List" 
> > > Sent: Thursday, September 24, 2015 1:24:12 PM
> > > Subject: [Gluster-users] gluster 3.7.3 - volume heal info hangs -
> > unknown heal status
> > >
> > > Hi!
> > >
> > > Our provider had network maintenance this night, so 2 of our 4 servers
> > got
> > > disconnected and reconnected. Since we knew this was coming, we shifted
> > all
> > > work load off the affected servers. This morning, most of the cluster
> > seems
> > > fine, but for one volume, no heal info can be retrieved, so we basically
> > > don't know about the healing state of the volume. The volume is a
> > replica 2
> > > volume between vhost4-int/brick1 and vhost3-int/brick2.
> > >
> > > The volume is accessible, but since I don't get any heal info, I don't
> > know
> > > if it is probably replicated. Any help to resolve this situation is
> > highly
> > > appreciated.
> > >
> > > hangs forever:
> > > [root@vhost4 ~]# gluster volume heal vol4 info
> > >
> > > glfsheal-vol4.log:
> > > [2015-09-24 07:47:59.284723] I [MSGID: 101190]
> > > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread
> > with
> > > index 1
> > > [2015-09-24 07:47:59.293735] I [MSGID: 101190]
> > > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread
> > with
> > > index 2
> > > [2015-09-24 07:47:59.294061] I [MSGID: 104045] [glfs-master.c:95:notify]
> > > 0-gfapi: New graph 76686f73-7434-2e61-6c6c-61626f757461 (0) coming up
> > > [2015-09-24 07:47:59.294081] I [MSGID: 114020] [client.c:2118:notify]
> > > 0-vol4-client-1: parent translators are ready, attempting connect on
> > > transport
> > > [2015-09-24 07:47:59.309470] I [MSGID: 114020] [client.c:2118:notify]
> > > 0-vol4-client-2: parent translators are ready, attempting connect on
> > > transport
> > > [2015-09-24 07:47:59.310525] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
> > > 0-vol4-client-1: changing port to 49155 (from 0)
> > > [2015-09-24 07:47:59.315958] I [MSGID: 114057]
> > > [client-handshake.c:1437:select_server_supported_programs]
> > 0-vol4-client-1:
> > > Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > > [2015-09-24 07:47:59.316481] I [MSGID: 114046]
> > > [client-handshake.c:1213:client_setvolume_cbk] 0-vol4-client-1:
> > Connected to
> > > vol4-client-1, attached to remote volume '/storage/brick2/brick2'.
> > > [2015-09-24 07:47:59.316495] I [MSGID: 114047]
> > > [client-handshake.c:1224:client_setvolume_cbk] 0-vol4-client-1: Server
> > and
> > > Client lk-version numbers are not same, reopening the fds
> > > [2015-09-24 07:47:59.316538] I [MSGID: 108005]
> > [afr-common.c:3960:afr_notify]
> > > 0-vol4-replicate-0: Subvolume 'vol4-client-1' came back up; going online.
> > > [2015-09-24 07:47:59.317150] I [MSGID: 114035]
> > > [client-handshake.c:193:client_set_lk_version_cbk] 0-vol4-client-1:
> > Server
> > > lk version = 1
> > > [2015-09-24 07:47:59.320898] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
> > > 0-vol4-client-2: changing port to 49154 (from 0)
> > > [2015-09-24 07:47:59.325633] I [MSGID: 114057]
> > > [client-handshake.c:1437:select_server_supported_programs]
> > 0-vol4-client-2:
> > > Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > > [2015-09-24 07:47:59.325780] I [MSGID: 114046]
> > > [client-handshake.c:1213:client_setvolume_cbk] 0-vol4-client-2:
> > Connected to
> > > vol4

Re: [Gluster-users] gluster 3.7.3 - volume heal info hangs - unknown heal status

[Anuradha Talur]

- Original Message -
> From: "Andreas Mather" 
> To: "Gluster-users@gluster.org List" 
> Sent: Thursday, September 24, 2015 1:24:12 PM
> Subject: [Gluster-users] gluster 3.7.3 - volume heal info hangs - unknown 
> heal status
> 
> Hi!
> 
> Our provider had network maintenance this night, so 2 of our 4 servers got
> disconnected and reconnected. Since we knew this was coming, we shifted all
> work load off the affected servers. This morning, most of the cluster seems
> fine, but for one volume, no heal info can be retrieved, so we basically
> don't know about the healing state of the volume. The volume is a replica 2
> volume between vhost4-int/brick1 and vhost3-int/brick2.
> 
> The volume is accessible, but since I don't get any heal info, I don't know
> if it is probably replicated. Any help to resolve this situation is highly
> appreciated.
> 
> hangs forever:
> [root@vhost4 ~]# gluster volume heal vol4 info
> 
> glfsheal-vol4.log:
> [2015-09-24 07:47:59.284723] I [MSGID: 101190]
> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with
> index 1
> [2015-09-24 07:47:59.293735] I [MSGID: 101190]
> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with
> index 2
> [2015-09-24 07:47:59.294061] I [MSGID: 104045] [glfs-master.c:95:notify]
> 0-gfapi: New graph 76686f73-7434-2e61-6c6c-61626f757461 (0) coming up
> [2015-09-24 07:47:59.294081] I [MSGID: 114020] [client.c:2118:notify]
> 0-vol4-client-1: parent translators are ready, attempting connect on
> transport
> [2015-09-24 07:47:59.309470] I [MSGID: 114020] [client.c:2118:notify]
> 0-vol4-client-2: parent translators are ready, attempting connect on
> transport
> [2015-09-24 07:47:59.310525] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
> 0-vol4-client-1: changing port to 49155 (from 0)
> [2015-09-24 07:47:59.315958] I [MSGID: 114057]
> [client-handshake.c:1437:select_server_supported_programs] 0-vol4-client-1:
> Using Program GlusterFS 3.3, Num (1298437), Version (330)
> [2015-09-24 07:47:59.316481] I [MSGID: 114046]
> [client-handshake.c:1213:client_setvolume_cbk] 0-vol4-client-1: Connected to
> vol4-client-1, attached to remote volume '/storage/brick2/brick2'.
> [2015-09-24 07:47:59.316495] I [MSGID: 114047]
> [client-handshake.c:1224:client_setvolume_cbk] 0-vol4-client-1: Server and
> Client lk-version numbers are not same, reopening the fds
> [2015-09-24 07:47:59.316538] I [MSGID: 108005] [afr-common.c:3960:afr_notify]
> 0-vol4-replicate-0: Subvolume 'vol4-client-1' came back up; going online.
> [2015-09-24 07:47:59.317150] I [MSGID: 114035]
> [client-handshake.c:193:client_set_lk_version_cbk] 0-vol4-client-1: Server
> lk version = 1
> [2015-09-24 07:47:59.320898] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
> 0-vol4-client-2: changing port to 49154 (from 0)
> [2015-09-24 07:47:59.325633] I [MSGID: 114057]
> [client-handshake.c:1437:select_server_supported_programs] 0-vol4-client-2:
> Using Program GlusterFS 3.3, Num (1298437), Version (330)
> [2015-09-24 07:47:59.325780] I [MSGID: 114046]
> [client-handshake.c:1213:client_setvolume_cbk] 0-vol4-client-2: Connected to
> vol4-client-2, attached to remote volume '/storage/brick1/brick1'.
> [2015-09-24 07:47:59.325791] I [MSGID: 114047]
> [client-handshake.c:1224:client_setvolume_cbk] 0-vol4-client-2: Server and
> Client lk-version numbers are not same, reopening the fds
> [2015-09-24 07:47:59.46] I [MSGID: 114035]
> [client-handshake.c:193:client_set_lk_version_cbk] 0-vol4-client-2: Server
> lk version = 1
> [2015-09-24 07:47:59.334545] I [MSGID: 108031]
> [afr-common.c:1745:afr_local_discovery_cbk] 0-vol4-replicate-0: selecting
> local read_child vol4-client-2
> [2015-09-24 07:47:59.335833] I [MSGID: 104041]
> [glfs-resolve.c:862:__glfs_active_subvol] 0-vol4: switched to graph
> 76686f73-7434-2e61-6c6c-61626f757461 (0)
> 
> Questions to this output:
> -) Why does it report " Using Program GlusterFS 3.3, Num (1298437), Version
> (330) ". We run 3.7.3 ?!
> -) guster logs timestamps in UTC not taking server timezone into account. Is
> there a way to fix this?
> 
> etc-glusterfs-glusterd.vol.log:
> no logs to after volume heal info command
> 
> storage-brick1-brick1.log:
> [2015-09-24 07:47:59.325720] I [login.c:81:gf_auth] 0-auth/login: allowed
> user names: 67ef1559-d3a1-403a-b8e7-fb145c3acf4e
> [2015-09-24 07:47:59.325743] I [MSGID: 115029]
> [server-handshake.c:610:server_setvolume] 0-vol4-server: accepted client
> from
> vhost4.allaboutapps.at-14900-2015/09/24-07:47:59:282313-vol4-client-2-0-0
> (version: 3.7.3)
> 
> storage-brick2-brick2.log:
> no logs to after volume heal info command
> 
> 
Hi Andreas,

Could you please provide the following information so that we can understand 
why the command is hanging?
When the command is hung, run the following command from one of the servers:
`gluster volume statedump `
This command will generate statedumps of glusterfsd processes in the servers. 
You can find them at /var/run/gluster . A typical statedump for a bri

Re: [Gluster-users] Splitbrain log entries

[Anuradha Talur]

- Original Message -
> From: "Jesper Led Lauridsen TS Infra server" 
> To: gluster-users@gluster.org
> Sent: Tuesday, August 18, 2015 12:26:05 PM
> Subject: [Gluster-users] Splitbrain log entries
> 
> 
> 
> Hello
> 
> 
> 
> When I execute ”gluster volume heal volume info split-brain” there is a lot
> of entries from about a month ago.
> 
> 
> 
> Firstly, the ‘info split-brain’ is this just informational/history logging?
> (md5sum on both replica’s is the same). Secondly is there a way to clear
> this log/information?

In 3.6.2, info split-brain shows history too. From 3.7 onwards it is just
informational. Restarting self-heal-deamon should clear this information.
> 
> 
> 
> # glusterd --version
> 
> glusterfs 3.6.2 built on Jan 22 2015 12:58:10
> 
> 
> 
> Thanks
> 
> Jesper
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] DIRECT_IO_TEST in split brain

[Anuradha Talur]

- Original Message -
> From: p...@email.cz
> To: "GLUSTER-USERS" 
> Sent: Wednesday, June 10, 2015 8:44:51 PM
> Subject: [Gluster-users] DIRECT_IO_TEST in split brain
> 
> hello,
> pls , how to eliminate this split brain on
> - centos 7.1
> - glusterfs-3.7.1-1.el7.x86_64

The following link will help you understand various ways in which split-brain 
can be resolved.
https://github.com/gluster/glusterfs/blob/master/doc/features/heal-info-and-split-brain-resolution.md
Could you go through it and see if it helps?
> 
> # gluster volume heal R2 info
> Brick cl1:/R2/R2/
> /__DIRECT_IO_TEST__ - Is in split-brain
> 
> Number of entries: 1
> 
> Brick cl3:/R2/R2/
> /__DIRECT_IO_TEST__ - Is in split-brain
> 
> Number of entries: 1
> 
> -
> # gluster volume status R2
> Status of volume: R2
> Gluster process TCP Port RDMA Port Online Pid
> --
> Brick 10.10.1.67:/R2/R2 49152 0 Y 3617
> Brick 10.10.1.82:/R2/R2 49152 0 Y 3337
> NFS Server on localhost 2049 0 Y 3605
> Self-heal Daemon on localhost N/A N/A Y 3610
> NFS Server on 10.10.1.82 2049 0 Y 3344
> Self-heal Daemon on 10.10.1.82 N/A N/A Y 3349
> NFS Server on 10.10.1.69 2049 0 Y 3432
> Self-heal Daemon on 10.10.1.69 N/A N/A Y 3440
> 
> Task Status of Volume R2
> --
> There are no active volume tasks
> 
> [root@cl1 ~]# gluster volume info R2
> 
> Volume Name: R2
> Type: Replicate
> Volume ID: 6c30118d-8f71-4593-9607-d0ded7401783
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 10.10.1.67:/R2/R2
> Brick2: 10.10.1.82:/R2/R2
> Options Reconfigured:
> cluster.quorum-count: 1
> storage.owner-gid: 36
> storage.owner-uid: 36
> cluster.server-quorum-type: none
> cluster.quorum-type: fixed
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.stat-prefetch: off
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> cluster.self-heal-daemon: enable
> 
> regs.Pa.
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Strange output from gluster volume heal info split-brain command

[Anuradha Talur]

- Original Message -
> From: "Ravishankar N" 
> To: "Andreas Hollaus" , 
> gluster-users@gluster.org, "Anuradha Talur" 
> Sent: Wednesday, June 10, 2015 7:09:34 AM
> Subject: Re: [Gluster-users] Strange output from gluster volume heal  
> info split-brain command
> 
> `info split-brain` has been re-implemented in in glusterfs 3.7 (the
> logic is now in glfs-heal.c) and should work correctly. CC'in Anuradha
> who implemented it for confirmation.
> 

As Ravi said, info split-brain should work correctly in glusterfs 3.7 .

> -Ravi
> 
> On 06/09/2015 08:08 PM, Andreas Hollaus wrote:
> > Hi,
> >
> > I detected a directory in split-brain on my file system, but I'm surprised
> > that
> > 'gluster volume heal  info split-brain' doesn't report it.
> > I would have expected it to show only split-brain issues, compared to
> > 'gluster volume
> > heal  info' that shows more info
> > regarding the healing process. Isn't this strange?
> >
> > # pwd
> > /c/Mirror_test_150608/FTC_switch
> > # gluster volume heal c_glstr info
> > Brick oamhost:/opt/lvmdir/c2/brick/
> > /Mirror_test_150608/FTC_switch/File_both_mp_write_restart/filesystest_5_2
> > /Mirror_test_150608/FTC_switch/File_both_mp_write_restart/filesystest_5_0
> > /Mirror_test_150608/FTC_switch/File_both_mp_write_restart - Is in
> > split-brain
> >
> > /Mirror_test_150608/FTC_switch/File_both_mp_write_restart/filesystest_5_3
> > /Mirror_test_150608/FTC_switch/File_both_mp_write_restart/filesystest_5_1
> > Number of entries: 5
> >
> > Brick oamhost:/opt/lvmdir/c2/brick/
> > /Mirror_test_150608/FTC_switch/File_both_mp_write_restart - Is in
> > split-brain
> >
> > /Mirror_test_150608/FTC_switch/File_both_mp_write_restart/filesystest_5_0
> > /Mirror_test_150608/FTC_switch/File_both_mp_write_restart/filesystest_5_1
> > /Mirror_test_150608/FTC_switch/File_both_mp_write_restart/filesystest_5_2
> > /Mirror_test_150608/FTC_switch/File_both_mp_write_restart/filesystest_5_3
> > Number of entries: 5
> >
> > # gluster volume heal c_glstr info split-brain
> > Gathering list of split brain entries on volume c_glstr has been successful
> >
> > Brick 10.32.0.48:/opt/lvmdir/c2/brick
> > Number of entries: 0
> >
> > Brick 10.32.1.144:/opt/lvmdir/c2/brick
> > Number of entries: 0

However in glusterfs 3.6.2, info split-brain wouldn't always provide updated
information. In glusterfs 3.6.2, you can gather required information from heal 
info,
or restarting self-heal daemon should also give you updated split-brain list.

> >
> > # gluster --version
> > glusterfs 3.6.2 built on Apr 22 2015 02:02:49
> > Repository revision: git://git.gluster.com/glusterfs.git
> > Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
> > GlusterFS comes with ABSOLUTELY NO WARRANTY.
> > You may redistribute copies of GlusterFS under the terms of the GNU General
> > Public
> > License.
> >
> >
> > Regards
> > Andreas
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> 
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Output_vol_info error in cli.log

[Anuradha Talur]

- Original Message -
> From: "RASTELLI Alessandro" 
> To: gluster-users@gluster.org
> Sent: Wednesday, May 13, 2015 2:38:43 PM
> Subject: [Gluster-users] Output_vol_info error in cli.log
> 
> 
> 
> Hi,
> 
> I’m getting this error every second in cli.log:
> 
> [2015-05-13 09:04:57.200421] E
> [cli-xml-output.c:2867:cli_xml_output_vol_info_end] 0-cli: Returning 0
> 

It is returning "0", there is no error here.
From the code, it looks like a log that was supposed to be logged in "DEBUG" 
level
has been logged as "ERROR" by mistake.

+CCing author of the patch.
> 
> 
> in attachment the full log.
> 
> How can I solve that?
> 
> 
> 
> Regards
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] ... i was able to produce a split brain...

[Anuradha Talur]

- Original Message -
> From: "Jeff Darcy" 
> To: "Pranith Kumar Karampuri" 
> Cc: "Joe Julian" , gluster-users@gluster.org, 
> "Anuradha Talur" 
> Sent: Wednesday, January 28, 2015 9:01:15 PM
> Subject: Re: [Gluster-users] ... i was able to produce a split brain...
> 
> > On 01/27/2015 11:43 PM, Joe Julian wrote:
> > > No, there's not. I've been asking for this for years.
> > Hey Joe,
> > Vijay and I were just talking about this today. We were
> > wondering if you could give us the inputs to make it a feature to
> > implement.
> > Here are the questions I have:
> > Basic requirements if I understand correctly are as follows:
> > 1) User should be able to fix the split-brain without any intervention
> > from admin as the user knows best about the data.
> > 2) He should be able to preview some-how about the data before selecting
> > the copy which he/she wants to preserve.
> 
> One possibility would be to implement something like DHT's
> filter_loc_subvol_key, though perhaps using child indices instead of
> translator names.  Another would be a script which can manipulate
> volfiles and use GFAPI to fetch a specific version of a file.  I've
> written several scripts which can do the necessary volfile manipulation.
> If we finally have a commitment to do something like this, actually
> implementing it will be the easy part.
> 

Hello,

Pranith and I had a discussion regarding this issue and here is what we have in 
our mind right now.

We plan to provide the user commands to execute from mount so that he can 
access the files in split-brain. This way he can choose which copy is to be 
used as source. The user will have to perform a set of getfattrs and setfattrs 
(on virtual xattrs) to decide which child to choose as source and inform AFR 
with his decision.

A) To know the split-brain status :
getfattr -n trusted.afr.split-brain-status 

This will provide user with the following details -
1) Whether the file is in metadata split-brain
2) Whether the file is in data split-brain

It will also list the name of afr-children to choose from. Something like :
Option0: client-0
Option1: client-1

We also tell the user what the user could do to view metadata/data info; like 
stat to get metadata etc.

B) Now the user has to choose one of the options (client-x/client-y..) to 
inspect the files.
e.g., setfattr -n trusted.afr.split-brain-choice -v client-0 
We save the read-child info in inode-ctx in order to provide the user access to 
the file in split-brain from that child. Once the user inspects the file, he 
proceeds to do the same from the other child of replica pair and makes an 
informed decision.

C) Once the above steps are done, AFR is to be informed with the final choice 
for source. This is achieved by -
(say the fresh copy is in client-0)
e.g., setfattr -n trusted.afr.split-brain-heal-finalize -v client-0 

This child will be chosen as source and split-brain resolution will be done.

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] v3.6.1 vs v3.5.2 self heal - help (Nagios related)

[Anuradha Talur]

- Original Message -
> From: "Joe Julian" 
> To: "Anuradha Talur" , "Vince Loschiavo" 
> 
> Cc: "gluster-users@gluster.org" 
> Sent: Friday, November 21, 2014 12:06:27 PM
> Subject: Re: [Gluster-users] v3.6.1 vs v3.5.2 self heal - help (Nagios 
> related)
> 
> 
> 
> On November 20, 2014 10:01:45 PM PST, Anuradha Talur 
> wrote:
> >
> >
> >- Original Message -
> >> From: "Vince Loschiavo" 
> >> To: "gluster-users@gluster.org" 
> >> Sent: Wednesday, November 19, 2014 9:50:50 PM
> >> Subject: [Gluster-users] v3.6.1 vs v3.5.2 self heal - help (Nagios
> >related)
> >> 
> >> 
> >> Hello Gluster Community,
> >> 
> >> I have been using the Nagios monitoring scripts, mentioned in the
> >below
> >> thread, on 3.5.2 with great success. The most useful of these is the
> >self
> >> heal.
> >> 
> >> However, I've just upgraded to 3.6.1 on the lab and the self heal
> >daemon has
> >> become quite aggressive. I continually get alerts/warnings on 3.6.1
> >that
> >> virt disk images need self heal, then they clear. This is not the
> >case on
> >> 3.5.2. This
> >> 
> >> Configuration:
> >> 2 node, 2 brick replicated volume with 2x1GB LAG network between the
> >peers
> >> using this volume as a QEMU/KVM virt image store through the fuse
> >mount on
> >> Centos 6.5.
> >> 
> >> Example:
> >> on 3.5.2:
> >> gluster volume heal volumename info: shows the bricks and number of
> >entries
> >> to be healed: 0
> >> 
> >> On v3.5.2 - During normal gluster operations, I can run this command
> >over and
> >> over again, 2-4 times per second, and it will always show 0 entries
> >to be
> >> healed. I've used this as an indicator that the bricks are
> >synchronized.
> >> 
> >> Last night, I upgraded to 3.6.1 in lab and I'm seeing different
> >behavior.
> >> Running gluster volume heal volumename info , during normal
> >operations, will
> >> show a file out-of-sync, seemingly between every block written to
> >disk then
> >> synced to the peer. I can run the command over and over again, 2-4
> >times per
> >> second, and it will almost always show something out of sync. The
> >individual
> >> files change, meaning:
> >> 
> >> Example:
> >> 1st Run: shows file1 out of sync
> >> 2nd run: shows file 2 and file 3 out of sync but file 1 is now in
> >sync (not
> >> in the list)
> >> 3rd run: shows file 3 and file 4 out of sync but file 1 and 2 are in
> >sync
> >> (not in the list).
> >> ...
> >> nth run: shows 0 files out of sync
> >> nth+1 run: shows file 3 and 12 out of sync.
> >> 
> >> From looking at the virtual machines running off this gluster volume,
> >it's
> >> obvious that gluster is working well. However, this obviously plays
> >havoc
> >> with Nagios and alerts. Nagios will run the heal info and get
> >different and
> >> non-useful results each time, and will send alerts.
> >> 
> >> Is this behavior change (3.5.2 vs 3.6.1) expected? Is there a way to
> >tune the
> >> settings or change the monitoring method to get better results into
> >Nagios.
> >> 
> >In 3.6.1 the way heal info command works is different from that in
> >3.5.2. In 3.6.1, it is self-heal daemon that gathers the entries that
> >might need healing. Currently, in 3.6.1, there isn't a method to
> >distinguish between a file that is being healed and a file with
> >on-going I/O while listing. Hence you see files with normal operation
> >too listed in the output of heal info command.
> 
> How did that regression pass?!
Test cases to check this condition was not written in regression tests.
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] v3.6.1 vs v3.5.2 self heal - help (Nagios related)

[Anuradha Talur]

- Original Message -
> From: "Vince Loschiavo" 
> To: "gluster-users@gluster.org" 
> Sent: Wednesday, November 19, 2014 9:50:50 PM
> Subject: [Gluster-users] v3.6.1 vs v3.5.2 self heal - help (Nagios related)
> 
> 
> Hello Gluster Community,
> 
> I have been using the Nagios monitoring scripts, mentioned in the below
> thread, on 3.5.2 with great success. The most useful of these is the self
> heal.
> 
> However, I've just upgraded to 3.6.1 on the lab and the self heal daemon has
> become quite aggressive. I continually get alerts/warnings on 3.6.1 that
> virt disk images need self heal, then they clear. This is not the case on
> 3.5.2. This
> 
> Configuration:
> 2 node, 2 brick replicated volume with 2x1GB LAG network between the peers
> using this volume as a QEMU/KVM virt image store through the fuse mount on
> Centos 6.5.
> 
> Example:
> on 3.5.2:
> gluster volume heal volumename info: shows the bricks and number of entries
> to be healed: 0
> 
> On v3.5.2 - During normal gluster operations, I can run this command over and
> over again, 2-4 times per second, and it will always show 0 entries to be
> healed. I've used this as an indicator that the bricks are synchronized.
> 
> Last night, I upgraded to 3.6.1 in lab and I'm seeing different behavior.
> Running gluster volume heal volumename info , during normal operations, will
> show a file out-of-sync, seemingly between every block written to disk then
> synced to the peer. I can run the command over and over again, 2-4 times per
> second, and it will almost always show something out of sync. The individual
> files change, meaning:
> 
> Example:
> 1st Run: shows file1 out of sync
> 2nd run: shows file 2 and file 3 out of sync but file 1 is now in sync (not
> in the list)
> 3rd run: shows file 3 and file 4 out of sync but file 1 and 2 are in sync
> (not in the list).
> ...
> nth run: shows 0 files out of sync
> nth+1 run: shows file 3 and 12 out of sync.
> 
> From looking at the virtual machines running off this gluster volume, it's
> obvious that gluster is working well. However, this obviously plays havoc
> with Nagios and alerts. Nagios will run the heal info and get different and
> non-useful results each time, and will send alerts.
> 
> Is this behavior change (3.5.2 vs 3.6.1) expected? Is there a way to tune the
> settings or change the monitoring method to get better results into Nagios.
> 
In 3.6.1 the way heal info command works is different from that in 3.5.2. In 
3.6.1, it is self-heal daemon that gathers the entries that might need healing. 
Currently, in 3.6.1, there isn't a method to distinguish between a file that is 
being healed and a file with on-going I/O while listing. Hence you see files 
with normal operation too listed in the output of heal info command.
> Thank you,
> 
> --
> -Vince Loschiavo
> 
> 
> On Wed, Nov 19, 2014 at 4:35 AM, Humble Devassy Chirammal <
> humble.deva...@gmail.com > wrote:
> 
> 
> 
> Hi Gopu,
> 
> Awesome !!
> 
> We can have a Gluster blog about this implementation.
> 
> --Humble
> 
> 
> 
> --Humble
> 
> 
> On Wed, Nov 19, 2014 at 5:38 PM, Gopu Krishnan < gopukrishnan...@gmail.com >
> wrote:
> 
> 
> 
> Thanks for all your help... I was able to configure nagios using the
> glusterfs plugin. Following link shows how I configured it. Hope it helps
> someone else.:
> 
> http://gopukrish.wordpress.com/2014/11/16/monitor-glusterfs-using-nagios-plugin/
> 
> On Sun, Nov 16, 2014 at 11:44 AM, Humble Devassy Chirammal <
> humble.deva...@gmail.com > wrote:
> 
> 
> 
> Hi,
> 
> Please look at this thread
> http://gluster.org/pipermail/gluster-users.old/2014-June/017819.html
> 
> Btw, if you are around, we have a talk on same topic in upcoming GlusterFS
> India meetup.
> 
> Details can be fetched from:
> http://www.meetup.com/glusterfs-India/
> 
> --Humble
> 
> --Humble
> 
> 
> On Sun, Nov 16, 2014 at 11:23 AM, Gopu Krishnan < gopukrishnan...@gmail.com >
> wrote:
> 
> 
> 
> How can we monitor the glusters and alert us if something happened wrong. I
> found some nagios plugins and didn't work until this time. I am still
> experimenting with those. Any suggestions would be much helpful
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster volume heal info question

[Anuradha Talur]

- Original Message -
> From: "Lindsay Mathieson" 
> To: "gluster-users" 
> Sent: Tuesday, November 18, 2014 4:24:51 PM
> Subject: [Gluster-users] gluster volume heal  info question
> 
> 
> 
> When I run the subject I get:
> 
> 
> 
> root@vnb:~# gluster volume heal datastore1 info
> 
> Brick vnb:/mnt/gluster-brick1/datastore/
> 
> /images/100/vm-100-disk-1.qcow2 - Possibly undergoing heal
> 
> 
> 
> 
> 
> 
> 
> /images/102/vm-102-disk-1.qcow2
> 
> 
> 
> 
> 
> /images/400/vm-400-disk-1.qcow2
> 
> Number of entries: 8
> 
> 
> 
> 
> 
> 
> 
> it has 8 entries but only one is possibly undergoing heal. Whats with the
> other 7?
> 
heal info command that you executed basically gives a list of files to be 
healed. So in the above output, 1 entry is possibly getting healed and other 7 
need to be healed.
> 
> 
> And what is a gfid?

In glusterfs, gfid (glusterfs ID) is similar to an inode number. It is a unique 
identification number given to each file.
> 
> 
> 
> thanks
> 
> 
> 
> 
> 
> --
> 
> Lindsay
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users