[Gluster-users] split brain due to conflicting trusted.glusterfs.mdata xattr?

Dear list, Today I received a "Directory not empty" error while trying to remove a directory from the FUSE mount of a distribute–replicate volume. Looking in the directory I found a few files with question marks: -?? ? ? ? ?? ._Log.out I checked the volume heal i

Re: [Gluster-users] split-brain errors under heavy load when one brick down

Thank you for replying! > Okay so 0-cm_shared-replicate-1 means these 3 bricks: > > Brick4: 172.23.0.6:/data/brick_cm_shared > Brick5: 172.23.0.7:/data/brick_cm_shared > Brick6: 172.23.0.8:/data/brick_cm_shared The above is correct. > Were there any pending self-heals for this volume? Is it po

Re: [Gluster-users] split-brain errors under heavy load when one brick down

On 16/09/19 7:34 pm, Erik Jacobson wrote: Example errors: ex1 [2019-09-06 18:26:42.665050] E [MSGID: 108008] [afr-read-txn.c:123:afr_read_txn_refresh_done] 0-cm_shared-replicate-1: Failing ACCESS on gfid ee3f5646-9368-4151-92a3-5b8e7db1fbf9: split-brain observed. [Input/output error] Okay so

[Gluster-users] split-brain errors under heavy load when one brick down

Hello all. I'm new to the list but not to gluster. We are using gluster to service NFS boot on a top500 cluster. It is a Distributed-Replicate volume 3x9. We are having a problem when one server in a subvolume goes down, we get random missing files and split-brain errors in the nfs.log file. We

[Gluster-users] split brain heal - socket not connected

Hi, unfortunatelly I have a directory in split-brain state. If I do a gluster volume heal split-brain source-brick gluster:/glusterfs/brick I get a socket not connected. How can I manually heal that directory? Best regards and thanks in advance Bene -- forumZFD Entschieden für Frieden|C

Re: [Gluster-users] split brain? but where?

Next I ran a test and your find worksI am wondering if I can simply delete this GFID? 8><- [root@glusterp2 fb]# find /bricks/brick1/gv0/ -samefile /bricks/brick1/gv0/.glusterfs/74/75/7475bd15-05a6-45c2-b395-bc9fd3d1763f /bricks/brick1/gv0/.glusterfs/74/75/7475bd15-05a6-45c2-b395-bc9fd3d176

Re: [Gluster-users] split brain? but where?

I tried looking for a file of the same size and the gfid doesnt show up, 8><--- [root@glusterp2 fb]# pwd /bricks/brick1/gv0/.glusterfs/ea/fb [root@glusterp2 fb]# ls -al total 3130892 drwx--. 2 root root 64 May 22 13:01 . drwx--. 4 root root 24 May 8 14:27 .. -rw---. 1

Re: [Gluster-users] split brain? but where?

I tried this already. 8><--- [root@glusterp2 fb]# find /bricks/brick1/gv0 -samefile /bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693 /bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693 [root@glusterp2 fb]# 8><--- gluster 4 Centos 7.4 8><--- df -h [roo

Re: [Gluster-users] split brain? but where?

Hi, Which version of gluster you are using? You can find which file is that using the following command find -samefile // Please provide the getfatr output of the file which is in split brain. The steps to recover from split-brain can be found here, http://gluster.readthedocs.io/en/latest/Troub

Re: [Gluster-users] split brain? but where?

How do I find what "eafb8799-4e7a-4264-9213-26997c5a4693" is? https://docs.gluster.org/en/v3/Troubleshooting/gfid-to-path/ On May 21, 2018 3:22:01 PM PDT, Thing wrote: >Hi, > >I seem to have a split brain issue, but I cannot figure out where this >is >and what it is, can someone help me pls,

[Gluster-users] split brain? but where?

Hi, I seem to have a split brain issue, but I cannot figure out where this is and what it is, can someone help me pls, I cant find what to fix here. == root@salt-001:~# salt gluster* cmd.run 'df -h' glusterp2.graywitch.co.nz: Filesystem S

[Gluster-users] Split brain

Hi, I am having a problem with a split-brain issue that does not seem to match up with documentation on how to solve it. gluster volume heal VMData2 info gives: Brick found2.ssd.org:/data/brick6/data /08aa5fc4-c9ba-4fcf-af57-72450b875d1a/images/d1164a9b-ba63-46c4-a9ec-76ea4a7a2c45/82a7027b

Re: [Gluster-users] Split brain directory

Hey, >From the getfattr output you have provided, the directory is clearly not in split brain. If all the bricks are being blamed by others then it is called split brain. In your case only client-13 that is Brick-14 in the volume info output had a pending entry heal on the directory. That is the l

[Gluster-users] Split brain directory

Hello, I'm trying to fix an issue with a Directory Split on a gluster 3.10.3. The effect consist of a specific file in this splitted directory to randomly be unavailable on some clients. I have gathered all the informations on this gist: https://gist.githubusercontent.com/lucagervasi/534e0024d34993

Re: [Gluster-users] split-brain

On 03/20/2017 06:31 PM, Bernhard Dübi wrote: Hi Ravi, thank you very much for looking into this The gluster volumes are used by CommVault Simpana to store backup data. Nothing/Nobody should access the underlying infrastructure. while looking at the xattrs of the files, I noticed that the only

Re: [Gluster-users] Split brain issue with non-existent file

On 07/15/2016 08:23 PM, Rob Janney wrote: Currently we are on 3.6: glusterfs 3.6.9 built on Mar 2 2016 18:21:17 That version does not have the real-time gfapi based version of the `info split-brain` command and it just prints prior event history from the selfheal daemon's memory. You can re

Re: [Gluster-users] Split brain issue with non-existent file

Currently we are on 3.6: glusterfs 3.6.9 built on Mar 2 2016 18:21:17 Yes, the file gfid file still was present at /.glusterfs/ab/da/abdab36b-b90b-4201-98fe-7a36059da81d On Thu, Jul 14, 2016 at 7:51 PM, Ravishankar N wrote: > > > On 07/15/2016 02:24 AM, Rob Janney wrote: > > I have a 2 node

Re: [Gluster-users] Split brain issue with non-existent file

On 07/15/2016 02:24 AM, Rob Janney wrote: I have a 2 node cluster that reports a single file as split brained .. but the file itself doesn't exist on either node. What is the version of gluster you are running? I did not find this scenario in the split brain docs, but based on other scenario

[Gluster-users] Split brain issue with non-existent file

I have a 2 node cluster that reports a single file as split brained .. but the file itself doesn't exist on either node. I did not find this scenario in the split brain docs, but based on other scenarios I deleted the gfid file and then ran a full heal, however; it still complains about split brai

Re: [Gluster-users] split-brain recovery automation, any plans?

13.07.2016 07:44, Pranith Kumar Karampuri пишет: On Tue, Jul 12, 2016 at 9:27 PM, Dmitry Melekhov > wrote: 12.07.2016 17:38, Pranith Kumar Karampuri пишет: Did you wait for heals to complete before upgrading second node? no... So basically if you hav

Re: [Gluster-users] split-brain recovery automation, any plans?

13.07.2016 00:20, Darrell Budic пишет: FYI, it’s my experience that “yum upgrade” will stop the running glistered (and possibly the running glusterfsds) This is main point- looks like glusterfsds are not stopped by upgrade, not sure, though... Thank you!

Re: [Gluster-users] split-brain recovery automation, any plans?

On Tue, Jul 12, 2016 at 9:27 PM, Dmitry Melekhov wrote: > > > 12.07.2016 17:38, Pranith Kumar Karampuri пишет: > > Did you wait for heals to complete before upgrading second node? > > > no... > So basically if you have operations in progress on the mount, you should wait for heals to complete b

Re: [Gluster-users] split-brain recovery automation, any plans?

FYI, it’s my experience that “yum upgrade” will stop the running glistered (and possibly the running glusterfsds) during it’s installation of new gluster components. I’ve also noticed it starts them back up again during the process. Ie, yesterday I upgraded a system to 3.7.13: systemctl stop gl

Re: [Gluster-users] split-brain recovery automation, any plans?

12.07.2016 17:38, Pranith Kumar Karampuri пишет: Did you wait for heals to complete before upgrading second node? no... On Tue, Jul 12, 2016 at 3:08 PM, Dmitry Melekhov > wrote: 12.07.2016 13:31, Pranith Kumar Karampuri пишет: On Mon, Jul 11, 2016 at 2

Re: [Gluster-users] split-brain recovery automation, any plans?

Did you wait for heals to complete before upgrading second node? On Tue, Jul 12, 2016 at 3:08 PM, Dmitry Melekhov wrote: > 12.07.2016 13:31, Pranith Kumar Karampuri пишет: > > > > On Mon, Jul 11, 2016 at 2:26 PM, Dmitry Melekhov < > d...@belkam.com> wrote: > >> 11.07.2016 12:47, Gandalf Corvot

Re: [Gluster-users] split-brain recovery automation, any plans?

12.07.2016 13:31, Pranith Kumar Karampuri пишет: On Mon, Jul 11, 2016 at 2:26 PM, Dmitry Melekhov > wrote: 11.07.2016 12:47, Gandalf Corvotempesta пишет: 2016-07-11 9:54 GMT+02:00 Dmitry Melekhov mailto:d...@belkam.com>>: We just got split-bra

Re: [Gluster-users] split-brain recovery automation, any plans?

On Mon, Jul 11, 2016 at 2:26 PM, Dmitry Melekhov wrote: > 11.07.2016 12:47, Gandalf Corvotempesta пишет: > >> 2016-07-11 9:54 GMT+02:00 Dmitry Melekhov : >> >>> We just got split-brain during update to 3.7.13 ;-) >>> >> This is an interesting point. >> Could you please tell me which replica count

Re: [Gluster-users] split-brain recovery automation, any plans?

11.07.2016 12:47, Gandalf Corvotempesta пишет: 2016-07-11 9:54 GMT+02:00 Dmitry Melekhov : We just got split-brain during update to 3.7.13 ;-) This is an interesting point. Could you please tell me which replica count did you set ? 3 With replica "3" split brain should not occurs, right ?

Re: [Gluster-users] split-brain recovery automation, any plans?

2016-07-11 9:54 GMT+02:00 Dmitry Melekhov : > We just got split-brain during update to 3.7.13 ;-) This is an interesting point. Could you please tell me which replica count did you set ? With replica "3" split brain should not occurs, right ? I'm planning a new cluster and I would like to be prot

Re: [Gluster-users] split-brain recovery automation, any plans?

On 07/11/2016 01:54 PM, Dmitry Melekhov wrote: 11.07.2016 12:18, Ravishankar N пишет: On 07/11/2016 01:24 PM, Dmitry Melekhov wrote: Hello! We just got split-brain during update to 3.7.13 ;-) This was our first time we hit this, and we found that it is not easy to follow and change xattrs m

Re: [Gluster-users] split-brain recovery automation, any plans?

11.07.2016 12:18, Ravishankar N пишет: On 07/11/2016 01:24 PM, Dmitry Melekhov wrote: Hello! We just got split-brain during update to 3.7.13 ;-) This was our first time we hit this, and we found that it is not easy to follow and change xattrs metadata. So, could you tell me, are there any p

Re: [Gluster-users] split-brain recovery automation, any plans?

On 07/11/2016 01:24 PM, Dmitry Melekhov wrote: Hello! We just got split-brain during update to 3.7.13 ;-) This was our first time we hit this, and we found that it is not easy to follow and change xattrs metadata. So, could you tell me, are there any plans to simplify this? Let's say, add to

[Gluster-users] split-brain recovery automation, any plans?

Hello! We just got split-brain during update to 3.7.13 ;-) This was our first time we hit this, and we found that it is not easy to follow and change xattrs metadata. So, could you tell me, are there any plans to simplify this? Let's say, add tool to choose preferred copy (brick) to heal othe

Re: [Gluster-users] Split brain?

On 02/10/2016 02:03 PM, Pranith Kumar Karampuri wrote: You need to follow the steps to understand and resolve split-brains: https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-bra

Re: [Gluster-users] Split brain?

You need to follow the steps to understand and resolve split-brains: https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md Feel free to let us know if you have any

Re: [Gluster-users] Split brain?

On 02/08/2016 09:05 PM, Rodolfo Gonzalez wrote: Hello, I think that I have a split-brain issue in a replicated-striped gluster cluster with 4 bricks: brick1+brick2 - brick3+brick4 In both brick3 and brick4 I'm getting these kind of messages: [2016-02-08 15:01:49.720343] E [afr-self-heal-ent

[Gluster-users] Split brain?

Hello, I think that I have a split-brain issue in a replicated-striped gluster cluster with 4 bricks: brick1+brick2 - brick3+brick4 In both brick3 and brick4 I'm getting these kind of messages: [2016-02-08 15:01:49.720343] E [afr-self-heal-entry.c:246:afr_selfheal_detect_gfid_and_type_mismatch] 0

Re: [Gluster-users] Split-brain after uploading file

any thoughts? Miloš > 30. 11. 2015 v 23:45, Miloš Kozák : > > I am using Gluster for a few years without any significant issue (after I > tweaked configuration for v3.5). My configuration is as follows: > > network.remote-dio: enable > cluster.eager-lock: enable > performance.stat-prefetch:

Re: [Gluster-users] split-brain files not shown in "gluster volume heal info split-brain" command

On 08/13/2015 07:42 PM, Andreas Hollaus wrote: Isn't the trusted.afr.dirty attribute missing from Brick 2? Shouldn't it be increased and decreased, but never removed? If one brick of a replica 2 setup is down, and files are written to, the dirty xattr is never set on the brick that is up. I

Re: [Gluster-users] split-brain files not shown in "gluster volume heal info split-brain" command

On 08/13/2015 07:30 PM, Lakshmi Anusha wrote: Hello, We managed to collect below command outputs: Brick1> getfattr -d -m . -e hex /opt/lvmdir/c2/brick/logfiles/security/EVENT_LOG.xml getfattr: Removing leading '/' from absolute path names # file: opt/lvmdir/c2/brick/logfiles/security/EVENT_

Re: [Gluster-users] split-brain files not shown in "gluster volume heal info split-brain" command

Hi, Isn't the trusted.afr.dirty attribute missing from Brick 2? Shouldn't it be increased and decreased, but never removed? Could that be the reason why GlusterFS is confused? What could be the reason for gfid mismatches? Regards Andreas > Brick1> getfattr -d -m . -e hex config.ior > # file: c

Re: [Gluster-users] split-brain files not shown in "gluster volume heal info split-brain" command

On 08/11/2015 03:28 PM, Lakshmi Anusha wrote: From the extended attributes, an entry split-brain seems to be appeared. Since gfid is different for the original file and its replica. Can you please let us know why the split-brain files are not shown in "gluster volume heal info split-brain" co

[Gluster-users] split-brain files not shown in "gluster volume heal info split-brain" command

Hello, We are using glusterfs version 3.7.2. There are few i/o errors reported during our testing and i/o errors are due to split brain files. We tried to find the split brain files with "gluster volume heal info split-brain". But strangely the command shows that the number of split brain files c

Re: [Gluster-users] Split brain after rebooting half of a two-node cluster

well, although I might be making that one up. Glad it worked out for you Peter. Cheers, Peter *From:*Ravishankar N [mailto:ravishan...@redhat.com] *Sent:* Wednesday, 5 August 2015 12:03 PM *To:* Peter Becker; Gluster-users@gluster.org *Subject:* Re: [Gluster-users] Split brain after rebooting

Re: [Gluster-users] Split brain after rebooting half of a two-node cluster

. Cheers, Peter From: Ravishankar N [mailto:ravishan...@redhat.com] Sent: Wednesday, 5 August 2015 12:03 PM To: Peter Becker; Gluster-users@gluster.org Subject: Re: [Gluster-users] Split brain after rebooting half of a two-node cluster On 08/05/2015 07:01 AM, Peter Becker wrote: Hi Ravi, Not

Re: [Gluster-users] Split brain after rebooting half of a two-node cluster

Thanks, Ravi - I was not aware of that. I'll update and try again. Peter From: Ravishankar N [mailto:ravishan...@redhat.com] Sent: Wednesday, 5 August 2015 12:03 PM To: Peter Becker; Gluster-users@gluster.org Subject: Re: [Gluster-users] Split brain after rebooting half of a two

Re: [Gluster-users] Split brain after rebooting half of a two-node cluster

that. Given that it’s a bit involved: are there other venues we could try first? Cheers, Peter *From:*Ravishankar N [mailto:ravishan...@redhat.com] *Sent:* Wednesday, 5 August 2015 11:12 AM *To:* Peter Becker; Gluster-users@gluster.org *Subject:* Re: [Gluster-users] Split brain after

Re: [Gluster-users] Split brain after rebooting half of a two-node cluster

org Subject: Re: [Gluster-users] Split brain after rebooting half of a two-node cluster On 08/05/2015 04:49 AM, Peter Becker wrote: qmaster@srvamqpy01:~$ gluster --version glusterfs 3.2.5 built on Jan 31 2012 07:39:59 FWIW, this is a rather old release. Can you see if the issue is recurring with

Re: [Gluster-users] Split brain after rebooting half of a two-node cluster

On 08/05/2015 04:49 AM, Peter Becker wrote: qmaster@srvamqpy01:~$ gluster --version glusterfs 3.2.5 built on Jan 31 2012 07:39:59 FWIW, this is a rather old release. Can you see if the issue is recurring with glusterfs 3.7? -Ravi ___ Gluster-use

[Gluster-users] Split brain after rebooting half of a two-node cluster

Hello, We are trying to run a pair of ActiveMQ nodes on top of glusterfs, using the approach described in http://activemq.apache.org/shared-file-system-master-slave.html This seemed to work at first, but if I start rebooting machines while under load I seem to quickly get into this problem:

[Gluster-users] Split brain or what

Hi, I have a wired situation. Is this as it should be? or what is going on # gluster --version glusterfs 3.6.2 built on Jan 22 2015 12:58:11 There are 8 files reported in split-brain (1024 times), but they seem to be identical. # gluster volume heal glu_rhevtst_dr2_data_01 info split-brain| awk

Re: [Gluster-users] split-brain.. okay, what next?

On 07/16/2015 01:28 AM, Игорь Бирюлин wrote: I have studied information on page: https://github.com/gluster/glusterfs/blob/master/doc/features/heal-info-and-split-brain-resolution.md and cannot solve split-brain by this instruction. I have tested it on gluster 3.6 and it doesn't work, only on

Re: [Gluster-users] split-brain.. okay, what next?

I have studied information on page: https://github.com/gluster/glusterfs/blob/master/doc/features/heal-info-and-split-brain-resolution.md and cannot solve split-brain by this instruction. I have tested it on gluster 3.6 and it doesn't work, only on gluster 3.7. I try to use on gluster 3.7.2. I ha

Re: [Gluster-users] split-brain.. okay, what next?

never mind. I do not have enough time to debug why basic commands of gluster do not work on production server. It was enough of tonight's system freeze due to not documented XFS settings MUST have to run glusterfs with XFS. I'll keep to EXT4 better. Anyway XFS for bricks did not solve my previous p

Re: [Gluster-users] split-brain.. okay, what next?

Thanx for pointing out... but doesn't seem to work... or i am too sleepy due to problems with glusterfs and debian8 in other topic which i'm fighting for month.. root@stor1:~# gluster volume heal HA-2TB-TT-Proxmox-cluster split-brain source-brick stor1:HA-2TB-TT-Proxmox-cluster/2TB /images/124/vm-

Re: [Gluster-users] split-brain.. okay, what next?

On 07/14/2015 11:19 AM, Roman wrote: Hi, played with glusterfs tonight and tried to use recommended XFS for gluster.. first try was pretty bad and all of my VM-s hanged (XFS wants allocsize=64k to create qcow2 files, which i didn't know about and tried to create VM on XFS without this config

[Gluster-users] split-brain.. okay, what next?

Hi, played with glusterfs tonight and tried to use recommended XFS for gluster.. first try was pretty bad and all of my VM-s hanged (XFS wants allocsize=64k to create qcow2 files, which i didn't know about and tried to create VM on XFS without this config line in fstab, which lead to a lot of IO-s

[Gluster-users] split brain

Hello , split brain happened a few hours before, would you define which copy is the newest ?? # gluster volume heal 1KVM12_P3 info Brick 1kvm1:/STORAGES/g1r5p3/GFS/ /7e5ca629-5e97-4220-a6b2-b93242e8f314/dom_md/ids - Is in split-brain Number of entries: 1 Brick 1kvm2:/STORAGES/g1r5p3/GFS/ /7e

Re: [Gluster-users] split brain on / just after installation

On 06/02/2015 09:10 AM, Carl L Hoffman wrote: Hello - I was wondering if someone could please help me. I've just setup Gluster 3.6 on two Ubuntu 14.04 hosts. Gluster is setup to replicate two volumes (prod-volume, dev-volume) between the two hosts. Replication is working fine. The gluster

[Gluster-users] split brain on / just after installation

Hello - I was wondering if someone could please help me. I've just setup Gluster 3.6 on two Ubuntu 14.04 hosts. Gluster is setup to replicate two volumes (prod-volume, dev-volume) between the two hosts. Replication is working fine. The glustershd.log shows: [2015-06-02 03:28:04.495162] E [a

[Gluster-users] Split-brain issue

Hi there, I have a split-brain situation on one of my volumes. The part that I'm not clear on is that it would appear to be on the root of the volume. [2015-04-13 22:07:35.729798] E [afr-self-heal-common.c:2262:afr_self_heal_completion_cbk] 0-www-conf-replicate-0: background meta-data self-heal

Re: [Gluster-users] Split-brain info gluster 3.6

This is fixed in http://review.gluster.org/9459 and should be available in 3.7. As a workaround, you can restart the selfheal daemon process (gluster v start force). This should clear its history. Thanks, Ravi On 02/20/2015 01:43 PM, Félix de Lelelis wrote: Hi, I generated a split-brain con

[Gluster-users] Split-brain info gluster 3.6

Hi, I generated a split-brain condition, and I solved it but with "gluster volume heal vol_name info split-brain" yet I can see past entries solved: Number of entries: 3 atpath on brick --- 2015-02-19 17:13:08 /split 2015-02-19 17:14:09 /split

Re: [Gluster-users] split-brain with quorum enabled

Hi, Sorry for the delay, it took a long time to reproduce. But currently we have the same issue again. This was happened afrer reseting all nodes. Quorum is enabled. Logs and details below. gluster> volume heal vm_storage_volume info split-brain Gathering list of split brain entries on volume vm_

Re: [Gluster-users] split-brain with quorum enabled

On 12/25/2014 08:05 PM, Alexey wrote: Hi all, We are using glusterfs setup with a quorum turned on and the configuration as the follows: Nodes: 3 Type: Replicate Number of Bricks: 1 x 3 = 3 cluster.quorum-type: fixed cluster.quorum-count: 2 cluster.data-self-heal-algorithm: diff cluster.serv

[Gluster-users] split-brain with quorum enabled

Hi all, We are using glusterfs setup with a quorum turned on and the configuration as the follows: Nodes: 3 Type: Replicate Number of Bricks: 1 x 3 = 3 cluster.quorum-type: fixed cluster.quorum-count: 2 cluster.data-self-heal-algorithm: diff cluster.server-quorum-ratio: 51% glusterfs version: 3.5

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

Ok, no problem. The issue is very rare, even with our setup - we have seen it only once on one site even though we have been in production for several months now. For now, we can live with that IMO. And, thanks again. Anirban___ Gluster-users mailin

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

that was the only work-around. Pranith Thanks again, Anirban *From: * Pranith Kumar Karampuri ; *To: * Anirban Ghoshal ; ; *Subject: * Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

I see. Thanks a tonne for the thorough explanation! :) I can see that our setup would be vulnerable here because the logger on one server is not generally aware of the state of the replica on the other server. So, it is possible that the log files may have been renamed before heal had a chance t

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

--- *From: * Pranith Kumar Karampuri ; *To: * Anirban Ghoshal ; ; *Subject: * Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors *Sent: * Sun, Oct 19, 2014 5:42:24 AM On 10/18/2014 04:36 PM, Anirban Ghoshal wrote: Hi, Yes, they do, and considerably. I

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

It is possible, yes, because these are actually a kind of log files. I suppose, like other logging frameworks these files an remain open for a considerable period, and then get renamed to support log rotate semantics. That said, I might need to check with the team that actually manages the log

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

irban Ghoshal ; gluster-users@gluster.org ; *Subject: * Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors *Sent: * Sat, Oct 18, 2014 12:26:08 AM hi, Could you see if the size of the file mismatches? Pranith On 10/18/2014 04:20 AM, Anirban Ghoshal wrote

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

Hi, Yes, they do, and considerably. I'd forgotten to mention that on my last email. Their mtimes, however, as far as i could tell on separate servers, seemed to coincide. Thanks, Anirban___ Gluster-users mailing list Gluster-users@gluster.org http://

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

hi, Could you see if the size of the file mismatches? Pranith On 10/18/2014 04:20 AM, Anirban Ghoshal wrote: Hi everyone, I have this really confusing split-brain here that's bothering me. I am running glusterfs 3.4.2 over linux 2.6.34. I have a replica 2 volume 'testvol' that is It se

[Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

Hi everyone, I have this really confusing split-brain here that's bothering me. I am running glusterfs 3.4.2 over linux 2.6.34. I have a replica 2 volume 'testvol' that is It seems I cannot read/stat/edit the file in question, and `gluster volume heal testvol info split-brain` shows nothing. He

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

On 09/19/2014 09:58 PM, Ramesh Natarajan wrote: I was able to run another set of tests this week and I was able to reproduce the issue again. Going by the extended attributes, I think i ran into the same issue I saw earlier.. Do you think i need to open up a bug report? hi Ramesh, I al

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

I was able to run another set of tests this week and I was able to reproduce the issue again. Going by the extended attributes, I think i ran into the same issue I saw earlier.. Do you think i need to open up a bug report? Brick 1: trusted.afr.PL2-client-0=0x trusted.afr

Re: [Gluster-users] Split brain that is not split brain

On 11/09/2014, at 9:44 AM, Pranith Kumar Karampuri wrote: > On 09/11/2014 01:13 PM, Ilya Ivanov wrote: >> Makes some sense. Yes, I meant "make a backup and delete", rather than just >> delete. >> >> If I may suggest, putting that debug link somewhere more visible would be be >> good, too. I woul

Re: [Gluster-users] Split brain that is not split brain

On 09/11/2014 01:13 PM, Ilya Ivanov wrote: Makes some sense. Yes, I meant "make a backup and delete", rather than just delete. If I may suggest, putting that debug link somewhere more visible would be be good, too. I wouldn't find without your help. Justin, where shall we put the doc? Prani

Re: [Gluster-users] Split brain that is not split brain

Makes some sense. Yes, I meant "make a backup and delete", rather than just delete. If I may suggest, putting that debug link somewhere more visible would be be good, too. I wouldn't find without your help. Thank you for the assistance. On Thu, Sep 11, 2014 at 9:14 AM, Pranith Kumar Karampuri

Re: [Gluster-users] Split brain that is not split brain

On 09/11/2014 11:37 AM, Ilya Ivanov wrote: I don't understand why there's such a complicated process to recover when I can just look at both files, decide which one I need and delete another one. If the file needs to be deleted the whole file needs to be copied which is fine for small files bu

Re: [Gluster-users] Split brain that is not split brain

I don't understand why there's such a complicated process to recover when I can just look at both files, decide which one I need and delete another one. On Thu, Sep 11, 2014 at 7:56 AM, Pranith Kumar Karampuri < pkara...@redhat.com> wrote: > > On 09/11/2014 09:29 AM, Ilya Ivanov wrote: > > Righ

Re: [Gluster-users] Split brain that is not split brain

On 09/11/2014 09:29 AM, Ilya Ivanov wrote: Right... I deleted it and now all appears to be fine. Still, could you please elaborate on "gfid split-brain"? Could you go through https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md Let us know if you would like something t

Re: [Gluster-users] Split brain that is not split brain

Right... I deleted it and now all appears to be fine. Still, could you please elaborate on "gfid split-brain"? On Thu, Sep 11, 2014 at 5:32 AM, Pranith Kumar Karampuri < pkara...@redhat.com> wrote: > > On 09/11/2014 12:16 AM, Ilya Ivanov wrote: > > Any insight? > > Was the other file's gfid d3d

Re: [Gluster-users] Split brain that is not split brain

On 09/11/2014 12:16 AM, Ilya Ivanov wrote: Any insight? Was the other file's gfid d3def9e1-c6d0-4b7d-a322-b5019305182e? Could you check if this file exists in /.glusterfs/d3/de/ When a file is deleted this file also needs to be deleted if there are no more hardlinks to the file Pranith On

Re: [Gluster-users] Split brain that is not split brain

Any insight? On Tue, Sep 9, 2014 at 8:35 AM, Ilya Ivanov wrote: > What's a gfid split-brain and how is it different from "normal" > split-brain? > > I accessed the file with stat, but "heal info" still shows "Number of > entries: 1" > > [root@gluster1 gluster]# getfattr -d -m. -e hex gv01/123 >

Re: [Gluster-users] Split brain that is not split brain

Ahh, thank you, now I get it. I deleted it on one node and it replicated to another one. Now I get the following output: [root@gluster1 var]# gluster volume heal gv01 info Brick gluster1:/home/gluster/gv01/ Number of entries: 1 Brick gluster2:/home/gluster/gv01/ Number of entries: 0 Is it norma

Re: [Gluster-users] Split brain that is not split brain

What's a gfid split-brain and how is it different from "normal" split-brain? I accessed the file with stat, but "heal info" still shows "Number of entries: 1" [root@gluster1 gluster]# getfattr -d -m. -e hex gv01/123 # getfattr -d -m. -e hex gv01/123 # file: gv01/123 trusted.afr.gv01-client-0=0x00

Re: [Gluster-users] Split brain that is not split brain

On 09/09/2014 11:35 AM, Ilya Ivanov wrote: Ahh, thank you, now I get it. I deleted it on one node and it replicated to another one. Now I get the following output: [root@gluster1 var]# gluster volume heal gv01 info Brick gluster1:/home/gluster/gv01/ Number of entries: 1 Brick gluster2:/home/

Re: [Gluster-users] Split brain that is not split brain

On 09/09/2014 01:54 AM, Ilya Ivanov wrote: Hello. I've Gluster 3.5.2 on Centos 6. A primitive replicated volume, as describe here . I tried to simulate split-brain by te

[Gluster-users] Split brain that is not split brain

Hello. I've Gluster 3.5.2 on Centos 6. A primitive replicated volume, as describe here . I tried to simulate split-brain by temporarily disconnecting the nodes and creating a

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

On 09/06/2014 04:53 AM, Jeff Darcy wrote: I have a replicate glusterfs setup on 3 Bricks ( replicate = 3 ). I have client and server quorum turned on. I rebooted one of the 3 bricks. When it came back up, the client started throwing error messages that one of the files went into split brain. Th

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

On 09/06/2014 07:06 PM, Ramesh Natarajan wrote: Client is also running the same version. It is running RHEL 6.5. Initially I was under the impression that the issue could be https://bugzilla.redhat.com/show_bug.cgi?id=1066996. But it doesn't seem like. Only one brick is taken out. I guess ther

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

Client is also running the same version. It is running RHEL 6.5. The mount command used is >>> mount -t glusterfs -o >>> defaults,enable-ino32,direct-io-mode=disable,log-level=WARNING,log-file=/var/log/gluster.log,backupvolfile-server=172.31.38.189,backupvolfile-server=172.31.12.218,background-

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

On 09/06/2014 06:17 PM, Ramesh Natarajan wrote: 3.5.2 installed via rpm from official gluster repo, running on amazon ami. Including the client i.e. mount bits? Pranith Thanks Ramesh On Sep 6, 2014, at 7:29 AM, Pranith Kumar Karampuri mailto:pkara...@redhat.com>> wrote: What is the glus

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

3.5.2 installed via rpm from official gluster repo, running on amazon ami. Thanks Ramesh > On Sep 6, 2014, at 7:29 AM, Pranith Kumar Karampuri > wrote: > > What is the glusterfs version where you ran into this issue? > > Pranith >> On 09/05/2014 11:52 PM, Ramesh Natarajan wrote: >> I have a r

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

What is the glusterfs version where you ran into this issue? Pranith On 09/05/2014 11:52 PM, Ramesh Natarajan wrote: I have a replicate glusterfs setup on 3 Bricks ( replicate = 3 ). I have client and server quorum turned on. I rebooted one of the 3 bricks. When it came back up, the client sta

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

> Thanks Jeff for the detailed explanation. You mentioned delayed changelog may > have prevented this issue. Can you please tell me how to enable it? It's on by default (cluster.post-op-delay-secs in the CLI) with a value of 1. However, it's inoperative in some cases, such as when multiple fds ar

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

Thanks Jeff for the detailed explanation. You mentioned delayed changelog may have prevented this issue. Can you please tell me how to enable it? Thanks Ramesh On Sep 5, 2014, at 6:23 PM, Jeff Darcy wrote: >> I have a replicate glusterfs setup on 3 Bricks ( replicate = 3 ). I have >> client

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

> I have a replicate glusterfs setup on 3 Bricks ( replicate = 3 ). I have > client and server quorum turned on. I rebooted one of the 3 bricks. When it > came back up, the client started throwing error messages that one of the > files went into split brain. This is a good example of how split bra

  1   2   >