Re: [Gluster-users] self heal errors on 3.1.1 clients
On 01/26/2011 07:25 PM, David Lloyd wrote: Well, I did this and it seems to have worked. I was just guessing really, didn't have any documentation or advice from anyone in the know. I just reset the attributes on the root directory for each brick that was not all zeroes. I found it easier to dump the attributes without the '-e hex' g4:~ # getfattr -d -m trusted.afr /mnt/glus1 /mnt/glus2 getfattr: Removing leading '/' from absolute path names # file: mnt/glus1 trusted.afr.glustervol1-client-2=0sAAEA trusted.afr.glustervol1-client-3=0s Then setfattr -n trusted.afr.glustervol1-client-2 -v 0s /mnt/glus1 I did that on all the bricks that didn't have all A's next time i stat-ed the root of the filesystem on the client the self heal worked ok. I'm not comfortable advising you to do this as I'm really feeling my way here, but it looks as though it worked for me. This seems really dangerous to me. On a brick xxx, the trusted.afr.yyy attribute consists of three unsigned 32-bit counters, indicating how many uncommitted operations (data, metadata, and namespace respectively) might exist at yyy. If xxx shows uncommitted operations at yyy but not vice versa, then we know that xxx is more up to date and it should be the source for self-heal. If two bricks show uncommitted operations at each other, then we're in the infamous split brain scenario. Some client was unable to clear the counter at xxx while another was unable to clear it at yyy, or both xxx and yyy went down after the operation was complete but before they could clear the counters for each other. In this case, it looks like a metadata operation (permission change) was in this state. If the permissions are in fact the same both places then it doesn't matter which way self-heal happens, or whether it happens at all. In fact, it seems to me that AFR should be able to detect this particular condition and not flag it as an error. In any case, I think you're probably fine in this case but in general it's a very bad idea to clear these flags manually because it can cause updates to be lost (if self-heal goes the wrong way) or files to remain in an inconsistent state (if no self-heal occurs). The real thing I'd wonder about is why both servers are so frequently becoming unavailable at the same instant (switch problem?) and why permission changes on the root are apparently so frequent that this ofen results in a split-brain. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] self heal errors on 3.1.1 clients
Yes, it seemed really dangerous to me too. But with the lack of documentation, and lack of response from gluster (and the data is still on the old system too), I thought I'd give it a shot. Thanks for the explanation. The split-brain problem seems to come up fairly regularly, but I've not found any clear explanation of what to do in this situation. I'm starting to worry about what appears to be a rationing of information from gluster.com to the the community at large. We're not in a position to purchase support, and I'm a sysadmin, not a developer. I hope to make a contribution in terms of testing and feedback and bug reports, but I'm seeing a lot of threads that seem to go nowhere, and it's getting a bit frustrating. David This seems really dangerous to me. On a brick xxx, the trusted.afr.yyy attribute consists of three unsigned 32-bit counters, indicating how many uncommitted operations (data, metadata, and namespace respectively) might exist at yyy. If xxx shows uncommitted operations at yyy but not vice versa, then we know that xxx is more up to date and it should be the source for self-heal. If two bricks show uncommitted operations at each other, then we're in the infamous split brain scenario. Some client was unable to clear the counter at xxx while another was unable to clear it at yyy, or both xxx and yyy went down after the operation was complete but before they could clear the counters for each other. In this case, it looks like a metadata operation (permission change) was in this state. If the permissions are in fact the same both places then it doesn't matter which way self-heal happens, or whether it happens at all. In fact, it seems to me that AFR should be able to detect this particular condition and not flag it as an error. In any case, I think you're probably fine in this case but in general it's a very bad idea to clear these flags manually because it can cause updates to be lost (if self-heal goes the wrong way) or files to remain in an inconsistent state (if no self-heal occurs). The real thing I'd wonder about is why both servers are so frequently becoming unavailable at the same instant (switch problem?) and why permission changes on the root are apparently so frequent that this ofen results in a split-brain. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- David Lloyd V Consultants www.v-consultants.co.uk tel: +44 7983 816501 skype: davidlloyd1243 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] self heal errors on 3.1.1 clients
David, The problem what you are facing is something we are already investigating. We still haven't root-caused it yet, but from what we have seen this happens only on / and only for metadata changelog. This shows up as just annoying logs but it should not affect your functionality. Avati On Thu, Jan 27, 2011 at 2:03 PM, David Lloyd david.ll...@v-consultants.co.uk wrote: Yes, it seemed really dangerous to me too. But with the lack of documentation, and lack of response from gluster (and the data is still on the old system too), I thought I'd give it a shot. Thanks for the explanation. The split-brain problem seems to come up fairly regularly, but I've not found any clear explanation of what to do in this situation. I'm starting to worry about what appears to be a rationing of information from gluster.com to the the community at large. We're not in a position to purchase support, and I'm a sysadmin, not a developer. I hope to make a contribution in terms of testing and feedback and bug reports, but I'm seeing a lot of threads that seem to go nowhere, and it's getting a bit frustrating. David This seems really dangerous to me. On a brick xxx, the trusted.afr.yyy attribute consists of three unsigned 32-bit counters, indicating how many uncommitted operations (data, metadata, and namespace respectively) might exist at yyy. If xxx shows uncommitted operations at yyy but not vice versa, then we know that xxx is more up to date and it should be the source for self-heal. If two bricks show uncommitted operations at each other, then we're in the infamous split brain scenario. Some client was unable to clear the counter at xxx while another was unable to clear it at yyy, or both xxx and yyy went down after the operation was complete but before they could clear the counters for each other. In this case, it looks like a metadata operation (permission change) was in this state. If the permissions are in fact the same both places then it doesn't matter which way self-heal happens, or whether it happens at all. In fact, it seems to me that AFR should be able to detect this particular condition and not flag it as an error. In any case, I think you're probably fine in this case but in general it's a very bad idea to clear these flags manually because it can cause updates to be lost (if self-heal goes the wrong way) or files to remain in an inconsistent state (if no self-heal occurs). The real thing I'd wonder about is why both servers are so frequently becoming unavailable at the same instant (switch problem?) and why permission changes on the root are apparently so frequent that this ofen results in a split-brain. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- David Lloyd V Consultants www.v-consultants.co.uk tel: +44 7983 816501 skype: davidlloyd1243 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] self heal errors on 3.1.1 clients
These errors are appearing in the file /var/log/glusterfs/mountpoint.log [2011-01-26 11:02:10.342349] I [afr-common.c:672:afr_lookup_done] pfs-ro1-replicate-5: split brain detected during lookup of /. [2011-01-26 11:02:10.342366] I [afr-common.c:716:afr_lookup_done] pfs-ro1-replicate-5: background meta-data data self-heal triggered. path: / [2011-01-26 11:02:10.342502] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] pfs-ro1-replicate-2: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes Apparently the issue is the root of the storage pool, which in my case on the backend storage servers is this path: /export/read-only - permissions are:drwxr-xr-x 12 root root 4096 Dec 28 12:09 /export/read-only/ Installation is GlusterFS 3.1.1 on servers and clients, servers running CentOS 5.5, clients running CentOS 5.2. The volume info header is below: Volume Name: pfs-ro1 Type: Distributed-Replicate Status: Started Number of Bricks: 10 x 2 = 20 Transport-type: tcp Any ideas? I don't see a permission issue on the directory or it's subs themselves. James Burnash, Unix Engineering -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Jeff Darcy Sent: Tuesday, January 25, 2011 10:45 AM To: gluster-users@gluster.org Subject: Re: [Gluster-users] Guideline needed for a xlator project On 01/25/2011 09:09 AM, Jeff Darcy wrote: I've attached a copy of a document I wrote a while ago when I started writing translators myself. Grrr, looks like the mailing-list software stripped it. It's also at http://cloudfs.org/docs/ if anybody's interested. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] self heal errors on 3.1.1 clients
We started getting the same problem at almost exactly the same time. get one of these messages every time I access the root of the mounted volume (and nowhere else, I think). This is also 3.1.1 I'm just starting to look in to it, I'll let you know if I get anywhere. David On 26 January 2011 16:38, Burnash, James jburn...@knight.com wrote: These errors are appearing in the file /var/log/glusterfs/mountpoint.log [2011-01-26 11:02:10.342349] I [afr-common.c:672:afr_lookup_done] pfs-ro1-replicate-5: split brain detected during lookup of /. [2011-01-26 11:02:10.342366] I [afr-common.c:716:afr_lookup_done] pfs-ro1-replicate-5: background meta-data data self-heal triggered. path: / [2011-01-26 11:02:10.342502] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] pfs-ro1-replicate-2: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes Apparently the issue is the root of the storage pool, which in my case on the backend storage servers is this path: /export/read-only - permissions are:drwxr-xr-x 12 root root 4096 Dec 28 12:09 /export/read-only/ Installation is GlusterFS 3.1.1 on servers and clients, servers running CentOS 5.5, clients running CentOS 5.2. The volume info header is below: Volume Name: pfs-ro1 Type: Distributed-Replicate Status: Started Number of Bricks: 10 x 2 = 20 Transport-type: tcp Any ideas? I don't see a permission issue on the directory or it's subs themselves. James Burnash, Unix Engineering -Original Message- From: gluster-users-boun...@gluster.org [mailto: gluster-users-boun...@gluster.org] On Behalf Of Jeff Darcy Sent: Tuesday, January 25, 2011 10:45 AM To: gluster-users@gluster.org Subject: Re: [Gluster-users] Guideline needed for a xlator project On 01/25/2011 09:09 AM, Jeff Darcy wrote: I've attached a copy of a document I wrote a while ago when I started writing translators myself. Grrr, looks like the mailing-list software stripped it. It's also at http://cloudfs.org/docs/ if anybody's interested. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- David Lloyd V Consultants www.v-consultants.co.uk tel: +44 7983 816501 skype: davidlloyd1243 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] self heal errors on 3.1.1 clients
I read on another thread about checking the getfattr output for each brick, but it tailed off before any explanation of what to do with this information We have 8 bricks in the volume. Config is: g1:~ # gluster volume info glustervol1 Volume Name: glustervol1 Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: g1:/mnt/glus1 Brick2: g2:/mnt/glus1 Brick3: g3:/mnt/glus1 Brick4: g4:/mnt/glus1 Brick5: g1:/mnt/glus2 Brick6: g2:/mnt/glus2 Brick7: g3:/mnt/glus2 Brick8: g4:/mnt/glus2 Options Reconfigured: performance.write-behind-window-size: 100mb performance.cache-size: 512mb performance.stat-prefetch: on and the getfattr outputs are: g1:~ # getfattr -d -e hex -m trusted.afr /mnt/glus1 getfattr: Removing leading '/' from absolute path names # file: mnt/glus1 trusted.afr.glustervol1-client-0=0x trusted.afr.glustervol1-client-1=0x g1:~ # getfattr -d -e hex -m trusted.afr /mnt/glus2 getfattr: Removing leading '/' from absolute path names # file: mnt/glus2 trusted.afr.glustervol1-client-4=0x trusted.afr.glustervol1-client-5=0x g2:~ # getfattr -d -e hex -m trusted.afr /mnt/glus1 getfattr: Removing leading '/' from absolute path names # file: mnt/glus1 trusted.afr.glustervol1-client-0=0x trusted.afr.glustervol1-client-1=0x g2:~ # getfattr -d -e hex -m trusted.afr /mnt/glus2 getfattr: Removing leading '/' from absolute path names # file: mnt/glus2 trusted.afr.glustervol1-client-4=0x trusted.afr.glustervol1-client-5=0x g3:~ # getfattr -d -e hex -m trusted.afr /mnt/glus1 getfattr: Removing leading '/' from absolute path names # file: mnt/glus1 trusted.afr.glustervol1-client-2=0x trusted.afr.glustervol1-client-3=0x0001 g3:~ # getfattr -d -e hex -m trusted.afr /mnt/glus2 getfattr: Removing leading '/' from absolute path names # file: mnt/glus2 trusted.afr.glustervol1-client-6=0x trusted.afr.glustervol1-client-7=0x g4:~ # getfattr -d -e hex -m trusted.afr /mnt/glus1 getfattr: Removing leading '/' from absolute path names # file: mnt/glus1 trusted.afr.glustervol1-client-2=0x0001 trusted.afr.glustervol1-client-3=0x g4:~ # getfattr -d -e hex -m trusted.afr /mnt/glus2 getfattr: Removing leading '/' from absolute path names # file: mnt/glus2 trusted.afr.glustervol1-client-6=0x trusted.afr.glustervol1-client-7=0x Hope someone can help. Things still seem to be working, but slowed down. Cheers David On 26 January 2011 17:07, David Lloyd david.ll...@v-consultants.co.ukwrote: We started getting the same problem at almost exactly the same time. get one of these messages every time I access the root of the mounted volume (and nowhere else, I think). This is also 3.1.1 I'm just starting to look in to it, I'll let you know if I get anywhere. David On 26 January 2011 16:38, Burnash, James jburn...@knight.com wrote: These errors are appearing in the file /var/log/glusterfs/mountpoint.log [2011-01-26 11:02:10.342349] I [afr-common.c:672:afr_lookup_done] pfs-ro1-replicate-5: split brain detected during lookup of /. [2011-01-26 11:02:10.342366] I [afr-common.c:716:afr_lookup_done] pfs-ro1-replicate-5: background meta-data data self-heal triggered. path: / [2011-01-26 11:02:10.342502] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] pfs-ro1-replicate-2: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes Apparently the issue is the root of the storage pool, which in my case on the backend storage servers is this path: /export/read-only - permissions are:drwxr-xr-x 12 root root 4096 Dec 28 12:09 /export/read-only/ Installation is GlusterFS 3.1.1 on servers and clients, servers running CentOS 5.5, clients running CentOS 5.2. The volume info header is below: Volume Name: pfs-ro1 Type: Distributed-Replicate Status: Started Number of Bricks: 10 x 2 = 20 Transport-type: tcp Any ideas? I don't see a permission issue on the directory or it's subs themselves. James Burnash, Unix Engineering ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
