Hi Anuradha, But in this case I need to do tail on each file which is time taking process and other end I can't pause my module until these file is getting healed.
Any how I need the output of the split-brain to resolve this problem. Regards, Abhishek On Wed, Mar 16, 2016 at 6:21 PM, ABHISHEK PALIWAL <abhishpali...@gmail.com> wrote: > Hi Anuradha, > > The issue is resolved but we have one more issue something similar to this > one in which the file is not getting sync after the steps followed, > mentioned in the link which you shared in the previous mail. > > And problem is that why split-brain command is not showing split-brain > entries. > > Regards, > Abhishek > > On Wed, Mar 16, 2016 at 6:06 PM, Anuradha Talur <ata...@redhat.com> wrote: > >> >> >> ----- Original Message ----- >> > From: "Anuradha Talur" <ata...@redhat.com> >> > To: "ABHISHEK PALIWAL" <abhishpali...@gmail.com> >> > Cc: gluster-us...@gluster.org, gluster-devel@gluster.org >> > Sent: Wednesday, March 16, 2016 5:32:26 PM >> > Subject: Re: [Gluster-users] gluster volume heal info split brain >> command not showing files in split-brain >> > >> > >> > >> > ----- Original Message ----- >> > > From: "ABHISHEK PALIWAL" <abhishpali...@gmail.com> >> > > To: "Anuradha Talur" <ata...@redhat.com> >> > > Cc: gluster-us...@gluster.org, gluster-devel@gluster.org >> > > Sent: Wednesday, March 16, 2016 4:39:26 PM >> > > Subject: Re: [Gluster-users] gluster volume heal info split brain >> command >> > > not showing files in split-brain >> > > >> > > Hi Anuradha, >> > > >> > > I am doing the same which is mentioned in the link you shared and It >> has >> > > been resolved the issue. >> > > >> > > But my question is if it is the split-brain scenario then why the >> command >> > > "gluster volume heal info split-brain" >> > > not showing these files in the output even not the parent directory >> is >> > > present in split-brain. >> > > >> > > Please find the requested logs >> >> Abhishek, >> >> Yes, ideally it should show. I will look into it. The only reason I can >> think of, is when parent directory did not have any pending markers to >> indicate split-brain; which is why I asked getfattr output for the parent >> directory too. But if the issue is resolved, there isn't much info we can >> get out of it. Thanks for sharing the logs. Will see what could have caused >> this. >> >> > Abhishek, >> > >> > Yes, ideally it should show. I will look into it. >> > I saw another case with this issue. The parent directory did not have >> any pe >> > > >> > > On Wed, Mar 16, 2016 at 4:20 PM, Anuradha Talur <ata...@redhat.com> >> wrote: >> > > >> > > > Hi Abhishek, >> > > > >> > > > The files that are reporting i/o error have gfid-mismatch. This >> situation >> > > > is called directory or entry split-brain. You can find steps to >> resolve >> > > > this kind of split brain here : >> > > > >> https://gluster.readthedocs.org/en/latest/Troubleshooting/split-brain/ . >> > > > >> > > > Ideally, the parent directories of these files have to be listed in >> heal >> > > > info split-brain output. Can you please get extended attributes of >> parent >> > > > directories of the files that show i/o error (Same getfattr command >> that >> > > > you previously used.) ? >> > > > >> > > > ----- Original Message ----- >> > > > > From: "ABHISHEK PALIWAL" <abhishpali...@gmail.com> >> > > > > To: "Anuradha Talur" <ata...@redhat.com> >> > > > > Cc: gluster-us...@gluster.org, gluster-devel@gluster.org >> > > > > Sent: Thursday, March 10, 2016 11:22:35 AM >> > > > > Subject: Re: [Gluster-users] gluster volume heal info split brain >> > > > command not showing files in split-brain >> > > > > >> > > > > Hi Anuradha, >> > > > > >> > > > > Please find the glusterfs and glusterd logs directory as an >> attachment. >> > > > > >> > > > > Regards, >> > > > > Abhishek >> > > > > >> > > > > >> > > > > >> > > > > On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL < >> > > > abhishpali...@gmail.com> >> > > > > wrote: >> > > > > >> > > > > > Hi Anuradha, >> > > > > > >> > > > > > Sorry for late reply. >> > > > > > >> > > > > > Please find the requested logs below: >> > > > > > >> > > > > > Remote: 10.32.0.48 >> > > > > > Local : 10.32.1.144 >> > > > > > >> > > > > > Local: >> > > > > > #gluster volume heal c_glusterfs info split-brain >> > > > > > Brick 10.32.1.144:/opt/lvmdir/c2/brick >> > > > > > Number of entries in split-brain: 0 >> > > > > > >> > > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick >> > > > > > Number of entries in split-brain: 0 >> > > > > > >> > > > > > Remote: >> > > > > > #gluster volume heal c_glusterfs info split-brain >> > > > > > Brick 10.32.1.144:/opt/lvmdir/c2/brick >> > > > > > Number of entries in split-brain: 0 >> > > > > > >> > > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick >> > > > > > Number of entries in split-brain: 0 >> > > > > > >> > > > > > auto-sync.sh. >> > > > > > Here you can see that i/o error is detected. Below is the >> required >> > > > > > meta >> > > > > > data from both the bricks. >> > > > > > >> > > > > > 1) >> > > > > > stat: cannot stat >> '/mnt/c//public_html/cello/ior_files/nameroot.ior': >> > > > > > Input/output error >> > > > > > Remote: >> > > > > > >> > > > > > getfattr -d -m . -e hex >> > > > > > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior >> > > > > > # file: >> opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior >> > > > > > trusted.afr.dirty=0x000000000000000000000000 >> > > > > > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f >> > > > > > trusted.gfid=0x771221a7bb3c4f1aade40ce9e38a95ee >> > > > > > >> > > > > > Local: >> > > > > > >> > > > > > getfattr -d -m . -e hex >> > > > > > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior >> > > > > > # file: >> opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior >> > > > > > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51 >> > > > > > trusted.gfid=0x8ea33f46703c4e2d95c09153c1b858fd >> > > > > > >> > > > > > >> > > > > > 2) >> > > > > > stat: cannot stat '/mnt/c//security/corbasecurity': Input/output >> > > > > > error >> > > > > > Remote: >> > > > > > >> > > > > > getfattr -d -m . -e hex >> opt/lvmdir/c2/brick/security/corbasecurity >> > > > > > # file: opt/lvmdir/c2/brick/security/corbasecurity >> > > > > > trusted.afr.dirty=0x000000000000000000000000 >> > > > > > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f >> > > > > > trusted.gfid=0xd298b7a0c8834f3e99abb39741363013 >> > > > > > >> > > > > > Local: >> > > > > > >> > > > > > getfattr -d -m . -e hex >> opt/lvmdir/c2/brick/security/corbasecurity >> > > > > > # file: opt/lvmdir/c2/brick/security/corbasecurity >> > > > > > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51 >> > > > > > trusted.gfid=0x890df0f706184b52803fac3242a2f15b >> > > > > > >> > > > > > I observed that getfattr command output doesn't show all the >> fields >> > > > > > all >> > > > > > the times. >> > > > > > >> > > > > > Here you can check that gluster split-brain command hasn't >> reported >> > > > > > any >> > > > > > split-brains but resulted in IO errors when accessed few files. >> > > > > > >> > > > > > Could you please tell me if "split-brain" command doesn't >> reported >> > > > > > any >> > > > > > entry as output, then is there any way through which we can >> find out >> > > > that >> > > > > > the files are in split-brain if we are getting the IO error on >> those >> > > > file. >> > > > > > >> > > > > > >> > > > > > Regards, >> > > > > > Abhishek >> > > > > > >> > > > > > >> > > > > > >> > > > > > On Thu, Mar 3, 2016 at 5:32 PM, Anuradha Talur < >> ata...@redhat.com> >> > > > wrote: >> > > > > > >> > > > > >> >> > > > > >> >> > > > > >> ----- Original Message ----- >> > > > > >> > From: "ABHISHEK PALIWAL" <abhishpali...@gmail.com> >> > > > > >> > To: gluster-us...@gluster.org, gluster-devel@gluster.org >> > > > > >> > Sent: Thursday, March 3, 2016 12:10:42 PM >> > > > > >> > Subject: [Gluster-users] gluster volume heal info split brain >> > > > command >> > > > > >> not showing files in split-brain >> > > > > >> > >> > > > > >> > >> > > > > >> > Hello, >> > > > > >> > >> > > > > >> > In gluster, we use the command "gluster volume heal >> c_glusterfs >> > > > > >> > info >> > > > > >> > split-brain" to find the files that are in split-brain >> scenario. >> > > > > >> > We run heal script (developed by Windriver prime team) on the >> > > > > >> > files >> > > > > >> reported >> > > > > >> > by above command to resolve split-brain issue. >> > > > > >> > >> > > > > >> > But we observed that the above command is not showing all >> files >> > > > > >> > that >> > > > > >> are in >> > > > > >> > split-brain, >> > > > > >> > even though split brain scenario actually exists on the node. >> > > > > >> > >> > > > > >> > Now a days this issue is seen more often and IO errors are >> > > > > >> > reported >> > > > when >> > > > > >> > tried to access these files under split-brain. >> > > > > >> > >> > > > > >> > Can you please check why this gluster command is not showing >> files >> > > > under >> > > > > >> > split-brain? >> > > > > >> > We can provide you required logs and support to resolve this >> > > > > >> > issue. >> > > > > >> Hi, >> > > > > >> >> > > > > >> Could you paste the output of getfattr -m. -de hex >> > > > > >> <path-to-files-in-split-brain> from all the bricks that the >> files >> > > > > >> lie >> > > > in? >> > > > > >> >> > > > > >> > >> > > > > >> > Please reply on this because I am not getting any reply from >> the >> > > > > >> community. >> > > > > >> > >> > > > > >> > -- >> > > > > >> > >> > > > > >> > Regards >> > > > > >> > Abhishek Paliwal >> > > > > >> > >> > > > > >> > _______________________________________________ >> > > > > >> > Gluster-users mailing list >> > > > > >> > gluster-us...@gluster.org >> > > > > >> > http://www.gluster.org/mailman/listinfo/gluster-users >> > > > > >> >> > > > > >> -- >> > > > > >> Thanks, >> > > > > >> Anuradha. >> > > > > >> >> > > > > > >> > > > > >> > > > >> > > > -- >> > > > Thanks, >> > > > Anuradha. >> > > > >> > > >> > > >> > > >> > > -- >> > > >> > > >> > > >> > > >> > > Regards >> > > Abhishek Paliwal >> > > >> > >> > -- >> > Thanks, >> > Anuradha. >> > >> >> -- >> Thanks, >> Anuradha. >> > > > > -- > > > > > Regards > Abhishek Paliwal > -- Regards Abhishek Paliwal
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel