On Fri, Jan 29, 2016 at 11:34:37AM +0000, Ronny Adsetts wrote: > Venky Shankar wrote on 29/01/2016 11:09: > > On Fri, Jan 29, 2016 at 10:46:14AM +0000, Ronny Adsetts wrote: > >> > >> No idea how it came about in the first place. Suspect it happened > >> during either an operating system upgrade or upgrading glusterfs > >> packages to those from gluster.org. > > > > If you're still willing to debug, please send across brick log file > > of the concerned node. > > Of course. Attached is the current brick log.
The log files does not have any clue about how some of the files ended up with missing .glusterfs linkage, though there are tons of errors of operations trying to access .glusterfs/.././..... (e.g. for gfid 7f0fa160-8e5d-44f5-88b3-8187b2397313). Probably an upgrade and restart of services in midst of an operation? > > My previous comments about having fixed the problem were a little premature. > I now have most of the files that were showing with zero links now in the > 'failed-heal' state: > > # gluster volume heal software statistics > [...] > Starting time of crawl: Fri Jan 29 11:15:53 2016 > > Ending time of crawl: Fri Jan 29 11:15:58 2016 > > Type of crawl: INDEX > No. of entries healed: 0 > No. of entries in split-brain: 0 > No. of heal failed entries: 1976 My guess would be, as soon as you fixed the backend linkages, self-heal was able to kick in as it could "find" these files on the replica. But why did the heal fail or why these files needed healing in the first place. Pranith, Ravi? Also, please provide self-heal logs. > > > The volume log file shows all the details but the "E" error entries look like > this: > > [2016-01-29 11:12:00.479724] E [client-rpc-fops.c:914:client3_3_flush_cbk] > 0-software-client-1: remote operation failed: Operation not permitted > > > And warnings like this: > > [2016-01-29 11:11:33.747428] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] > 0-software-client-1: remote operation failed: No such file or directory. > Path: /win_patches/NDP20SP2-KB2446704-v2-x86.exe > (9aa6b79a-b18f-42f2-ae9a-a886c523a0e7) > [2016-01-29 11:11:33.749192] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] > 0-software-client-1: remote operation failed: No such file or directory. > Path: <gfid:9aa6b79a-b18f-42f2-ae9a-a886c523a0e7> > (9aa6b79a-b18f-42f2-ae9a-a886c523a0e7) > > > Ronny > > -- > Ronny Adsetts > Technical Director > Amazing Internet Ltd, London > t: +44 20 8977 8943 > w: www.amazinginternet.com > > Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ > Registered in England. Company No. 4042957 > > _______________________________________________ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
