-Atin Sent from one plus one On Aug 3, 2015 8:31 PM, "Osborne, Paul (paul.osbo...@canterbury.ac.uk)" < paul.osbo...@canterbury.ac.uk> wrote: > > Hi, > > > OK I have tracked through the logs which of the hosts apparently has a lock open: > > > [2015-08-03 14:55:37.602717] I [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: Received status volume req for volume blogs > > [2015-08-03 14:51:57.791081] E [glusterd-utils.c:148:glusterd_lock] 0-management: Unable to get lock for uuid: 76e4398c-e00a-4f3b-9206-4f885c4e5206, lock held by: 76e4398c-e00a-4f3b-9206-4f885c4e5206 > This indicates that cluster is still operating at older op version. You would need to bump up the op version to 30604 using Gluster volume set all cluster.op-version 30604 > > I have identified the UID for each peer via gluster peer status and working backwards. > > I see that gluster volume clear-locks may the locks on the volume - but is not clear from the logs is what the path is that has the lock or the kind that is locked. > > Incidentally my clients (using NFS) through manual testing appear to still be able to read/write to the volume - it is the volume status and heal checks that are failing. All of my clients and servers have been sequentially rebooted in the hope that this would clear any issue - however that doe not appear to be the case. > > > > Thanks > > Paul > > > > > Paul Osborne > Senior Systems Engineer > Canterbury Christ Church University > Tel: 01227 782751 > > > ________________________________ > From: Atin Mukherjee <atin.mukherje...@gmail.com> > Sent: 03 August 2015 15:22 > To: Osborne, Paul (paul.osbo...@canterbury.ac.uk) > Cc: gluster-users@gluster.org > Subject: Re: [Gluster-users] Locking failed - since upgrade to 3.6.4 > > > Could you check the glusterd log at the other nodes, that would give you the hint of the exact issue. Also looking at .cmd_log_history will give you the time interval at which volume status commands are executed. If the gap is in milisecs then you are bound to hit it and its expected. > > -Atin > Sent from one plus one > > On Aug 3, 2015 7:32 PM, "Osborne, Paul (paul.osbo...@canterbury.ac.uk)" < paul.osbo...@canterbury.ac.uk> wrote: >> >> >> Hi, >> >> Last week I upgraded one of my gluster clusters (3 hosts with bricks as replica 3) to 3.6.4 from 3.5.4 and all seemed well. >> >> Today I am getting reports that locking has failed: >> >> >> gfse-cant-01:/var/log/glusterfs# gluster volume status >> Locking failed on gfse-rh-01.core.canterbury.ac.uk. Please check log file for details. >> Locking failed on gfse-isr-01.core.canterbury.ac.uk. Please check log file for details. >> >> Logs: >> [2015-08-03 13:45:29.974560] E [glusterd-syncop.c:1640:gd_sync_task_begin] 0-management: Locking Peers Failed. >> [2015-08-03 13:49:48.273159] E [glusterd-syncop.c:105:gd_collate_errors] 0-: Locking failed on gfse-rh-01.core.canterbury.ac.uk. Please ch >> eck log file for details. >> [2015-08-03 13:49:48.273778] E [glusterd-syncop.c:105:gd_collate_errors] 0-: Locking failed on gfse-isr-01.core.canterbury.ac.uk. Please c >> heck log file for details. >> >> >> I am wondering if this is a new feature due to 3.6.4 or something that has gone wrong. >> >> Restarting gluster entirely (btw the restart script does not actually appear to kill the processes...) resolves the issue but then it repeats a few minutes later which is rather suboptimal for a running service. >> >> Googling suggests that there may be simultaneous actions going on that can cause a locking issue. >> >> I know that I have nagios running volume status <volname> for each of my volumes on each host every few minutes however this is not new and has been in place for the last 8-9 months that against 3.5 without issue so would hope that this is not causing the issue. >> >> I am not sure where to look now tbh. >> >> >> >> >> Paul Osborne >> Senior Systems Engineer >> Canterbury Christ Church University >> Tel: 01227 782751 >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users