Thanks, I'll do that. Is it possible/likely that this lower op-version is also causing the issue I posted on gluster-users earlier: 0-rpcsvc: rpc actor failed to complete successfully https://www.mail-archive.com/gluster-users@gluster.org/msg20569.html
Any pointers on that would be greatly appreciated, since we've had multiple occurences of this since Sunday, three today only. Thanks, On 10 June 2015 at 14:42, Atin Mukherjee <amukh...@redhat.com> wrote: > > > On 06/10/2015 05:32 PM, Tiemen Ruiten wrote: > > Hello Atin, > > > > We are running 3.7.0 on our storage nodes and suffer from the same issue. > > Is it safe to perform the same command or should we first upgrade to > 3.7.1? > You should upgrade to 3.7.1 > > > > On 10 June 2015 at 13:45, Atin Mukherjee <amukh...@redhat.com> wrote: > > > >> > >> > >> On 06/10/2015 02:58 PM, Sergio Traldi wrote: > >>> On 06/10/2015 10:27 AM, Krishnan Parthasarathi wrote: > >>>>> Hi all, > >>>>> I two servers with 3.7.1 and have the same problem of this issue: > >>>>> http://comments.gmane.org/gmane.comp.file-systems.gluster.user/20693 > >>>>> > >>>>> My servers packages: > >>>>> # rpm -qa | grep gluster | sort > >>>>> glusterfs-3.7.1-1.el6.x86_64 > >>>>> glusterfs-api-3.7.1-1.el6.x86_64 > >>>>> glusterfs-cli-3.7.1-1.el6.x86_64 > >>>>> glusterfs-client-xlators-3.7.1-1.el6.x86_64 > >>>>> glusterfs-fuse-3.7.1-1.el6.x86_64 > >>>>> glusterfs-geo-replication-3.7.1-1.el6.x86_64 > >>>>> glusterfs-libs-3.7.1-1.el6.x86_64 > >>>>> glusterfs-server-3.7.1-1.el6.x86_64 > >>>>> > >>>>> Command: > >>>>> # gluster volume status > >>>>> Another transaction is in progress. Please try again after sometime. > >> The problem is although you are running 3.7.1 binaries the cluster > >> op-version is set to 30501, because of glusterd still goes for acquiring > >> cluster lock instead of volume wise lock for every request. Command log > >> history indicates glusterD is getting multiple volume's status requests > >> and because of it fails to acquire cluster lock. Could you bump up your > >> cluster's op-version by the following command and recheck? > >> > >> gluster volume set all cluster.op-version 30701 > >> > >> ~Atin > >>>>> > >>>>> > >>>>> In /var/log/gluster/etc-glusterfs-glusterd.vol.log I found: > >>>>> > >>>>> [2015-06-09 16:12:38.949842] E [glusterd-utils.c:164:glusterd_lock] > >>>>> 0-management: Unable to get lock for uuid: > >>>>> 99a41a2a-2ce5-461c-aec0-510bd5b37bf2, lock held by: > >>>>> 04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c > >>>>> [2015-06-09 16:12:38.949864] E > >>>>> [glusterd-syncop.c:1766:gd_sync_task_begin] > >>>>> 0-management: Unable to acquire lock > >>>>> > >>>>> I check the files: > >>>>> From server 1: > >>>>> # cat /var/lib/glusterd/peers/04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c > >>>>> uuid=04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c > >>>>> state=3 > >>>>> hostname1=192.168.61.101 > >>>>> > >>>>> From server 2: > >>>>> # cat /var/lib/glusterd/peers/99a41a2a-2ce5-461c-aec0-510bd5b37bf2 > >>>>> uuid=99a41a2a-2ce5-461c-aec0-510bd5b37bf2 > >>>>> state=3 > >>>>> hostname1=192.168.61.100 > >>>> Could you attach the complete glusterd log file and cmd-history.log > >>>> file under /var/log/glusterfs directory? Could you provide a more > >>>> detailed listing of things you did before hitting this issue? > >>> Hi Krishnan, > >>> thanks to a quick answer. > >>> In attach you can found the two log you request: > >>> cmd_history.log > >>> etc-glusterfs-glusterd.vol.log > >>> > >>> We use the gluster volume as openstack nova, glance, cinder backend. > >>> > >>> The volume is configured using 2 bricks mounted by an iscsi device: > >>> [root@cld-stg-01 glusterfs]# gluster volume info volume-nova-prod > >>> Volume Name: volume-nova-prod > >>> Type: Distribute > >>> Volume ID: 4bbef4c8-0441-4e81-a2c5-559401adadc0 > >>> Status: Started > >>> Number of Bricks: 2 > >>> Transport-type: tcp > >>> Bricks: > >>> Brick1: 192.168.61.100:/brickOpenstack/nova-prod/mpathb > >>> Brick2: 192.168.61.101:/brickOpenstack/nova-prod/mpathb > >>> Options Reconfigured: > >>> storage.owner-gid: 162 > >>> storage.owner-uid: 162 > >>> > >>> Last week we update openstack from havana to icehouse and we rename the > >>> storage hosts but we didn't change the IP. > >>> All volume have been created using ip addresses. > >>> > >>> So last week we stop all services (openstack gluster and also iscsi). > >>> We change the name in DNS of private ip of the 2 nics. > >>> We reboot the storage servers > >>> We start agian iscsi, multipath, glusterd process. > >>> We have to stop and start the volumes, but after that everything works > >>> fine. > >>> Now we don't observe any other problems except this. > >>> > >>> We have a nagios probe which check the volume status each 5 minutes to > >>> ensure all gluster process is working fine and so we find this problem > I > >>> post. > >>> > >>> Cheer > >>> Sergio > >>> > >>> > >>> _______________________________________________ > >>> Gluster-users mailing list > >>> gluster-us...@gluster.org > >>> http://www.gluster.org/mailman/listinfo/gluster-users > >>> > >> > >> -- > >> ~Atin > >> _______________________________________________ > >> Gluster-users mailing list > >> gluster-us...@gluster.org > >> http://www.gluster.org/mailman/listinfo/gluster-users > >> > > > > > > > > -- > ~Atin > -- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel