And I'll mention again, those Debian clients are not running 3.6. As shown in previous email, they are both running:
# apt-cache policy glusterfs-client glusterfs-client: Installed: 3.7.6-2 Candidate: 3.7.6-2 Version table: *** 3.7.6-2 0 500 http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/ wheezy/main amd64 Packages # glusterfs --version glusterfs 3.7.6 built on Feb 4 2016 06:25:19 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. On Fri, Mar 4, 2016 at 8:53 AM, Steve Dainard <sdain...@spd1.com> wrote: > Except that even when I unmount gluster from both Debian clients, I'm > still getting the same error, and a dump of those clients after unmounting > still shows: > > glusterd.client1.identifier=10.0.231.10:1023 > glusterd.client1.volname=storage > glusterd.client1.max-op-version=30603 > glusterd.client1.min-op-version=1 > > glusterd.client4.identifier=10.0.231.11:1022 > glusterd.client4.volname=storage > glusterd.client4.max-op-version=30603 > glusterd.client4.min-op-version=1 > > > > > > On Thu, Mar 3, 2016 at 5:28 PM, Atin Mukherjee <atin.mukherje...@gmail.com > > wrote: > >> -Atin >> Sent from one plus one >> On 04-Mar-2016 3:35 am, "Steve Dainard" <sdain...@spd1.com> wrote: >> > >> > FYI Gluster storage node hostnames are gluster0[1-6]. >> > >> > Full dump attached. I see a few clients not on 30706. Most notably the >> two debian 7 servers (using packages from gluster.org) seem to be >> running lower op versions than the centos7 machines (every other client in >> 10.0.231.0/24 subnet). >> > >> > glusterd.client1.identifier=10.0.231.10:1023 <-- debian 7, glusterfs >> 3.7.6 built on Feb 4 2016 06:25:19 >> > glusterd.client1.volname=storage >> > glusterd.client1.max-op-version=30603 >> This is running with 3.6.3 and that's why volume set fails. Unmount this >> client and upgrade and remount it back. >> >> > glusterd.client1.min-op-version=1 >> > >> > glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02 >> (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs) >> > glusterd.client2.volname= >> > glusterd.client2.max-op-version=0 >> > glusterd.client2.min-op-version=0 >> > >> > glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no actual >> mounts) >> > glusterd.client3.volname= >> > glusterd.client3.max-op-version=0 >> > glusterd.client3.min-op-version=0 >> > >> > glusterd.client4.identifier=10.0.231.11:1022 <--- debian 7, glusterfs >> 3.7.6 built on Feb 4 2016 06:25:19 >> > glusterd.client4.volname=storage >> > glusterd.client4.max-op-version=30603 >> > glusterd.client4.min-op-version=1 >> > >> > glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no actual >> mounts) >> > glusterd.client5.volname= >> > glusterd.client5.max-op-version=0 >> > glusterd.client5.min-op-version=0 >> > >> > glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04 >> (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs) >> > glusterd.client6.volname= >> > glusterd.client6.max-op-version=0 >> > glusterd.client6.min-op-version=0 >> > >> > glusterd.client7.identifier=10.0.231.50:65529 >> > >> glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage >> > glusterd.client7.max-op-version=30706 >> > glusterd.client7.min-op-version=1 >> > >> > ... >> > >> > Debian package info: >> > apt-cache policy glusterfs-client >> > glusterfs-client: >> > Installed: 3.7.6-2 >> > Candidate: 3.7.6-2 >> > Version table: >> > *** 3.7.6-2 0 >> > 500 >> http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/ >> wheezy/main amd64 Packages >> > >> > >> > Thanks >> > >> > On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <gg...@redhat.com> wrote: >> >> >> >> Hi Steve, >> >> >> >> As atin pointed out to take statedump by running #kill -SIGUSR1 >> $(pidof glusterd) command. it will create .dump file in /var/run/gluster/ >> directory. client-op-version information will be present in dump file. >> >> >> >> Thanks, >> >> ~Gaurav >> >> >> >> ----- Original Message ----- >> >> From: "Steve Dainard" <sdain...@spd1.com> >> >> To: "Gaurav Garg" <gg...@redhat.com> >> >> Cc: "gluster-users@gluster.org List" <gluster-users@gluster.org> >> >> Sent: Thursday, March 3, 2016 12:07:25 AM >> >> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or >> more connected clients cannot support the feature being set >> >> >> >> From the the client side logs I can see version info on mount: >> >> >> >> Final graph: >> >> >> +------------------------------------------------------------------------------+ >> >> 1: volume storage-client-0 >> >> 2: type protocol/client >> >> 3: option clnt-lk-version 1 >> >> 4: option volfile-checksum 0 >> >> 5: option volfile-key /storage >> >> 6: option client-version 3.7.6 >> >> 7: option process-uuid >> >> >> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0 >> >> 8: option fops-version 1298437 >> >> 9: option ping-timeout 42 >> >> 10: option remote-host 10.0.231.50 >> >> 11: option remote-subvolume /mnt/raid6-storage/storage >> >> 12: option transport-type socket >> >> 13: option send-gids true >> >> 14: end-volume >> >> 15: >> >> 16: volume storage-client-1 >> >> 17: type protocol/client >> >> 18: option clnt-lk-version 1 >> >> 19: option volfile-checksum 0 >> >> 20: option volfile-key /storage >> >> 21: option client-version 3.7.6 >> >> 22: option process-uuid >> >> >> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0 >> >> 23: option fops-version 1298437 >> >> 24: option ping-timeout 42 >> >> 25: option remote-host 10.0.231.51 >> >> 26: option remote-subvolume /mnt/raid6-storage/storage >> >> 27: option transport-type socket >> >> 28: option send-gids true >> >> 29: end-volume >> >> 30: >> >> 31: volume storage-client-2 >> >> 32: type protocol/client >> >> 33: option clnt-lk-version 1 >> >> 34: option volfile-checksum 0 >> >> 35: option volfile-key /storage >> >> 36: option client-version 3.7.6 >> >> 37: option process-uuid >> >> >> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0 >> >> 38: option fops-version 1298437 >> >> 39: option ping-timeout 42 >> >> 40: option remote-host 10.0.231.52 >> >> 41: option remote-subvolume /mnt/raid6-storage/storage >> >> 42: option transport-type socket >> >> 43: option send-gids true >> >> 44: end-volume >> >> 45: >> >> 46: volume storage-client-3 >> >> 47: type protocol/client >> >> 48: option clnt-lk-version 1 >> >> 49: option volfile-checksum 0 >> >> 50: option volfile-key /storage >> >> 51: option client-version 3.7.6 >> >> 52: option process-uuid >> >> >> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0 >> >> 53: option fops-version 1298437 >> >> 54: option ping-timeout 42 >> >> 55: option remote-host 10.0.231.53 >> >> 56: option remote-subvolume /mnt/raid6-storage/storage >> >> 57: option transport-type socket >> >> 58: option send-gids true >> >> 59: end-volume >> >> 60: >> >> 61: volume storage-client-4 >> >> 62: type protocol/client >> >> 63: option ping-timeout 42 >> >> 64: option remote-host 10.0.231.54 >> >> 65: option remote-subvolume /mnt/raid6-storage/storage >> >> 66: option transport-type socket >> >> 67: option send-gids true >> >> 68: end-volume >> >> 69: >> >> 70: volume storage-client-5 >> >> 71: type protocol/client >> >> 72: option ping-timeout 42 >> >> 73: option remote-host 10.0.231.55 >> >> 74: option remote-subvolume /mnt/raid6-storage/storage >> >> 75: option transport-type socket >> >> 76: option send-gids true >> >> 77: end-volume >> >> 78: >> >> 79: volume storage-dht >> >> 80: type cluster/distribute >> >> 81: subvolumes storage-client-0 storage-client-1 storage-client-2 >> >> storage-client-3 storage-client-4 storage-client-5 >> >> 82: end-volume >> >> >> >> >> >> But not the client op-version, how can I retrieve this info? >> >> >> >> Thanks >> >> >> >> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <gg...@redhat.com> wrote: >> >> >> >> > Hi Steve, >> >> > >> >> > Which version you have upgraded client, could you tell us client >> >> > op-version after upgrade ? >> >> > >> >> > >> >> > have you upgraded all of your clients ? >> >> > >> >> > >> >> > Thanks, >> >> > Gaurav >> >> > >> >> > >> >> > ----- Original Message ----- >> >> > From: "Steve Dainard" <sdain...@spd1.com> >> >> > To: "gluster-users@gluster.org List" <gluster-users@gluster.org> >> >> > Sent: Wednesday, March 2, 2016 1:10:27 AM >> >> > Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or >> more >> >> > connected clients cannot support the feature being set >> >> > >> >> > Gluster 3.7.6 >> >> > 'storage' is a distributed volume >> >> > >> >> > # gluster volume set storage rebal-throttle lazy >> >> > volume set: failed: One or more connected clients cannot support the >> >> > feature being set. These clients need to be upgraded or disconnected >> before >> >> > running this command again >> >> > >> >> > I found a client connected using version 3.6.7 so I upgraded & >> >> > umount/mount the gluster volume on the client but I'm still getting >> this >> >> > error. >> >> > >> >> > I've run grep "accepted client from" /var/log/glusterfs/bricks/* | >> grep -v >> >> > 3.7.6 and I get a few returns from the client above, all dated last >> week. >> >> > >> >> > I've run 'gluster volume status storage clients' and checked the >> connected >> >> > clients manually, they're all running 3.7.6. >> >> > >> >> > /var/log/gluster/etc-glusterfs-glusterd.vol.log: >> >> > [2016-03-01 19:23:20.180821] E [MSGID: 106022] >> >> > [glusterd-utils.c:10154:glusterd_check_client_op_version_support] >> >> > 0-management: One or more c >> >> > lients don't support the required op-version >> >> > [2016-03-01 19:23:20.180853] E [MSGID: 106301] >> >> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of >> >> > operation 'Volume Set' fa >> >> > iled on localhost : One or more connected clients cannot support the >> >> > feature being set. These clients need to be upgraded or disconnected >> be >> >> > fore running this command again >> >> > >> >> > Also tried setting the diagnostics.brick-log-level logging level and >> got >> >> > the same error. >> >> > >> >> > /var/lib/glusterd/vols/storage/info: >> >> > type=0 >> >> > count=6 >> >> > status=1 >> >> > sub_count=0 >> >> > stripe_count=1 >> >> > replica_count=1 >> >> > disperse_count=0 >> >> > redundancy_count=0 >> >> > version=26 >> >> > transport-type=0 >> >> > volume-id=26d355cb-c486-481f-ac16-e25390e73775 >> >> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c >> >> > password= >> >> > op-version=3 >> >> > client-op-version=3 >> >> > quota-version=1 >> >> > parent_volname=N/A >> >> > restored_from_snap=00000000-0000-0000-0000-000000000000 >> >> > snap-max-hard-limit=256 >> >> > features.quota-deem-statfs=on >> >> > features.inode-quota=on >> >> > diagnostics.brick-log-level=WARNING >> >> > features.quota=on >> >> > performance.readdir-ahead=on >> >> > performance.cache-size=1GB >> >> > performance.stat-prefetch=on >> >> > brick-0=10.0.231.50:-mnt-raid6-storage-storage >> >> > brick-1=10.0.231.51:-mnt-raid6-storage-storage >> >> > brick-2=10.0.231.52:-mnt-raid6-storage-storage >> >> > brick-3=10.0.231.53:-mnt-raid6-storage-storage >> >> > brick-4=10.0.231.54:-mnt-raid6-storage-storage >> >> > brick-5=10.0.231.55:-mnt-raid6-storage-storage >> >> > >> >> > >> >> > _______________________________________________ >> >> > Gluster-users mailing list >> >> > Gluster-users@gluster.org >> >> > http://www.gluster.org/mailman/listinfo/gluster-users >> >> > >> > >> > >> > >> > _______________________________________________ >> > Gluster-users mailing list >> > Gluster-users@gluster.org >> > http://www.gluster.org/mailman/listinfo/gluster-users >> >> >
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users