Except that even when I unmount gluster from both Debian clients, I'm still getting the same error, and a dump of those clients after unmounting still shows:
glusterd.client1.identifier=10.0.231.10:1023 glusterd.client1.volname=storage glusterd.client1.max-op-version=30603 glusterd.client1.min-op-version=1 glusterd.client4.identifier=10.0.231.11:1022 glusterd.client4.volname=storage glusterd.client4.max-op-version=30603 glusterd.client4.min-op-version=1 On Thu, Mar 3, 2016 at 5:28 PM, Atin Mukherjee <atin.mukherje...@gmail.com> wrote: > -Atin > Sent from one plus one > On 04-Mar-2016 3:35 am, "Steve Dainard" <sdain...@spd1.com> wrote: > > > > FYI Gluster storage node hostnames are gluster0[1-6]. > > > > Full dump attached. I see a few clients not on 30706. Most notably the > two debian 7 servers (using packages from gluster.org) seem to be running > lower op versions than the centos7 machines (every other client in > 10.0.231.0/24 subnet). > > > > glusterd.client1.identifier=10.0.231.10:1023 <-- debian 7, glusterfs > 3.7.6 built on Feb 4 2016 06:25:19 > > glusterd.client1.volname=storage > > glusterd.client1.max-op-version=30603 > This is running with 3.6.3 and that's why volume set fails. Unmount this > client and upgrade and remount it back. > > > glusterd.client1.min-op-version=1 > > > > glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02 > (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs) > > glusterd.client2.volname= > > glusterd.client2.max-op-version=0 > > glusterd.client2.min-op-version=0 > > > > glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no actual > mounts) > > glusterd.client3.volname= > > glusterd.client3.max-op-version=0 > > glusterd.client3.min-op-version=0 > > > > glusterd.client4.identifier=10.0.231.11:1022 <--- debian 7, glusterfs > 3.7.6 built on Feb 4 2016 06:25:19 > > glusterd.client4.volname=storage > > glusterd.client4.max-op-version=30603 > > glusterd.client4.min-op-version=1 > > > > glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no actual > mounts) > > glusterd.client5.volname= > > glusterd.client5.max-op-version=0 > > glusterd.client5.min-op-version=0 > > > > glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04 > (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs) > > glusterd.client6.volname= > > glusterd.client6.max-op-version=0 > > glusterd.client6.min-op-version=0 > > > > glusterd.client7.identifier=10.0.231.50:65529 > > > glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage > > glusterd.client7.max-op-version=30706 > > glusterd.client7.min-op-version=1 > > > > ... > > > > Debian package info: > > apt-cache policy glusterfs-client > > glusterfs-client: > > Installed: 3.7.6-2 > > Candidate: 3.7.6-2 > > Version table: > > *** 3.7.6-2 0 > > 500 > http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/ > wheezy/main amd64 Packages > > > > > > Thanks > > > > On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <gg...@redhat.com> wrote: > >> > >> Hi Steve, > >> > >> As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof > glusterd) command. it will create .dump file in /var/run/gluster/ > directory. client-op-version information will be present in dump file. > >> > >> Thanks, > >> ~Gaurav > >> > >> ----- Original Message ----- > >> From: "Steve Dainard" <sdain...@spd1.com> > >> To: "Gaurav Garg" <gg...@redhat.com> > >> Cc: "gluster-users@gluster.org List" <gluster-users@gluster.org> > >> Sent: Thursday, March 3, 2016 12:07:25 AM > >> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or > more connected clients cannot support the feature being set > >> > >> From the the client side logs I can see version info on mount: > >> > >> Final graph: > >> > +------------------------------------------------------------------------------+ > >> 1: volume storage-client-0 > >> 2: type protocol/client > >> 3: option clnt-lk-version 1 > >> 4: option volfile-checksum 0 > >> 5: option volfile-key /storage > >> 6: option client-version 3.7.6 > >> 7: option process-uuid > >> > template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0 > >> 8: option fops-version 1298437 > >> 9: option ping-timeout 42 > >> 10: option remote-host 10.0.231.50 > >> 11: option remote-subvolume /mnt/raid6-storage/storage > >> 12: option transport-type socket > >> 13: option send-gids true > >> 14: end-volume > >> 15: > >> 16: volume storage-client-1 > >> 17: type protocol/client > >> 18: option clnt-lk-version 1 > >> 19: option volfile-checksum 0 > >> 20: option volfile-key /storage > >> 21: option client-version 3.7.6 > >> 22: option process-uuid > >> > template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0 > >> 23: option fops-version 1298437 > >> 24: option ping-timeout 42 > >> 25: option remote-host 10.0.231.51 > >> 26: option remote-subvolume /mnt/raid6-storage/storage > >> 27: option transport-type socket > >> 28: option send-gids true > >> 29: end-volume > >> 30: > >> 31: volume storage-client-2 > >> 32: type protocol/client > >> 33: option clnt-lk-version 1 > >> 34: option volfile-checksum 0 > >> 35: option volfile-key /storage > >> 36: option client-version 3.7.6 > >> 37: option process-uuid > >> > template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0 > >> 38: option fops-version 1298437 > >> 39: option ping-timeout 42 > >> 40: option remote-host 10.0.231.52 > >> 41: option remote-subvolume /mnt/raid6-storage/storage > >> 42: option transport-type socket > >> 43: option send-gids true > >> 44: end-volume > >> 45: > >> 46: volume storage-client-3 > >> 47: type protocol/client > >> 48: option clnt-lk-version 1 > >> 49: option volfile-checksum 0 > >> 50: option volfile-key /storage > >> 51: option client-version 3.7.6 > >> 52: option process-uuid > >> > template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0 > >> 53: option fops-version 1298437 > >> 54: option ping-timeout 42 > >> 55: option remote-host 10.0.231.53 > >> 56: option remote-subvolume /mnt/raid6-storage/storage > >> 57: option transport-type socket > >> 58: option send-gids true > >> 59: end-volume > >> 60: > >> 61: volume storage-client-4 > >> 62: type protocol/client > >> 63: option ping-timeout 42 > >> 64: option remote-host 10.0.231.54 > >> 65: option remote-subvolume /mnt/raid6-storage/storage > >> 66: option transport-type socket > >> 67: option send-gids true > >> 68: end-volume > >> 69: > >> 70: volume storage-client-5 > >> 71: type protocol/client > >> 72: option ping-timeout 42 > >> 73: option remote-host 10.0.231.55 > >> 74: option remote-subvolume /mnt/raid6-storage/storage > >> 75: option transport-type socket > >> 76: option send-gids true > >> 77: end-volume > >> 78: > >> 79: volume storage-dht > >> 80: type cluster/distribute > >> 81: subvolumes storage-client-0 storage-client-1 storage-client-2 > >> storage-client-3 storage-client-4 storage-client-5 > >> 82: end-volume > >> > >> > >> But not the client op-version, how can I retrieve this info? > >> > >> Thanks > >> > >> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <gg...@redhat.com> wrote: > >> > >> > Hi Steve, > >> > > >> > Which version you have upgraded client, could you tell us client > >> > op-version after upgrade ? > >> > > >> > > >> > have you upgraded all of your clients ? > >> > > >> > > >> > Thanks, > >> > Gaurav > >> > > >> > > >> > ----- Original Message ----- > >> > From: "Steve Dainard" <sdain...@spd1.com> > >> > To: "gluster-users@gluster.org List" <gluster-users@gluster.org> > >> > Sent: Wednesday, March 2, 2016 1:10:27 AM > >> > Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more > >> > connected clients cannot support the feature being set > >> > > >> > Gluster 3.7.6 > >> > 'storage' is a distributed volume > >> > > >> > # gluster volume set storage rebal-throttle lazy > >> > volume set: failed: One or more connected clients cannot support the > >> > feature being set. These clients need to be upgraded or disconnected > before > >> > running this command again > >> > > >> > I found a client connected using version 3.6.7 so I upgraded & > >> > umount/mount the gluster volume on the client but I'm still getting > this > >> > error. > >> > > >> > I've run grep "accepted client from" /var/log/glusterfs/bricks/* | > grep -v > >> > 3.7.6 and I get a few returns from the client above, all dated last > week. > >> > > >> > I've run 'gluster volume status storage clients' and checked the > connected > >> > clients manually, they're all running 3.7.6. > >> > > >> > /var/log/gluster/etc-glusterfs-glusterd.vol.log: > >> > [2016-03-01 19:23:20.180821] E [MSGID: 106022] > >> > [glusterd-utils.c:10154:glusterd_check_client_op_version_support] > >> > 0-management: One or more c > >> > lients don't support the required op-version > >> > [2016-03-01 19:23:20.180853] E [MSGID: 106301] > >> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of > >> > operation 'Volume Set' fa > >> > iled on localhost : One or more connected clients cannot support the > >> > feature being set. These clients need to be upgraded or disconnected > be > >> > fore running this command again > >> > > >> > Also tried setting the diagnostics.brick-log-level logging level and > got > >> > the same error. > >> > > >> > /var/lib/glusterd/vols/storage/info: > >> > type=0 > >> > count=6 > >> > status=1 > >> > sub_count=0 > >> > stripe_count=1 > >> > replica_count=1 > >> > disperse_count=0 > >> > redundancy_count=0 > >> > version=26 > >> > transport-type=0 > >> > volume-id=26d355cb-c486-481f-ac16-e25390e73775 > >> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c > >> > password= > >> > op-version=3 > >> > client-op-version=3 > >> > quota-version=1 > >> > parent_volname=N/A > >> > restored_from_snap=00000000-0000-0000-0000-000000000000 > >> > snap-max-hard-limit=256 > >> > features.quota-deem-statfs=on > >> > features.inode-quota=on > >> > diagnostics.brick-log-level=WARNING > >> > features.quota=on > >> > performance.readdir-ahead=on > >> > performance.cache-size=1GB > >> > performance.stat-prefetch=on > >> > brick-0=10.0.231.50:-mnt-raid6-storage-storage > >> > brick-1=10.0.231.51:-mnt-raid6-storage-storage > >> > brick-2=10.0.231.52:-mnt-raid6-storage-storage > >> > brick-3=10.0.231.53:-mnt-raid6-storage-storage > >> > brick-4=10.0.231.54:-mnt-raid6-storage-storage > >> > brick-5=10.0.231.55:-mnt-raid6-storage-storage > >> > > >> > > >> > _______________________________________________ > >> > Gluster-users mailing list > >> > Gluster-users@gluster.org > >> > http://www.gluster.org/mailman/listinfo/gluster-users > >> > > > > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users@gluster.org > > http://www.gluster.org/mailman/listinfo/gluster-users > >
glusterdump.1590.dump.1457110194
Description: Binary data
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users