On Mon, 3 Jul 2017 at 12:28, Pawan Alwandi <pa...@platform.sh> wrote:
> Hello Atin, > > I've gotten around to this and was able to get upgrade done using 3.7.0 > before moving to 3.11. For some reason 3.7.9 wasn't working well. > > On 3.11 though I notice that gluster/nfs is really made optional and > nfs-ganesha is being recommended. We have plans to switch to nfs-ganesha > on new clusters but would like to have glusterfs-gnfs on existing clusters > so a seamless upgrade without downtime is possible. > > [2017-07-03 06:43:25.511893] I [MSGID: 106600] > [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so > xlator is not installed > > I was really looking for glusterfs-gnfs package and noticed that .deb is > missing - > https://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/8/apt/pool/main/g/glusterfs/ > (fwiw, only the rpm is available). Is it possible that glusterfs-gnfs be > made available for debian too? > Kaleb - can you please help answering to this query? > > Thanks, > Pawan > > > On Wed, May 31, 2017 at 5:26 PM, Atin Mukherjee <amukh...@redhat.com> > wrote: > >> >> >> On Wed, May 31, 2017 at 3:53 PM, Pawan Alwandi <pa...@platform.sh> wrote: >> >>> Hello Atin, >>> >>> Sure. A note though, we are running gluster on Debain Jessie/Wheezy >>> hosts, but if you let me know what info you would need I'll work to collect >>> that and send across. >>> >> >> Basically I need glusterd log file (starting from last restart) along >> with the brick logs collected from all the nodes. >> >> >>> Pawan >>> >>> On Wed, May 31, 2017 at 2:10 PM, Atin Mukherjee <amukh...@redhat.com> >>> wrote: >>> >>>> Pawan, >>>> >>>> I'd need the sosreport from all the nodes to debug and figure out >>>> what's going wrong. You'd have to give me some time as I have some critical >>>> backlog items to work on. >>>> >>>> On Wed, 31 May 2017 at 11:30, Pawan Alwandi <pa...@platform.sh> wrote: >>>> >>>>> Hello Atin, >>>>> >>>>> I've tried restarting gluster one after another, but still see the >>>>> same result. >>>>> >>>>> >>>>> On Tue, May 30, 2017 at 10:40 AM, Atin Mukherjee <amukh...@redhat.com> >>>>> wrote: >>>>> >>>>>> Pawan - I couldn't reach to any conclusive analysis so far. But, >>>>>> looking at the client (nfs) & glusterd log files, it does look like that >>>>>> there is an issue w.r.t peer connections. Does restarting all the >>>>>> glusterd >>>>>> one by one solve this? >>>>>> >>>>>> On Mon, May 29, 2017 at 4:50 PM, Pawan Alwandi <pa...@platform.sh> >>>>>> wrote: >>>>>> >>>>>>> Sorry for big attachment in previous mail...last 1000 lines of those >>>>>>> logs attached now. >>>>>>> >>>>>>> On Mon, May 29, 2017 at 4:44 PM, Pawan Alwandi <pa...@platform.sh> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, May 25, 2017 at 9:54 PM, Atin Mukherjee < >>>>>>>> amukh...@redhat.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, 25 May 2017 at 19:11, Pawan Alwandi <pa...@platform.sh> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hello Atin, >>>>>>>>>> >>>>>>>>>> Yes, glusterd on other instances are up and running. Below is >>>>>>>>>> the requested output on all the three hosts. >>>>>>>>>> >>>>>>>>>> Host 1 >>>>>>>>>> >>>>>>>>>> # gluster peer status >>>>>>>>>> Number of Peers: 2 >>>>>>>>>> >>>>>>>>>> Hostname: 192.168.0.7 >>>>>>>>>> Uuid: 5ec54b4f-f60c-48c6-9e55-95f2bb58f633 >>>>>>>>>> State: Peer in Cluster (Disconnected) >>>>>>>>>> >>>>>>>>> >>>>>>>>> Glusterd is disconnected here. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hostname: 192.168.0.6 >>>>>>>>>> Uuid: 83e9a0b9-6bd5-483b-8516-d8928805ed95 >>>>>>>>>> State: Peer in Cluster (Disconnected) >>>>>>>>>> >>>>>>>>> >>>>>>>>> Same as above >>>>>>>>> >>>>>>>>> Can you please check what does glusterd log have to say here about >>>>>>>>> these disconnects? >>>>>>>>> >>>>>>>> >>>>>>>> glusterd keeps logging this every 3s >>>>>>>> >>>>>>>> [2017-05-29 11:04:52.182782] W [socket.c:852:__socket_keepalive] >>>>>>>> 0-socket: failed to set keep idle -1 on socket 5, Invalid argument >>>>>>>> [2017-05-29 11:04:52.182808] E [socket.c:2966:socket_connect] >>>>>>>> 0-management: Failed to set keep-alive: Invalid argument >>>>>>>> [2017-05-29 11:04:52.183032] W [socket.c:852:__socket_keepalive] >>>>>>>> 0-socket: failed to set keep idle -1 on socket 20, Invalid argument >>>>>>>> [2017-05-29 11:04:52.183052] E [socket.c:2966:socket_connect] >>>>>>>> 0-management: Failed to set keep-alive: Invalid argument >>>>>>>> [2017-05-29 11:04:52.183622] E [rpc-clnt.c:362:saved_frames_unwind] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f767c46d483] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f767c2384ce] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7f767c239c8e] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f767c23a4a8] >>>>>>>> ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) >>>>>>>> op(DUMP(1)) >>>>>>>> called at 2017-05-29 11:04:52.183210 (xid=0x23419) >>>>>>>> [2017-05-29 11:04:52.183735] W >>>>>>>> [glusterd-locks.c:681:glusterd_mgmt_v3_unlock] >>>>>>>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x4b) >>>>>>>> [0x7f767734dffb] >>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x14a) >>>>>>>> [0x7f7677357c6a] >>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x4c3) >>>>>>>> [0x7f76773f0ef3] ) 0-management: Lock for vol shared not held >>>>>>>> [2017-05-29 11:04:52.183928] E [rpc-clnt.c:362:saved_frames_unwind] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f767c46d483] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f767c2384ce] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7f767c239c8e] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f767c23a4a8] >>>>>>>> ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) >>>>>>>> op(DUMP(1)) >>>>>>>> called at 2017-05-29 11:04:52.183422 (xid=0x23419) >>>>>>>> [2017-05-29 11:04:52.184027] W >>>>>>>> [glusterd-locks.c:681:glusterd_mgmt_v3_unlock] >>>>>>>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x4b) >>>>>>>> [0x7f767734dffb] >>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x14a) >>>>>>>> [0x7f7677357c6a] >>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x4c3) >>>>>>>> [0x7f76773f0ef3] ) 0-management: Lock for vol shared not held >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> # gluster volume status >>>>>>>>>> Status of volume: shared >>>>>>>>>> Gluster process TCP Port RDMA Port >>>>>>>>>> Online Pid >>>>>>>>>> >>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>> Brick 192.168.0.5:/data/exports/shared 49152 0 >>>>>>>>>> Y 2105 >>>>>>>>>> NFS Server on localhost 2049 0 >>>>>>>>>> Y 2089 >>>>>>>>>> Self-heal Daemon on localhost N/A N/A >>>>>>>>>> Y 2097 >>>>>>>>>> >>>>>>>>> >>>>>>>>> Volume status output does show all the bricks are up. So I'm not >>>>>>>>> sure why are you seeing the volume as read only. Can you please >>>>>>>>> provide the >>>>>>>>> mount log? >>>>>>>>> >>>>>>>> >>>>>>>> The attached tar has nfs.log, etc-glusterfs-glusterd.vol.log, >>>>>>>> glustershd.log from host1. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Task Status of Volume shared >>>>>>>>>> >>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>> There are no active volume tasks >>>>>>>>>> >>>>>>>>>> Host 2 >>>>>>>>>> >>>>>>>>>> # gluster peer status >>>>>>>>>> Number of Peers: 2 >>>>>>>>>> >>>>>>>>>> Hostname: 192.168.0.7 >>>>>>>>>> Uuid: 5ec54b4f-f60c-48c6-9e55-95f2bb58f633 >>>>>>>>>> State: Peer in Cluster (Connected) >>>>>>>>>> >>>>>>>>>> Hostname: 192.168.0.5 >>>>>>>>>> Uuid: 7f2a6e11-2a53-4ab4-9ceb-8be6a9f2d073 >>>>>>>>>> State: Peer in Cluster (Connected) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> # gluster volume status >>>>>>>>>> Status of volume: shared >>>>>>>>>> Gluster process Port Online Pid >>>>>>>>>> >>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>> Brick 192.168.0.5:/data/exports/shared 49152 Y >>>>>>>>>> 2105 >>>>>>>>>> Brick 192.168.0.6:/data/exports/shared 49152 Y >>>>>>>>>> 2188 >>>>>>>>>> Brick 192.168.0.7:/data/exports/shared 49152 Y >>>>>>>>>> 2453 >>>>>>>>>> NFS Server on localhost 2049 Y 2194 >>>>>>>>>> Self-heal Daemon on localhost N/A Y 2199 >>>>>>>>>> NFS Server on 192.168.0.5 2049 Y 2089 >>>>>>>>>> Self-heal Daemon on 192.168.0.5 N/A Y 2097 >>>>>>>>>> NFS Server on 192.168.0.7 2049 Y 2458 >>>>>>>>>> Self-heal Daemon on 192.168.0.7 N/A Y 2463 >>>>>>>>>> >>>>>>>>>> Task Status of Volume shared >>>>>>>>>> >>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>> There are no active volume tasks >>>>>>>>>> >>>>>>>>>> Host 3 >>>>>>>>>> >>>>>>>>>> # gluster peer status >>>>>>>>>> Number of Peers: 2 >>>>>>>>>> >>>>>>>>>> Hostname: 192.168.0.5 >>>>>>>>>> Uuid: 7f2a6e11-2a53-4ab4-9ceb-8be6a9f2d073 >>>>>>>>>> State: Peer in Cluster (Connected) >>>>>>>>>> >>>>>>>>>> Hostname: 192.168.0.6 >>>>>>>>>> Uuid: 83e9a0b9-6bd5-483b-8516-d8928805ed95 >>>>>>>>>> State: Peer in Cluster (Connected) >>>>>>>>>> >>>>>>>>>> # gluster volume status >>>>>>>>>> Status of volume: shared >>>>>>>>>> Gluster process Port Online Pid >>>>>>>>>> >>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>> Brick 192.168.0.5:/data/exports/shared 49152 Y >>>>>>>>>> 2105 >>>>>>>>>> Brick 192.168.0.6:/data/exports/shared 49152 Y >>>>>>>>>> 2188 >>>>>>>>>> Brick 192.168.0.7:/data/exports/shared 49152 Y >>>>>>>>>> 2453 >>>>>>>>>> NFS Server on localhost 2049 Y 2458 >>>>>>>>>> Self-heal Daemon on localhost N/A Y 2463 >>>>>>>>>> NFS Server on 192.168.0.6 2049 Y 2194 >>>>>>>>>> Self-heal Daemon on 192.168.0.6 N/A Y 2199 >>>>>>>>>> NFS Server on 192.168.0.5 2049 Y 2089 >>>>>>>>>> Self-heal Daemon on 192.168.0.5 N/A Y 2097 >>>>>>>>>> >>>>>>>>>> Task Status of Volume shared >>>>>>>>>> >>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>> There are no active volume tasks >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, May 24, 2017 at 8:32 PM, Atin Mukherjee < >>>>>>>>>> amukh...@redhat.com> wrote: >>>>>>>>>> >>>>>>>>>>> Are the other glusterd instances are up? output of gluster peer >>>>>>>>>>> status & gluster volume status please? >>>>>>>>>>> >>>>>>>>>>> On Wed, May 24, 2017 at 4:20 PM, Pawan Alwandi < >>>>>>>>>>> pa...@platform.sh> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks Atin, >>>>>>>>>>>> >>>>>>>>>>>> So I got gluster downgraded to 3.7.9 on host 1 and now have the >>>>>>>>>>>> glusterfs and glusterfsd processes come up. But I see the volume >>>>>>>>>>>> is >>>>>>>>>>>> mounted read only. >>>>>>>>>>>> >>>>>>>>>>>> I see these being logged every 3s: >>>>>>>>>>>> >>>>>>>>>>>> [2017-05-24 10:45:44.440435] W >>>>>>>>>>>> [socket.c:852:__socket_keepalive] 0-socket: failed to set keep >>>>>>>>>>>> idle -1 on >>>>>>>>>>>> socket 17, Invalid argument >>>>>>>>>>>> [2017-05-24 10:45:44.440475] E [socket.c:2966:socket_connect] >>>>>>>>>>>> 0-management: Failed to set keep-alive: Invalid argument >>>>>>>>>>>> [2017-05-24 10:45:44.440734] W >>>>>>>>>>>> [socket.c:852:__socket_keepalive] 0-socket: failed to set keep >>>>>>>>>>>> idle -1 on >>>>>>>>>>>> socket 20, Invalid argument >>>>>>>>>>>> [2017-05-24 10:45:44.440754] E [socket.c:2966:socket_connect] >>>>>>>>>>>> 0-management: Failed to set keep-alive: Invalid argument >>>>>>>>>>>> [2017-05-24 10:45:44.441354] E >>>>>>>>>>>> [rpc-clnt.c:362:saved_frames_unwind] (--> >>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f767c46d483] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f767c2384ce] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7f767c239c8e] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f767c23a4a8] >>>>>>>>>>>> ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) >>>>>>>>>>>> op(DUMP(1)) >>>>>>>>>>>> called at 2017-05-24 10:45:44.440945 (xid=0xbf) >>>>>>>>>>>> [2017-05-24 10:45:44.441505] W >>>>>>>>>>>> [glusterd-locks.c:681:glusterd_mgmt_v3_unlock] >>>>>>>>>>>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x4b) >>>>>>>>>>>> [0x7f767734dffb] >>>>>>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x14a) >>>>>>>>>>>> [0x7f7677357c6a] >>>>>>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x4c3) >>>>>>>>>>>> [0x7f76773f0ef3] ) 0-management: Lock for vol shared not held >>>>>>>>>>>> [2017-05-24 10:45:44.441660] E >>>>>>>>>>>> [rpc-clnt.c:362:saved_frames_unwind] (--> >>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f767c46d483] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f767c2384ce] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7f767c239c8e] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f767c23a4a8] >>>>>>>>>>>> ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) >>>>>>>>>>>> op(DUMP(1)) >>>>>>>>>>>> called at 2017-05-24 10:45:44.441086 (xid=0xbf) >>>>>>>>>>>> [2017-05-24 10:45:44.441790] W >>>>>>>>>>>> [glusterd-locks.c:681:glusterd_mgmt_v3_unlock] >>>>>>>>>>>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x4b) >>>>>>>>>>>> [0x7f767734dffb] >>>>>>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x14a) >>>>>>>>>>>> [0x7f7677357c6a] >>>>>>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x4c3) >>>>>>>>>>>> [0x7f76773f0ef3] ) 0-management: Lock for vol shared not held >>>>>>>>>>>> >>>>>>>>>>>> The heal info says this: >>>>>>>>>>>> >>>>>>>>>>>> # gluster volume heal shared info >>>>>>>>>>>> Brick 192.168.0.5:/data/exports/shared >>>>>>>>>>>> Number of entries: 0 >>>>>>>>>>>> >>>>>>>>>>>> Brick 192.168.0.6:/data/exports/shared >>>>>>>>>>>> Status: Transport endpoint is not connected >>>>>>>>>>>> >>>>>>>>>>>> Brick 192.168.0.7:/data/exports/shared >>>>>>>>>>>> Status: Transport endpoint is not connected >>>>>>>>>>>> >>>>>>>>>>>> Any idea whats up here? >>>>>>>>>>>> >>>>>>>>>>>> Pawan >>>>>>>>>>>> >>>>>>>>>>>> On Mon, May 22, 2017 at 9:42 PM, Atin Mukherjee < >>>>>>>>>>>> amukh...@redhat.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, May 22, 2017 at 9:05 PM, Pawan Alwandi < >>>>>>>>>>>>> pa...@platform.sh> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, May 22, 2017 at 8:36 PM, Atin Mukherjee < >>>>>>>>>>>>>> amukh...@redhat.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, May 22, 2017 at 7:51 PM, Atin Mukherjee < >>>>>>>>>>>>>>> amukh...@redhat.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sorry Pawan, I did miss the other part of the attachments. >>>>>>>>>>>>>>>> So looking from the glusterd.info file from all the hosts, >>>>>>>>>>>>>>>> it looks like host2 and host3 do not have the correct >>>>>>>>>>>>>>>> op-version. Can you >>>>>>>>>>>>>>>> please set the op-version as "operating-version=30702" in >>>>>>>>>>>>>>>> host2 and host3 >>>>>>>>>>>>>>>> and restart glusterd instance one by one on all the nodes? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please ensure that all the hosts are upgraded to the same >>>>>>>>>>>>>>> bits before doing this change. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Having to upgrade all 3 hosts to newer version before gluster >>>>>>>>>>>>>> could work successfully on any of them means application >>>>>>>>>>>>>> downtime. The >>>>>>>>>>>>>> applications running on these hosts are expected to be highly >>>>>>>>>>>>>> available. >>>>>>>>>>>>>> So with the way the things are right now, is an online upgrade >>>>>>>>>>>>>> possible? >>>>>>>>>>>>>> My upgrade steps are: (1) stop the applications (2) umount the >>>>>>>>>>>>>> gluster >>>>>>>>>>>>>> volume, and then (3) upgrade gluster one host at a time. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> One of the way to mitigate this is to first do an online >>>>>>>>>>>>> upgrade to glusterfs-3.7.9 (op-version:30707) given this bug was >>>>>>>>>>>>> introduced >>>>>>>>>>>>> in 3.7.10 and then come to 3.11. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Our goal is to get gluster upgraded to 3.11 from 3.6.9, and >>>>>>>>>>>>>> to make this an online upgrade we are okay to take two steps >>>>>>>>>>>>>> 3.6.9 -> 3.7 >>>>>>>>>>>>>> and then 3.7 to 3.11. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Apparently it looks like there is a bug which you have >>>>>>>>>>>>>>>> uncovered, during peer handshaking if one of the glusterd >>>>>>>>>>>>>>>> instance is >>>>>>>>>>>>>>>> running with old bits then during validating the handshake >>>>>>>>>>>>>>>> request there is >>>>>>>>>>>>>>>> a possibility that uuid received will be blank and the same >>>>>>>>>>>>>>>> was ignored >>>>>>>>>>>>>>>> however there was a patch http://review.gluster.org/13519 >>>>>>>>>>>>>>>> which had some additional changes which was always looking at >>>>>>>>>>>>>>>> this field >>>>>>>>>>>>>>>> and doing some extra checks which was causing the handshake to >>>>>>>>>>>>>>>> fail. For >>>>>>>>>>>>>>>> now, the above workaround should suffice. I'll be sending a >>>>>>>>>>>>>>>> patch pretty >>>>>>>>>>>>>>>> soon. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Posted a patch https://review.gluster.org/#/c/17358 . >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, May 22, 2017 at 11:35 AM, Pawan Alwandi < >>>>>>>>>>>>>>>> pa...@platform.sh> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hello Atin, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The tar's have the content of `/var/lib/glusterd` too for >>>>>>>>>>>>>>>>> all 3 nodes, please check again. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, May 22, 2017 at 11:32 AM, Atin Mukherjee < >>>>>>>>>>>>>>>>> amukh...@redhat.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Pawan, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I see you have provided the log files from the nodes, >>>>>>>>>>>>>>>>>> however it'd be really helpful if you can provide me the >>>>>>>>>>>>>>>>>> content of >>>>>>>>>>>>>>>>>> /var/lib/glusterd from all the nodes to get to the root >>>>>>>>>>>>>>>>>> cause of this issue. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, May 19, 2017 at 12:09 PM, Pawan Alwandi < >>>>>>>>>>>>>>>>>> pa...@platform.sh> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hello Atin, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks for continued support. I've attached requested >>>>>>>>>>>>>>>>>>> files from all 3 nodes. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> (I think we already verified the UUIDs to be correct, >>>>>>>>>>>>>>>>>>> anyway let us know if you find any more info in the logs) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Pawan >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Thu, May 18, 2017 at 11:45 PM, Atin Mukherjee < >>>>>>>>>>>>>>>>>>> amukh...@redhat.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Thu, 18 May 2017 at 23:40, Atin Mukherjee < >>>>>>>>>>>>>>>>>>>> amukh...@redhat.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Wed, 17 May 2017 at 12:47, Pawan Alwandi >>>>>>>>>>>>>>>>>>>>> <pa...@platform.sh> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hello Atin, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I realized that these >>>>>>>>>>>>>>>>>>>>>> http://gluster.readthedocs.io/en/latest/Upgrade-Guide/upgrade_to_3.10/ >>>>>>>>>>>>>>>>>>>>>> instructions only work for upgrades from 3.7, while we >>>>>>>>>>>>>>>>>>>>>> are running 3.6.2. >>>>>>>>>>>>>>>>>>>>>> Are there instructions/suggestion you have for us to >>>>>>>>>>>>>>>>>>>>>> upgrade from 3.6 >>>>>>>>>>>>>>>>>>>>>> version? >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I believe upgrade from 3.6 to 3.7 and then to 3.10 >>>>>>>>>>>>>>>>>>>>>> would work, but I see similar errors reported when I >>>>>>>>>>>>>>>>>>>>>> upgraded to 3.7 too. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> For what its worth, I was able to set the op-version >>>>>>>>>>>>>>>>>>>>>> (gluster v set all cluster.op-version 30702) but that >>>>>>>>>>>>>>>>>>>>>> doesn't seem to help. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:33.700014] I [MSGID: 100030] >>>>>>>>>>>>>>>>>>>>>> [glusterfsd.c:2338:main] 0-/usr/sbin/glusterd: Started >>>>>>>>>>>>>>>>>>>>>> running >>>>>>>>>>>>>>>>>>>>>> /usr/sbin/glusterd version 3.7.20 (args: >>>>>>>>>>>>>>>>>>>>>> /usr/sbin/glusterd -p >>>>>>>>>>>>>>>>>>>>>> /var/run/glusterd.pid) >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:33.703808] I [MSGID: 106478] >>>>>>>>>>>>>>>>>>>>>> [glusterd.c:1383:init] 0-management: Maximum allowed >>>>>>>>>>>>>>>>>>>>>> open file descriptors >>>>>>>>>>>>>>>>>>>>>> set to 65536 >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:33.703836] I [MSGID: 106479] >>>>>>>>>>>>>>>>>>>>>> [glusterd.c:1432:init] 0-management: Using >>>>>>>>>>>>>>>>>>>>>> /var/lib/glusterd as working >>>>>>>>>>>>>>>>>>>>>> directory >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:33.708866] W [MSGID: 103071] >>>>>>>>>>>>>>>>>>>>>> [rdma.c:4594:__gf_rdma_ctx_create] 0-rpc-transport/rdma: >>>>>>>>>>>>>>>>>>>>>> rdma_cm event >>>>>>>>>>>>>>>>>>>>>> channel creation failed [No such device] >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:33.709011] W [MSGID: 103055] >>>>>>>>>>>>>>>>>>>>>> [rdma.c:4901:init] 0-rdma.management: Failed to >>>>>>>>>>>>>>>>>>>>>> initialize IB Device >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:33.709033] W >>>>>>>>>>>>>>>>>>>>>> [rpc-transport.c:359:rpc_transport_load] >>>>>>>>>>>>>>>>>>>>>> 0-rpc-transport: 'rdma' >>>>>>>>>>>>>>>>>>>>>> initialization failed >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:33.709088] W >>>>>>>>>>>>>>>>>>>>>> [rpcsvc.c:1642:rpcsvc_create_listener] 0-rpc-service: >>>>>>>>>>>>>>>>>>>>>> cannot create >>>>>>>>>>>>>>>>>>>>>> listener, initing the transport failed >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:33.709105] E [MSGID: 106243] >>>>>>>>>>>>>>>>>>>>>> [glusterd.c:1656:init] 0-management: creation of 1 >>>>>>>>>>>>>>>>>>>>>> listeners failed, >>>>>>>>>>>>>>>>>>>>>> continuing with succeeded transport >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.480043] I [MSGID: 106513] >>>>>>>>>>>>>>>>>>>>>> [glusterd-store.c:2068:glusterd_restore_op_version] >>>>>>>>>>>>>>>>>>>>>> 0-glusterd: retrieved >>>>>>>>>>>>>>>>>>>>>> op-version: 30600 >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.605779] I [MSGID: 106498] >>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:3640:glusterd_friend_add_from_peerinfo] >>>>>>>>>>>>>>>>>>>>>> 0-management: >>>>>>>>>>>>>>>>>>>>>> connect returned 0 >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.607059] I >>>>>>>>>>>>>>>>>>>>>> [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: >>>>>>>>>>>>>>>>>>>>>> setting >>>>>>>>>>>>>>>>>>>>>> frame-timeout to 600 >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.607670] I >>>>>>>>>>>>>>>>>>>>>> [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: >>>>>>>>>>>>>>>>>>>>>> setting >>>>>>>>>>>>>>>>>>>>>> frame-timeout to 600 >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.607025] I [MSGID: 106498] >>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:3640:glusterd_friend_add_from_peerinfo] >>>>>>>>>>>>>>>>>>>>>> 0-management: >>>>>>>>>>>>>>>>>>>>>> connect returned 0 >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.608125] I [MSGID: 106544] >>>>>>>>>>>>>>>>>>>>>> [glusterd.c:159:glusterd_uuid_init] 0-management: >>>>>>>>>>>>>>>>>>>>>> retrieved UUID: >>>>>>>>>>>>>>>>>>>>>> 7f2a6e11-2a53-4ab4-9ceb-8be6a9f2d073 >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Final graph: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> +------------------------------------------------------------------------------+ >>>>>>>>>>>>>>>>>>>>>> 1: volume management >>>>>>>>>>>>>>>>>>>>>> 2: type mgmt/glusterd >>>>>>>>>>>>>>>>>>>>>> 3: option rpc-auth.auth-glusterfs on >>>>>>>>>>>>>>>>>>>>>> 4: option rpc-auth.auth-unix on >>>>>>>>>>>>>>>>>>>>>> 5: option rpc-auth.auth-null on >>>>>>>>>>>>>>>>>>>>>> 6: option rpc-auth-allow-insecure on >>>>>>>>>>>>>>>>>>>>>> 7: option transport.socket.listen-backlog 128 >>>>>>>>>>>>>>>>>>>>>> 8: option event-threads 1 >>>>>>>>>>>>>>>>>>>>>> 9: option ping-timeout 0 >>>>>>>>>>>>>>>>>>>>>> 10: option transport.socket.read-fail-log off >>>>>>>>>>>>>>>>>>>>>> 11: option transport.socket.keepalive-interval 2 >>>>>>>>>>>>>>>>>>>>>> 12: option transport.socket.keepalive-time 10 >>>>>>>>>>>>>>>>>>>>>> 13: option transport-type rdma >>>>>>>>>>>>>>>>>>>>>> 14: option working-directory /var/lib/glusterd >>>>>>>>>>>>>>>>>>>>>> 15: end-volume >>>>>>>>>>>>>>>>>>>>>> 16: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> +------------------------------------------------------------------------------+ >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.609868] I [MSGID: 101190] >>>>>>>>>>>>>>>>>>>>>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: >>>>>>>>>>>>>>>>>>>>>> Started thread >>>>>>>>>>>>>>>>>>>>>> with index 1 >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.610839] W >>>>>>>>>>>>>>>>>>>>>> [socket.c:596:__socket_rwv] 0-management: readv on >>>>>>>>>>>>>>>>>>>>>> 192.168.0.7:24007 failed (No data available) >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.611907] E >>>>>>>>>>>>>>>>>>>>>> [rpc-clnt.c:370:saved_frames_unwind] (--> >>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fd6c2d70bb3] >>>>>>>>>>>>>>>>>>>>>> (--> >>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fd6c2b3a2df] >>>>>>>>>>>>>>>>>>>>>> (--> >>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd6c2b3a3fe] >>>>>>>>>>>>>>>>>>>>>> (--> >>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7fd6c2b3ba39] >>>>>>>>>>>>>>>>>>>>>> (--> >>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7fd6c2b3c380] >>>>>>>>>>>>>>>>>>>>>> ))))) 0-management: forced unwinding frame >>>>>>>>>>>>>>>>>>>>>> type(GLUSTERD-DUMP) op(DUMP(1)) >>>>>>>>>>>>>>>>>>>>>> called at 2017-05-17 06:48:35.609965 (xid=0x1) >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.611928] E [MSGID: 106167] >>>>>>>>>>>>>>>>>>>>>> [glusterd-handshake.c:2091:__glusterd_peer_dump_version_cbk] >>>>>>>>>>>>>>>>>>>>>> 0-management: >>>>>>>>>>>>>>>>>>>>>> Error through RPC layer, retry again later >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.611944] I [MSGID: 106004] >>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:5201:__glusterd_peer_rpc_notify] >>>>>>>>>>>>>>>>>>>>>> 0-management: Peer >>>>>>>>>>>>>>>>>>>>>> <192.168.0.7> (<5ec54b4f-f60c-48c6-9e55-95f2bb58f633>), >>>>>>>>>>>>>>>>>>>>>> in state <Peer in >>>>>>>>>>>>>>>>>>>>>> Cluster>, has disconnected from glusterd. >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.612024] W >>>>>>>>>>>>>>>>>>>>>> [glusterd-locks.c:681:glusterd_mgmt_v3_unlock] >>>>>>>>>>>>>>>>>>>>>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.20/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x4b) >>>>>>>>>>>>>>>>>>>>>> [0x7fd6bdc4912b] >>>>>>>>>>>>>>>>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.20/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x160) >>>>>>>>>>>>>>>>>>>>>> [0x7fd6bdc52dd0] >>>>>>>>>>>>>>>>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.20/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x4c3) >>>>>>>>>>>>>>>>>>>>>> [0x7fd6bdcef1b3] ) 0-management: Lock for vol shared not >>>>>>>>>>>>>>>>>>>>>> held >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.612039] W [MSGID: 106118] >>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:5223:__glusterd_peer_rpc_notify] >>>>>>>>>>>>>>>>>>>>>> 0-management: Lock not >>>>>>>>>>>>>>>>>>>>>> released for shared >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.612079] W >>>>>>>>>>>>>>>>>>>>>> [socket.c:596:__socket_rwv] 0-management: readv on >>>>>>>>>>>>>>>>>>>>>> 192.168.0.6:24007 failed (No data available) >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.612179] E >>>>>>>>>>>>>>>>>>>>>> [rpc-clnt.c:370:saved_frames_unwind] (--> >>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fd6c2d70bb3] >>>>>>>>>>>>>>>>>>>>>> (--> >>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7fd6c2b3a2df] >>>>>>>>>>>>>>>>>>>>>> (--> >>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd6c2b3a3fe] >>>>>>>>>>>>>>>>>>>>>> (--> >>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7fd6c2b3ba39] >>>>>>>>>>>>>>>>>>>>>> (--> >>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7fd6c2b3c380] >>>>>>>>>>>>>>>>>>>>>> ))))) 0-management: forced unwinding frame >>>>>>>>>>>>>>>>>>>>>> type(GLUSTERD-DUMP) op(DUMP(1)) >>>>>>>>>>>>>>>>>>>>>> called at 2017-05-17 06:48:35.610007 (xid=0x1) >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.612197] E [MSGID: 106167] >>>>>>>>>>>>>>>>>>>>>> [glusterd-handshake.c:2091:__glusterd_peer_dump_version_cbk] >>>>>>>>>>>>>>>>>>>>>> 0-management: >>>>>>>>>>>>>>>>>>>>>> Error through RPC layer, retry again later >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.612211] I [MSGID: 106004] >>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:5201:__glusterd_peer_rpc_notify] >>>>>>>>>>>>>>>>>>>>>> 0-management: Peer >>>>>>>>>>>>>>>>>>>>>> <192.168.0.6> (<83e9a0b9-6bd5-483b-8516-d8928805ed95>), >>>>>>>>>>>>>>>>>>>>>> in state <Peer in >>>>>>>>>>>>>>>>>>>>>> Cluster>, has disconnected from glusterd. >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.612292] W >>>>>>>>>>>>>>>>>>>>>> [glusterd-locks.c:681:glusterd_mgmt_v3_unlock] >>>>>>>>>>>>>>>>>>>>>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.20/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x4b) >>>>>>>>>>>>>>>>>>>>>> [0x7fd6bdc4912b] >>>>>>>>>>>>>>>>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.20/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x160) >>>>>>>>>>>>>>>>>>>>>> [0x7fd6bdc52dd0] >>>>>>>>>>>>>>>>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.20/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x4c3) >>>>>>>>>>>>>>>>>>>>>> [0x7fd6bdcef1b3] ) 0-management: Lock for vol shared not >>>>>>>>>>>>>>>>>>>>>> held >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.613432] W [MSGID: 106118] >>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:5223:__glusterd_peer_rpc_notify] >>>>>>>>>>>>>>>>>>>>>> 0-management: Lock not >>>>>>>>>>>>>>>>>>>>>> released for shared >>>>>>>>>>>>>>>>>>>>>> [2017-05-17 06:48:35.614317] E [MSGID: 106170] >>>>>>>>>>>>>>>>>>>>>> [glusterd-handshake.c:1051:gd_validate_mgmt_hndsk_req] >>>>>>>>>>>>>>>>>>>>>> 0-management: >>>>>>>>>>>>>>>>>>>>>> Request from peer 192.168.0.6:991 has an entry in >>>>>>>>>>>>>>>>>>>>>> peerinfo, but uuid does not match >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Apologies for delay. My initial suspect was correct. >>>>>>>>>>>>>>>>>>>>> You have an incorrect UUID in the peer file which is >>>>>>>>>>>>>>>>>>>>> causing this. Can you >>>>>>>>>>>>>>>>>>>>> please provide me the >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Clicked the send button accidentally! >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Can you please send me the content of /var/lib/glusterd >>>>>>>>>>>>>>>>>>>> & glusterd log from all the nodes? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Mon, May 15, 2017 at 10:31 PM, Atin Mukherjee < >>>>>>>>>>>>>>>>>>>>>> amukh...@redhat.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Mon, 15 May 2017 at 11:58, Pawan Alwandi >>>>>>>>>>>>>>>>>>>>>>> <pa...@platform.sh> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi Atin, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I see below error. Do I require gluster to be >>>>>>>>>>>>>>>>>>>>>>>> upgraded on all 3 hosts for this to work? Right now I >>>>>>>>>>>>>>>>>>>>>>>> have host 1 running >>>>>>>>>>>>>>>>>>>>>>>> 3.10.1 and host 2 & 3 running 3.6.2 >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> # gluster v set all cluster.op-version 31001 >>>>>>>>>>>>>>>>>>>>>>>> volume set: failed: Required op_version (31001) is >>>>>>>>>>>>>>>>>>>>>>>> not supported >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Yes you should given 3.6 version is EOLed. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Mon, May 15, 2017 at 3:32 AM, Atin Mukherjee < >>>>>>>>>>>>>>>>>>>>>>>> amukh...@redhat.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Sun, 14 May 2017 at 21:43, Atin Mukherjee < >>>>>>>>>>>>>>>>>>>>>>>>> amukh...@redhat.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Allright, I see that you haven't bumped up the >>>>>>>>>>>>>>>>>>>>>>>>>> op-version. Can you please execute: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> gluster v set all cluster.op-version 30101 and >>>>>>>>>>>>>>>>>>>>>>>>>> then restart glusterd on all the nodes and check the >>>>>>>>>>>>>>>>>>>>>>>>>> brick status? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> s/30101/31001 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 14, 2017 at 8:55 PM, Pawan Alwandi < >>>>>>>>>>>>>>>>>>>>>>>>>> pa...@platform.sh> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Atin, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking at this. Below is the output >>>>>>>>>>>>>>>>>>>>>>>>>>> you requested for. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Again, I'm seeing those errors after upgrading >>>>>>>>>>>>>>>>>>>>>>>>>>> gluster on host 1. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Host 1 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> # cat /var/lib/glusterd/glusterd.info >>>>>>>>>>>>>>>>>>>>>>>>>>> UUID=7f2a6e11-2a53-4ab4-9ceb-8be6a9f2d073 >>>>>>>>>>>>>>>>>>>>>>>>>>> operating-version=30600 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> # cat /var/lib/glusterd/peers/* >>>>>>>>>>>>>>>>>>>>>>>>>>> uuid=5ec54b4f-f60c-48c6-9e55-95f2bb58f633 >>>>>>>>>>>>>>>>>>>>>>>>>>> state=3 >>>>>>>>>>>>>>>>>>>>>>>>>>> hostname1=192.168.0.7 >>>>>>>>>>>>>>>>>>>>>>>>>>> uuid=83e9a0b9-6bd5-483b-8516-d8928805ed95 >>>>>>>>>>>>>>>>>>>>>>>>>>> state=3 >>>>>>>>>>>>>>>>>>>>>>>>>>> hostname1=192.168.0.6 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> # gluster --version >>>>>>>>>>>>>>>>>>>>>>>>>>> glusterfs 3.10.1 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Host 2 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> # cat /var/lib/glusterd/glusterd.info >>>>>>>>>>>>>>>>>>>>>>>>>>> UUID=83e9a0b9-6bd5-483b-8516-d8928805ed95 >>>>>>>>>>>>>>>>>>>>>>>>>>> operating-version=30600 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> # cat /var/lib/glusterd/peers/* >>>>>>>>>>>>>>>>>>>>>>>>>>> uuid=5ec54b4f-f60c-48c6-9e55-95f2bb58f633 >>>>>>>>>>>>>>>>>>>>>>>>>>> state=3 >>>>>>>>>>>>>>>>>>>>>>>>>>> hostname1=192.168.0.7 >>>>>>>>>>>>>>>>>>>>>>>>>>> uuid=7f2a6e11-2a53-4ab4-9ceb-8be6a9f2d073 >>>>>>>>>>>>>>>>>>>>>>>>>>> state=3 >>>>>>>>>>>>>>>>>>>>>>>>>>> hostname1=192.168.0.5 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> # gluster --version >>>>>>>>>>>>>>>>>>>>>>>>>>> glusterfs 3.6.2 built on Jan 21 2015 14:23:44 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Host 3 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> # cat /var/lib/glusterd/glusterd.info >>>>>>>>>>>>>>>>>>>>>>>>>>> UUID=5ec54b4f-f60c-48c6-9e55-95f2bb58f633 >>>>>>>>>>>>>>>>>>>>>>>>>>> operating-version=30600 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> # cat /var/lib/glusterd/peers/* >>>>>>>>>>>>>>>>>>>>>>>>>>> uuid=7f2a6e11-2a53-4ab4-9ceb-8be6a9f2d073 >>>>>>>>>>>>>>>>>>>>>>>>>>> state=3 >>>>>>>>>>>>>>>>>>>>>>>>>>> hostname1=192.168.0.5 >>>>>>>>>>>>>>>>>>>>>>>>>>> uuid=83e9a0b9-6bd5-483b-8516-d8928805ed95 >>>>>>>>>>>>>>>>>>>>>>>>>>> state=3 >>>>>>>>>>>>>>>>>>>>>>>>>>> hostname1=192.168.0.6 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> # gluster --version >>>>>>>>>>>>>>>>>>>>>>>>>>> glusterfs 3.6.2 built on Jan 21 2015 14:23:44 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, May 13, 2017 at 6:28 PM, Atin Mukherjee >>>>>>>>>>>>>>>>>>>>>>>>>>> <amukh...@redhat.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I have already asked for the following earlier: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please provide output of following from >>>>>>>>>>>>>>>>>>>>>>>>>>>> all the nodes: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat /var/lib/glusterd/glusterd.info >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat /var/lib/glusterd/peers/* >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, 13 May 2017 at 12:22, Pawan Alwandi >>>>>>>>>>>>>>>>>>>>>>>>>>>> <pa...@platform.sh> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello folks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does anyone have any idea whats going on here? >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Pawan >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 10, 2017 at 5:02 PM, Pawan Alwandi >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pa...@platform.sh> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm trying to upgrade gluster from 3.6.2 to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.10.1 but don't see the glusterfsd and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> glusterfs processes coming up. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://gluster.readthedocs.io/en/latest/Upgrade-Guide/upgrade_to_3.10/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is the process that I'm trying to follow. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is a 3 node server setup with a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> replicated volume having replica count of 3. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Logs below: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:03.507959] I [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 100030] [glusterfsd.c:2460:main] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-/usr/sbin/glusterd: Started running >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /usr/sbin/glusterd version 3.10.1 (args: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /usr/sbin/glusterd -p >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /var/run/glusterd.pid) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:03.512827] I [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106478] [glusterd.c:1449:init] 0-management: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Maximum allowed open file >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> descriptors set to 65536 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:03.512855] I [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106479] [glusterd.c:1496:init] 0-management: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Using /var/lib/glusterd as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> working directory >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:03.520426] W [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 103071] [rdma.c:4590:__gf_rdma_ctx_create] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-rpc-transport/rdma: rdma_cm >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> event channel creation failed [No such device] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:03.520452] W [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 103055] [rdma.c:4897:init] 0-rdma.management: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Failed to initialize IB Device >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:03.520465] W >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [rpc-transport.c:350:rpc_transport_load] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-rpc-transport: 'rdma' >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initialization failed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:03.520518] W >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [rpcsvc.c:1661:rpcsvc_create_listener] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-rpc-service: cannot create >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> listener, initing the transport failed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:03.520534] E [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106243] [glusterd.c:1720:init] 0-management: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creation of 1 listeners >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed, continuing with succeeded transport >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:04.931764] I [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106513] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [glusterd-store.c:2197:glusterd_restore_op_version] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-glusterd: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> retrieved op-version: 30600 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:04.964354] I [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106544] [glusterd.c:158:glusterd_uuid_init] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-management: retrieved UUID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7f2a6e11-2a53-4ab4-9ceb-8be6a9f2d073 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:04.993944] I [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106498] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-management: connect returned 0 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:04.995864] I [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106498] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-management: connect returned 0 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:04.995879] W [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106062] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:3466:glusterd_transport_inet_options_build] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-glusterd: Failed to get tcp-user-timeout >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:04.995903] I >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [rpc-clnt.c:1059:rpc_clnt_connection_init] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-management: setting >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> frame-timeout to 600 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:04.996325] I >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [rpc-clnt.c:1059:rpc_clnt_connection_init] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-management: setting >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> frame-timeout to 600 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Final graph: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +------------------------------------------------------------------------------+ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: volume management >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: type mgmt/glusterd >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: option rpc-auth.auth-glusterfs on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: option rpc-auth.auth-unix on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5: option rpc-auth.auth-null on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6: option rpc-auth-allow-insecure on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7: option >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> transport.socket.listen-backlog 128 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8: option event-threads 1 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9: option ping-timeout 0 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 10: option >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> transport.socket.read-fail-log off >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 11: option >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> transport.socket.keepalive-interval 2 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 12: option >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> transport.socket.keepalive-time 10 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 13: option transport-type rdma >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 14: option working-directory >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /var/lib/glusterd >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 15: end-volume >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +------------------------------------------------------------------------------+ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:04.996310] W [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106062] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:3466:glusterd_transport_inet_options_build] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-glusterd: Failed to get tcp-user-timeout >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:05.000461] I [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 101190] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [event-epoll.c:629:event_dispatch_epoll_worker] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-epoll: Started >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread with index 1 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:05.001493] W >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [socket.c:593:__socket_rwv] 0-management: readv >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 192.168.0.7:24007 failed (No data available) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:05.001513] I [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106004] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-management: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Peer <192.168.0.7> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (<5ec54b4f-f60c-48c6-9e55-95f2bb58f633>), in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> state <Peer >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in Cluster>, h >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as disconnected from glusterd. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:05.001677] W >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x20559) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [0x7f0bf9d74559] -->/usr/lib/x86_64-linux-gnu >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29cf0) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [0x7f0bf9d7dcf0] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd5ba3) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [0x7f0bf9e29ba3] ) 0-management: Lock for vol >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shared no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> t held >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:05.001696] W [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106118] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-management: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lock not released for shared >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:05.003099] E >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [rpc-clnt.c:365:saved_frames_unwind] (--> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13c)[0x7f0bfeeca73c] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(s >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> aved_frames_unwind+0x1cf)[0x7f0bfec904bf] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (--> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f0bfec905de] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (--> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 91)[0x7f0bfec91c21] (--> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x290)[0x7f0bfec92710] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ))))) 0-management: forced unwinding frame >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type(GLUSTERD-DUMP) op(DUMP(1)) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> called at 2017-05-10 09:0 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:05.000627 (xid=0x1) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:05.003129] E [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106167] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [glusterd-handshake.c:2181:__glusterd_peer_dump_version_cbk] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-management: Error through RPC layer, retry >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> again later >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:05.003251] W >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [socket.c:593:__socket_rwv] 0-management: readv >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 192.168.0.6:24007 failed (No data available) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:05.003267] I [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106004] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-management: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Peer <192.168.0.6> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (<83e9a0b9-6bd5-483b-8516-d8928805ed95>), in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> state <Peer >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in Cluster>, h >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as disconnected from glusterd. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:05.003318] W >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x20559) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [0x7f0bf9d74559] -->/usr/lib/x86_64-linux-gnu >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29cf0) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [0x7f0bf9d7dcf0] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd5ba3) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [0x7f0bf9e29ba3] ) 0-management: Lock for vol >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shared no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> t held >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:05.003329] W [MSGID: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 106118] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0-management: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lock not released for shared >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2017-05-10 09:07:05.003457] E >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [rpc-clnt.c:365:saved_frames_unwind] (--> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13c)[0x7f0bfeeca73c] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(s >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> aved_frames_unwind+0x1cf)[0x7f0bfec904bf] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (--> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f0bfec905de] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (--> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 91)[0x7f0bfec91c21] (--> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x290)[0x7f0bfec92710] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ))))) 0-management: forced unwinding frame >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type(GLUSTERD-DUMP) op(DUMP(1)) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> called at 2017-05-10 09:0 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:05.001407 (xid=0x1) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are a bunch of errors reported but I'm >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not sure which is signal and which ones are >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> noise. Does anyone have any >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idea whats going on here? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Pawan >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Gluster-users@gluster.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>> - Atin (atinm) >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>> - Atin (atinm) >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> - Atin (atinm) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> - Atin (atinm) >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> - Atin (atinm) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> - Atin (atinm) >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> -- >>>> - Atin (atinm) >>>> >>> >>> >> > -- - Atin (atinm)
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users