Re: [Gluster-devel] [Gluster-users] getting "Transport endpoint is not connected" in glusterfs mount log file.
Abhishek, Both Rafi and I tried to look at the logs but the file seems to be corrupted. I was saying that there is connection problem because the following log appeard in between lot of connection failures in the logs you posted. Are you on IRC #gluster-dev? [2016-10-31 04:06:03.628539] I [MSGID: 108019] [afr-transaction.c:1224:afr_post_blocking_inodelk_cbk] 0-c_glusterfs-replicate-0: Blocking inodelks failed. On Fri, Nov 11, 2016 at 1:05 PM, ABHISHEK PALIWALwrote: > Hi Pranith, > > Could you please tell tell me the logs showing that the mount is not > available to connect to both the bricks. > > On Fri, Nov 11, 2016 at 12:05 PM, Pranith Kumar Karampuri < > pkara...@redhat.com> wrote: > >> As per the logs, the mount is not able to connect to both the bricks. Are >> the connections fine? >> >> On Fri, Nov 11, 2016 at 10:20 AM, ABHISHEK PALIWAL < >> abhishpali...@gmail.com> wrote: >> >>> Hi, >>> >>> Its an urgent case. >>> >>> Atleast provide your views on this >>> >>> On Wed, Nov 9, 2016 at 11:08 AM, ABHISHEK PALIWAL < >>> abhishpali...@gmail.com> wrote: >>> Hi, We could see that sync is getting failed to sync the GlusterFS bricks due to error trace "Transport endpoint is not connected " [2016-10-31 04:06:03.627395] E [MSGID: 114031] [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected] [2016-10-31 04:06:03.628381] I [socket.c:3308:socket_submit_request] 0-c_glusterfs-client-9: not connected (priv->connected = 0) [2016-10-31 04:06:03.628432] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f5f Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (c_glusterfs-client-9) [2016-10-31 04:06:03.628466] E [MSGID: 114031] [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected] [2016-10-31 04:06:03.628475] I [MSGID: 108019] [afr-lk-common.c:1086:afr_lock_blocking] 0-c_glusterfs-replicate-0: unable to lock on even one child [2016-10-31 04:06:03.628539] I [MSGID: 108019] [afr-transaction.c:1224:afr_post_blocking_inodelk_cbk] 0-c_glusterfs-replicate-0: Blocking inodelks failed. [2016-10-31 04:06:03.628630] W [fuse-bridge.c:1282:fuse_err_cbk] 0-glusterfs-fuse: 20790: FLUSH() ERR => -1 (Transport endpoint is not connected) [2016-10-31 04:06:03.629149] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] ) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-10-31 04:06:03.624346 (xid=0x7f5a) [2016-10-31 04:06:03.629183] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-c_glusterfs-client-9: changing port to 49391 (from 0) [2016-10-31 04:06:03.629210] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not connected] Could you please tell us the reason why we are getting these trace and how to resolve this. Logs are attached here please share your analysis. Thanks in advanced -- Regards Abhishek Paliwal >>> >>> >>> >>> -- >>> >>> >>> >>> >>> Regards >>> Abhishek Paliwal >>> >>> ___ >>> Gluster-users mailing list >>> gluster-us...@gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >> >> >> >> -- >> Pranith >> > > > > -- > > > > > Regards > Abhishek Paliwal > -- Pranith ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] getting "Transport endpoint is not connected" in glusterfs mount log file.
Hi Pranith, Could you please tell tell me the logs showing that the mount is not available to connect to both the bricks. On Fri, Nov 11, 2016 at 12:05 PM, Pranith Kumar Karampuri < pkara...@redhat.com> wrote: > As per the logs, the mount is not able to connect to both the bricks. Are > the connections fine? > > On Fri, Nov 11, 2016 at 10:20 AM, ABHISHEK PALIWAL < > abhishpali...@gmail.com> wrote: > >> Hi, >> >> Its an urgent case. >> >> Atleast provide your views on this >> >> On Wed, Nov 9, 2016 at 11:08 AM, ABHISHEK PALIWAL < >> abhishpali...@gmail.com> wrote: >> >>> Hi, >>> >>> We could see that sync is getting failed to sync the GlusterFS bricks >>> due to error trace "Transport endpoint is not connected " >>> >>> [2016-10-31 04:06:03.627395] E [MSGID: 114031] >>> [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: >>> remote operation failed [Transport endpoint is not connected] >>> [2016-10-31 04:06:03.628381] I [socket.c:3308:socket_submit_request] >>> 0-c_glusterfs-client-9: not connected (priv->connected = 0) >>> [2016-10-31 04:06:03.628432] W [rpc-clnt.c:1586:rpc_clnt_submit] >>> 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f5f Program: >>> GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport >>> (c_glusterfs-client-9) >>> [2016-10-31 04:06:03.628466] E [MSGID: 114031] >>> [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: >>> remote operation failed [Transport endpoint is not connected] >>> [2016-10-31 04:06:03.628475] I [MSGID: 108019] >>> [afr-lk-common.c:1086:afr_lock_blocking] 0-c_glusterfs-replicate-0: >>> unable to lock on even one child >>> [2016-10-31 04:06:03.628539] I [MSGID: 108019] >>> [afr-transaction.c:1224:afr_post_blocking_inodelk_cbk] >>> 0-c_glusterfs-replicate-0: Blocking inodelks failed. >>> [2016-10-31 04:06:03.628630] W [fuse-bridge.c:1282:fuse_err_cbk] >>> 0-glusterfs-fuse: 20790: FLUSH() ERR => -1 (Transport endpoint is not >>> connected) >>> [2016-10-31 04:06:03.629149] E [rpc-clnt.c:362:saved_frames_unwind] >>> (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] >>> (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] >>> (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] >>> (--> >>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] >>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] >>> ) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) >>> op(LOOKUP(27)) called at 2016-10-31 04:06:03.624346 (xid=0x7f5a) >>> [2016-10-31 04:06:03.629183] I [rpc-clnt.c:1847:rpc_clnt_reconfig] >>> 0-c_glusterfs-client-9: changing port to 49391 (from 0) >>> [2016-10-31 04:06:03.629210] W [MSGID: 114031] >>> [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: >>> remote operation failed. Path: >>> /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews >>> (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not >>> connected] >>> >>> >>> Could you please tell us the reason why we are getting these trace and >>> how to resolve this. >>> >>> Logs are attached here please share your analysis. >>> >>> Thanks in advanced >>> >>> -- >>> Regards >>> Abhishek Paliwal >>> >> >> >> >> -- >> >> >> >> >> Regards >> Abhishek Paliwal >> >> ___ >> Gluster-users mailing list >> gluster-us...@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > > > -- > Pranith > -- Regards Abhishek Paliwal ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] getting "Transport endpoint is not connected" in glusterfs mount log file.
Hi Rafi KC, I have already attached all the logs in my first mail and I am getting these logs on 25th board. You can find the logs at logs/d/usr/002500_glusterfiles/varlog_glusterfs/brick/ //Abhishek On Fri, Nov 11, 2016 at 11:50 AM, Mohammed Rafi K Cwrote: > Hi Abhishek, > > Could you please see if you are bricks are healthy or not, may be you can > do a gluster volume status or you can look into the logs. If bricks are not > running can you please attach the bricks logs in /var/log/gluster/bricks/ . > > > Rafi KC > > On 11/11/2016 10:20 AM, ABHISHEK PALIWAL wrote: > > Hi, > > Its an urgent case. > > Atleast provide your views on this > > On Wed, Nov 9, 2016 at 11:08 AM, ABHISHEK PALIWAL < > abhishpali...@gmail.com> wrote: > >> Hi, >> >> We could see that sync is getting failed to sync the GlusterFS bricks due >> to error trace "Transport endpoint is not connected " >> >> [2016-10-31 04:06:03.627395] E [MSGID: 114031] >> [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: >> remote operation failed [Transport endpoint is not connected] >> [2016-10-31 04:06:03.628381] I [socket.c:3308:socket_submit_request] >> 0-c_glusterfs-client-9: not connected (priv->connected = 0) >> [2016-10-31 04:06:03.628432] W [rpc-clnt.c:1586:rpc_clnt_submit] >> 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f5f Program: >> GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport >> (c_glusterfs-client-9) >> [2016-10-31 04:06:03.628466] E [MSGID: 114031] >> [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: >> remote operation failed [Transport endpoint is not connected] >> [2016-10-31 04:06:03.628475] I [MSGID: 108019] >> [afr-lk-common.c:1086:afr_lock_blocking] 0-c_glusterfs-replicate-0: >> unable to lock on even one child >> [2016-10-31 04:06:03.628539] I [MSGID: 108019] >> [afr-transaction.c:1224:afr_post_blocking_inodelk_cbk] >> 0-c_glusterfs-replicate-0: Blocking inodelks failed. >> [2016-10-31 04:06:03.628630] W [fuse-bridge.c:1282:fuse_err_cbk] >> 0-glusterfs-fuse: 20790: FLUSH() ERR => -1 (Transport endpoint is not >> connected) >> [2016-10-31 04:06:03.629149] E [rpc-clnt.c:362:saved_frames_unwind] (--> >> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] >> (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] >> (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] >> (--> >> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] >> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] >> ) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) >> op(LOOKUP(27)) called at 2016-10-31 04:06:03.624346 (xid=0x7f5a) >> [2016-10-31 04:06:03.629183] I [rpc-clnt.c:1847:rpc_clnt_reconfig] >> 0-c_glusterfs-client-9: changing port to 49391 (from 0) >> [2016-10-31 04:06:03.629210] W [MSGID: 114031] >> [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: >> remote operation failed. Path: >> /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews >> (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not >> connected] >> >> >> Could you please tell us the reason why we are getting these trace and >> how to resolve this. >> >> Logs are attached here please share your analysis. >> >> Thanks in advanced >> >> -- >> Regards >> Abhishek Paliwal >> > > > > -- > > > > > Regards > Abhishek Paliwal > > > ___ > Gluster-users mailing > listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users > > > -- Regards Abhishek Paliwal ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] getting "Transport endpoint is not connected" in glusterfs mount log file.
Hi Abhishek, Could you please see if you are bricks are healthy or not, may be you can do a gluster volume status or you can look into the logs. If bricks are not running can you please attach the bricks logs in /var/log/gluster/bricks/ . Rafi KC On 11/11/2016 10:20 AM, ABHISHEK PALIWAL wrote: > Hi, > > Its an urgent case. > > Atleast provide your views on this > > On Wed, Nov 9, 2016 at 11:08 AM, ABHISHEK PALIWAL >> wrote: > > Hi, > > We could see that sync is getting failed to sync the GlusterFS > bricks due to error trace "Transport endpoint is not connected " > > [2016-10-31 04:06:03.627395] E [MSGID: 114031] > [client-rpc-fops.c:1673:client3_3_finodelk_cbk] > 0-c_glusterfs-client-9: remote operation failed [Transport > endpoint is not connected] > [2016-10-31 04:06:03.628381] I > [socket.c:3308:socket_submit_request] 0-c_glusterfs-client-9: not > connected (priv->connected = 0) > [2016-10-31 04:06:03.628432] W [rpc-clnt.c:1586:rpc_clnt_submit] > 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f5f > Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport > (c_glusterfs-client-9) > [2016-10-31 04:06:03.628466] E [MSGID: 114031] > [client-rpc-fops.c:1673:client3_3_finodelk_cbk] > 0-c_glusterfs-client-9: remote operation failed [Transport > endpoint is not connected] > [2016-10-31 04:06:03.628475] I [MSGID: 108019] > [afr-lk-common.c:1086:afr_lock_blocking] > 0-c_glusterfs-replicate-0: unable to lock on even one child > [2016-10-31 04:06:03.628539] I [MSGID: 108019] > [afr-transaction.c:1224:afr_post_blocking_inodelk_cbk] > 0-c_glusterfs-replicate-0: Blocking inodelks failed. > [2016-10-31 04:06:03.628630] W [fuse-bridge.c:1282:fuse_err_cbk] > 0-glusterfs-fuse: 20790: FLUSH() ERR => -1 (Transport endpoint is > not connected) > [2016-10-31 04:06:03.629149] E > [rpc-clnt.c:362:saved_frames_unwind] (--> > /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] > (--> > /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] > (--> > /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] > (--> > > /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] > (--> > /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] > ) 0-c_glusterfs-client-9: forced unwinding frame > type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-10-31 > 04:06:03.624346 (xid=0x7f5a) > [2016-10-31 04:06:03.629183] I [rpc-clnt.c:1847:rpc_clnt_reconfig] > 0-c_glusterfs-client-9: changing port to 49391 (from 0) > [2016-10-31 04:06:03.629210] W [MSGID: 114031] > [client-rpc-fops.c:2971:client3_3_lookup_cbk] > 0-c_glusterfs-client-9: remote operation failed. Path: > /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews > (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not > connected] > > > Could you please tell us the reason why we are getting these trace > and how to resolve this. > > Logs are attached here please share your analysis. > > Thanks in advanced > > -- > Regards > Abhishek Paliwal > > > > > -- > > > > > Regards > Abhishek Paliwal > > > ___ > Gluster-users mailing list > gluster-us...@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] getting "Transport endpoint is not connected" in glusterfs mount log file.
On 11-Nov-2016 11:50, "Mohammed Rafi K C"wrote: > > Hi Abhishek, > > Could you please see if you are bricks are healthy or not, may be you can do a gluster volume status or you can look into the logs. If bricks are not running can you please attach the bricks logs in /var/log/gluster/bricks/ . > > > Rafi KC Also, please provide details of Gluster version and setup. > > > On 11/11/2016 10:20 AM, ABHISHEK PALIWAL wrote: >> >> Hi, >> >> Its an urgent case. >> >> Atleast provide your views on this >> >> On Wed, Nov 9, 2016 at 11:08 AM, ABHISHEK PALIWAL < abhishpali...@gmail.com> wrote: >>> >>> Hi, >>> >>> We could see that sync is getting failed to sync the GlusterFS bricks due to error trace "Transport endpoint is not connected " >>> >>> [2016-10-31 04:06:03.627395] E [MSGID: 114031] [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected] >>> [2016-10-31 04:06:03.628381] I [socket.c:3308:socket_submit_request] 0-c_glusterfs-client-9: not connected (priv->connected = 0) >>> [2016-10-31 04:06:03.628432] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f5f Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (c_glusterfs-client-9) >>> [2016-10-31 04:06:03.628466] E [MSGID: 114031] [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected] >>> [2016-10-31 04:06:03.628475] I [MSGID: 108019] [afr-lk-common.c:1086:afr_lock_blocking] 0-c_glusterfs-replicate-0: unable to lock on even one child >>> [2016-10-31 04:06:03.628539] I [MSGID: 108019] [afr-transaction.c:1224:afr_post_blocking_inodelk_cbk] 0-c_glusterfs-replicate-0: Blocking inodelks failed. >>> [2016-10-31 04:06:03.628630] W [fuse-bridge.c:1282:fuse_err_cbk] 0-glusterfs-fuse: 20790: FLUSH() ERR => -1 (Transport endpoint is not connected) >>> [2016-10-31 04:06:03.629149] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] ) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-10-31 04:06:03.624346 (xid=0x7f5a) >>> [2016-10-31 04:06:03.629183] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-c_glusterfs-client-9: changing port to 49391 (from 0) >>> [2016-10-31 04:06:03.629210] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not connected] >>> >>> >>> Could you please tell us the reason why we are getting these trace and how to resolve this. >>> >>> Logs are attached here please share your analysis. >>> >>> Thanks in advanced >>> >>> -- >>> Regards >>> Abhishek Paliwal >> >> >> >> >> -- >> >> >> >> >> Regards >> Abhishek Paliwal >> >> >> ___ Gluster-users mailing list gluster-us...@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users > > > > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel