Re: [Gluster-users] socket.c:2161:socket_connect_finish (Connection refused)
Olav, I suggest you use this workaround till the next release which is a few days away. happy to help :-) Pranith. On 06/11/2014 08:35 PM, Olav Peeters wrote: Thanks allot, Pranith! All seems back to normal again. Looking forward to the release of 3.5.1 ! Cheers, Olav On 11/06/14 09:30, Pranith Kumar Karampuri wrote: hey Just do "gluster volume start force" and things should be back to normal Pranith On 06/11/2014 12:56 PM, Olav Peeters wrote: Pranith, how could I move all data from the two problem bricks temporarily until the release of 3.5.1? Like this? # gluster volume replace-brick VOLNAME BRICK NEW-BRICK start Will this work if the bricks are offline? Or is there some other way to get the bricks back online manually? Would it help to do all fuse connections via NFS until after the fix? Cheers, Olav On 11/06/14 08:44, Olav Peeters wrote: OK, thanks for the info! Regards, Olav On 11/06/14 08:38, Pranith Kumar Karampuri wrote: On 06/11/2014 12:03 PM, Olav Peeters wrote: Thanks Pranith! I see this at the end of the log files of one of the problem bricks (the first two errors are repeated several times): [2014-06-10 09:55:28.354659] E [rpcsvc.c:1206:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x103c59, Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.sr_vol01-server) [2014-06-10 09:55:28.354683] E [server.c:190:server_submit_reply] (-->/usr/lib64/glusterfs/3.5.0/xlator/performance/io-threads.so(iot_finodelk_cbk+0xb9) [0x7f8c8e82f189] (-->/usr/lib64/glusterfs/3.5.0/xlator/debug/io-stats.so(io_stats_finodelk_cbk+0xed) [0x7f8c8e1f22ed] (-->/usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_finodelk_cbk+0xad) [0x7f8c8dfc555d]))) 0-: Reply submission failed pending frames: frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) ... ... frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-06-10 09:55:28configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.5.0 /lib64/libc.so.6(+0x329a0)[0x7f8c94aac9a0] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(grant_blocked_inode_locks+0xc1)[0x7f8c8ea54061] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(pl_inodelk_client_cleanup+0x249)[0x7f8c8ea54569] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(+0x6f0a)[0x7f8c8ea49f0a] /usr/lib64/libglusterfs.so.0(gf_client_disconnect+0x5d)[0x7f8c964d701d] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_connection_cleanup+0x458)[0x7f8c8dfbda48] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_rpc_notify+0x183)[0x7f8c8dfb9713] /usr/lib64/libgfrpc.so.0(rpcsvc_handle_disconnect+0x105)[0x7f8c96261d35] /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x1a0)[0x7f8c96263880] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f8c96264f98] /usr/lib64/glusterfs/3.5.0/rpc-transport/socket.so(+0xa9a1)[0x7f8c914c39a1] /usr/lib64/libglusterfs.so.0(+0x672f7)[0x7f8c964d92f7] /usr/sbin/glusterfsd(main+0x564)[0x4075e4] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f8c94a98d1d] /usr/sbin/glusterfsd[0x404679] - Again no info to be found online about the error. Any idea? This is because of bug 1089470 which is fixed for 3.5.1. Which will be releasing shortly. Pranith Olav On 11/06/14 04:42, Pranith Kumar Karampuri wrote: Olav, Check logs of the bricks to see why the bricks went down. Pranith On 06/11/2014 04:02 AM, Olav Peeters wrote: Hi, I upgraded from glusterfs 3.4 to 3.5 about 8 days ago. Everything was running fine until this morning. In a fuse mount we were having write issues. Creating and deleting files became an issue all of a sudden without any new changes to the cluster. In /var/log/glusterfs/glustershd.log every couple of seconds I'm getting this: [2014-06-10 22:23:52.055128] I [rpc-clnt.c:1685:rpc_clnt_reconfig] 0-sr_vol01-client-13: changing port to 49156 (from 0) [2014-06-10 22:23:52.060153] E [socket.c:2161:socket_connect_finish] 0-sr_vol01-client-13: connection to ip-of-one-of-the-gluster-nodes:49156 failed (Connection refused) # gluster volume status sr_vol01 shows that two bricks of the 18 are offline. rebalance fails Iptables was stopped on all nodes If I cd into the two bricks which are offline according to the gluster v status, I can read/write without any problems... The disks are clearly fine. They are mounted, they are available. I cannot find much info online about the error. Does anyone have an idea what could be wrong? How can I get the two bricks back online? Cheers, Olav ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___
Re: [Gluster-users] socket.c:2161:socket_connect_finish (Connection refused)
Thanks allot, Pranith! All seems back to normal again. Looking forward to the release of 3.5.1 ! Cheers, Olav On 11/06/14 09:30, Pranith Kumar Karampuri wrote: hey Just do "gluster volume start force" and things should be back to normal Pranith On 06/11/2014 12:56 PM, Olav Peeters wrote: Pranith, how could I move all data from the two problem bricks temporarily until the release of 3.5.1? Like this? # gluster volume replace-brick VOLNAME BRICK NEW-BRICK start Will this work if the bricks are offline? Or is there some other way to get the bricks back online manually? Would it help to do all fuse connections via NFS until after the fix? Cheers, Olav On 11/06/14 08:44, Olav Peeters wrote: OK, thanks for the info! Regards, Olav On 11/06/14 08:38, Pranith Kumar Karampuri wrote: On 06/11/2014 12:03 PM, Olav Peeters wrote: Thanks Pranith! I see this at the end of the log files of one of the problem bricks (the first two errors are repeated several times): [2014-06-10 09:55:28.354659] E [rpcsvc.c:1206:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x103c59, Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.sr_vol01-server) [2014-06-10 09:55:28.354683] E [server.c:190:server_submit_reply] (-->/usr/lib64/glusterfs/3.5.0/xlator/performance/io-threads.so(iot_finodelk_cbk+0xb9) [0x7f8c8e82f189] (-->/usr/lib64/glusterfs/3.5.0/xlator/debug/io-stats.so(io_stats_finodelk_cbk+0xed) [0x7f8c8e1f22ed] (-->/usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_finodelk_cbk+0xad) [0x7f8c8dfc555d]))) 0-: Reply submission failed pending frames: frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) ... ... frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-06-10 09:55:28configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.5.0 /lib64/libc.so.6(+0x329a0)[0x7f8c94aac9a0] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(grant_blocked_inode_locks+0xc1)[0x7f8c8ea54061] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(pl_inodelk_client_cleanup+0x249)[0x7f8c8ea54569] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(+0x6f0a)[0x7f8c8ea49f0a] /usr/lib64/libglusterfs.so.0(gf_client_disconnect+0x5d)[0x7f8c964d701d] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_connection_cleanup+0x458)[0x7f8c8dfbda48] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_rpc_notify+0x183)[0x7f8c8dfb9713] /usr/lib64/libgfrpc.so.0(rpcsvc_handle_disconnect+0x105)[0x7f8c96261d35] /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x1a0)[0x7f8c96263880] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f8c96264f98] /usr/lib64/glusterfs/3.5.0/rpc-transport/socket.so(+0xa9a1)[0x7f8c914c39a1] /usr/lib64/libglusterfs.so.0(+0x672f7)[0x7f8c964d92f7] /usr/sbin/glusterfsd(main+0x564)[0x4075e4] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f8c94a98d1d] /usr/sbin/glusterfsd[0x404679] - Again no info to be found online about the error. Any idea? This is because of bug 1089470 which is fixed for 3.5.1. Which will be releasing shortly. Pranith Olav On 11/06/14 04:42, Pranith Kumar Karampuri wrote: Olav, Check logs of the bricks to see why the bricks went down. Pranith On 06/11/2014 04:02 AM, Olav Peeters wrote: Hi, I upgraded from glusterfs 3.4 to 3.5 about 8 days ago. Everything was running fine until this morning. In a fuse mount we were having write issues. Creating and deleting files became an issue all of a sudden without any new changes to the cluster. In /var/log/glusterfs/glustershd.log every couple of seconds I'm getting this: [2014-06-10 22:23:52.055128] I [rpc-clnt.c:1685:rpc_clnt_reconfig] 0-sr_vol01-client-13: changing port to 49156 (from 0) [2014-06-10 22:23:52.060153] E [socket.c:2161:socket_connect_finish] 0-sr_vol01-client-13: connection to ip-of-one-of-the-gluster-nodes:49156 failed (Connection refused) # gluster volume status sr_vol01 shows that two bricks of the 18 are offline. rebalance fails Iptables was stopped on all nodes If I cd into the two bricks which are offline according to the gluster v status, I can read/write without any problems... The disks are clearly fine. They are mounted, they are available. I cannot find much info online about the error. Does anyone have an idea what could be wrong? How can I get the two bricks back online? Cheers, Olav ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] socket.c:2161:socket_connect_finish (Connection refused)
hey Just do "gluster volume start force" and things should be back to normal Pranith On 06/11/2014 12:56 PM, Olav Peeters wrote: Pranith, how could I move all data from the two problem bricks temporarily until the release of 3.5.1? Like this? # gluster volume replace-brick VOLNAME BRICK NEW-BRICK start Will this work if the bricks are offline? Or is there some other way to get the bricks back online manually? Would it help to do all fuse connections via NFS until after the fix? Cheers, Olav On 11/06/14 08:44, Olav Peeters wrote: OK, thanks for the info! Regards, Olav On 11/06/14 08:38, Pranith Kumar Karampuri wrote: On 06/11/2014 12:03 PM, Olav Peeters wrote: Thanks Pranith! I see this at the end of the log files of one of the problem bricks (the first two errors are repeated several times): [2014-06-10 09:55:28.354659] E [rpcsvc.c:1206:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x103c59, Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.sr_vol01-server) [2014-06-10 09:55:28.354683] E [server.c:190:server_submit_reply] (-->/usr/lib64/glusterfs/3.5.0/xlator/performance/io-threads.so(iot_finodelk_cbk+0xb9) [0x7f8c8e82f189] (-->/usr/lib64/glusterfs/3.5.0/xlator/debug/io-stats.so(io_stats_finodelk_cbk+0xed) [0x7f8c8e1f22ed] (-->/usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_finodelk_cbk+0xad) [0x7f8c8dfc555d]))) 0-: Reply submission failed pending frames: frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) ... ... frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-06-10 09:55:28configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.5.0 /lib64/libc.so.6(+0x329a0)[0x7f8c94aac9a0] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(grant_blocked_inode_locks+0xc1)[0x7f8c8ea54061] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(pl_inodelk_client_cleanup+0x249)[0x7f8c8ea54569] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(+0x6f0a)[0x7f8c8ea49f0a] /usr/lib64/libglusterfs.so.0(gf_client_disconnect+0x5d)[0x7f8c964d701d] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_connection_cleanup+0x458)[0x7f8c8dfbda48] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_rpc_notify+0x183)[0x7f8c8dfb9713] /usr/lib64/libgfrpc.so.0(rpcsvc_handle_disconnect+0x105)[0x7f8c96261d35] /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x1a0)[0x7f8c96263880] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f8c96264f98] /usr/lib64/glusterfs/3.5.0/rpc-transport/socket.so(+0xa9a1)[0x7f8c914c39a1] /usr/lib64/libglusterfs.so.0(+0x672f7)[0x7f8c964d92f7] /usr/sbin/glusterfsd(main+0x564)[0x4075e4] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f8c94a98d1d] /usr/sbin/glusterfsd[0x404679] - Again no info to be found online about the error. Any idea? This is because of bug 1089470 which is fixed for 3.5.1. Which will be releasing shortly. Pranith Olav On 11/06/14 04:42, Pranith Kumar Karampuri wrote: Olav, Check logs of the bricks to see why the bricks went down. Pranith On 06/11/2014 04:02 AM, Olav Peeters wrote: Hi, I upgraded from glusterfs 3.4 to 3.5 about 8 days ago. Everything was running fine until this morning. In a fuse mount we were having write issues. Creating and deleting files became an issue all of a sudden without any new changes to the cluster. In /var/log/glusterfs/glustershd.log every couple of seconds I'm getting this: [2014-06-10 22:23:52.055128] I [rpc-clnt.c:1685:rpc_clnt_reconfig] 0-sr_vol01-client-13: changing port to 49156 (from 0) [2014-06-10 22:23:52.060153] E [socket.c:2161:socket_connect_finish] 0-sr_vol01-client-13: connection to ip-of-one-of-the-gluster-nodes:49156 failed (Connection refused) # gluster volume status sr_vol01 shows that two bricks of the 18 are offline. rebalance fails Iptables was stopped on all nodes If I cd into the two bricks which are offline according to the gluster v status, I can read/write without any problems... The disks are clearly fine. They are mounted, they are available. I cannot find much info online about the error. Does anyone have an idea what could be wrong? How can I get the two bricks back online? Cheers, Olav ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] socket.c:2161:socket_connect_finish (Connection refused)
Pranith, how could I move all data from the two problem bricks temporarily until the release of 3.5.1? Like this? # gluster volume replace-brick VOLNAME BRICK NEW-BRICK start Will this work if the bricks are offline? Or is there some other way to get the bricks back online manually? Would it help to do all fuse connections via NFS until after the fix? Cheers, Olav On 11/06/14 08:44, Olav Peeters wrote: OK, thanks for the info! Regards, Olav On 11/06/14 08:38, Pranith Kumar Karampuri wrote: On 06/11/2014 12:03 PM, Olav Peeters wrote: Thanks Pranith! I see this at the end of the log files of one of the problem bricks (the first two errors are repeated several times): [2014-06-10 09:55:28.354659] E [rpcsvc.c:1206:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x103c59, Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.sr_vol01-server) [2014-06-10 09:55:28.354683] E [server.c:190:server_submit_reply] (-->/usr/lib64/glusterfs/3.5.0/xlator/performance/io-threads.so(iot_finodelk_cbk+0xb9) [0x7f8c8e82f189] (-->/usr/lib64/glusterfs/3.5.0/xlator/debug/io-stats.so(io_stats_finodelk_cbk+0xed) [0x7f8c8e1f22ed] (-->/usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_finodelk_cbk+0xad) [0x7f8c8dfc555d]))) 0-: Reply submission failed pending frames: frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) ... ... frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-06-10 09:55:28configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.5.0 /lib64/libc.so.6(+0x329a0)[0x7f8c94aac9a0] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(grant_blocked_inode_locks+0xc1)[0x7f8c8ea54061] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(pl_inodelk_client_cleanup+0x249)[0x7f8c8ea54569] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(+0x6f0a)[0x7f8c8ea49f0a] /usr/lib64/libglusterfs.so.0(gf_client_disconnect+0x5d)[0x7f8c964d701d] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_connection_cleanup+0x458)[0x7f8c8dfbda48] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_rpc_notify+0x183)[0x7f8c8dfb9713] /usr/lib64/libgfrpc.so.0(rpcsvc_handle_disconnect+0x105)[0x7f8c96261d35] /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x1a0)[0x7f8c96263880] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f8c96264f98] /usr/lib64/glusterfs/3.5.0/rpc-transport/socket.so(+0xa9a1)[0x7f8c914c39a1] /usr/lib64/libglusterfs.so.0(+0x672f7)[0x7f8c964d92f7] /usr/sbin/glusterfsd(main+0x564)[0x4075e4] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f8c94a98d1d] /usr/sbin/glusterfsd[0x404679] - Again no info to be found online about the error. Any idea? This is because of bug 1089470 which is fixed for 3.5.1. Which will be releasing shortly. Pranith Olav On 11/06/14 04:42, Pranith Kumar Karampuri wrote: Olav, Check logs of the bricks to see why the bricks went down. Pranith On 06/11/2014 04:02 AM, Olav Peeters wrote: Hi, I upgraded from glusterfs 3.4 to 3.5 about 8 days ago. Everything was running fine until this morning. In a fuse mount we were having write issues. Creating and deleting files became an issue all of a sudden without any new changes to the cluster. In /var/log/glusterfs/glustershd.log every couple of seconds I'm getting this: [2014-06-10 22:23:52.055128] I [rpc-clnt.c:1685:rpc_clnt_reconfig] 0-sr_vol01-client-13: changing port to 49156 (from 0) [2014-06-10 22:23:52.060153] E [socket.c:2161:socket_connect_finish] 0-sr_vol01-client-13: connection to ip-of-one-of-the-gluster-nodes:49156 failed (Connection refused) # gluster volume status sr_vol01 shows that two bricks of the 18 are offline. rebalance fails Iptables was stopped on all nodes If I cd into the two bricks which are offline according to the gluster v status, I can read/write without any problems... The disks are clearly fine. They are mounted, they are available. I cannot find much info online about the error. Does anyone have an idea what could be wrong? How can I get the two bricks back online? Cheers, Olav ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] socket.c:2161:socket_connect_finish (Connection refused)
OK, thanks for the info! Regards, Olav On 11/06/14 08:38, Pranith Kumar Karampuri wrote: On 06/11/2014 12:03 PM, Olav Peeters wrote: Thanks Pranith! I see this at the end of the log files of one of the problem bricks (the first two errors are repeated several times): [2014-06-10 09:55:28.354659] E [rpcsvc.c:1206:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x103c59, Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.sr_vol01-server) [2014-06-10 09:55:28.354683] E [server.c:190:server_submit_reply] (-->/usr/lib64/glusterfs/3.5.0/xlator/performance/io-threads.so(iot_finodelk_cbk+0xb9) [0x7f8c8e82f189] (-->/usr/lib64/glusterfs/3.5.0/xlator/debug/io-stats.so(io_stats_finodelk_cbk+0xed) [0x7f8c8e1f22ed] (-->/usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_finodelk_cbk+0xad) [0x7f8c8dfc555d]))) 0-: Reply submission failed pending frames: frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) ... ... frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-06-10 09:55:28configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.5.0 /lib64/libc.so.6(+0x329a0)[0x7f8c94aac9a0] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(grant_blocked_inode_locks+0xc1)[0x7f8c8ea54061] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(pl_inodelk_client_cleanup+0x249)[0x7f8c8ea54569] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(+0x6f0a)[0x7f8c8ea49f0a] /usr/lib64/libglusterfs.so.0(gf_client_disconnect+0x5d)[0x7f8c964d701d] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_connection_cleanup+0x458)[0x7f8c8dfbda48] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_rpc_notify+0x183)[0x7f8c8dfb9713] /usr/lib64/libgfrpc.so.0(rpcsvc_handle_disconnect+0x105)[0x7f8c96261d35] /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x1a0)[0x7f8c96263880] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f8c96264f98] /usr/lib64/glusterfs/3.5.0/rpc-transport/socket.so(+0xa9a1)[0x7f8c914c39a1] /usr/lib64/libglusterfs.so.0(+0x672f7)[0x7f8c964d92f7] /usr/sbin/glusterfsd(main+0x564)[0x4075e4] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f8c94a98d1d] /usr/sbin/glusterfsd[0x404679] - Again no info to be found online about the error. Any idea? This is because of bug 1089470 which is fixed for 3.5.1. Which will be releasing shortly. Pranith Olav On 11/06/14 04:42, Pranith Kumar Karampuri wrote: Olav, Check logs of the bricks to see why the bricks went down. Pranith On 06/11/2014 04:02 AM, Olav Peeters wrote: Hi, I upgraded from glusterfs 3.4 to 3.5 about 8 days ago. Everything was running fine until this morning. In a fuse mount we were having write issues. Creating and deleting files became an issue all of a sudden without any new changes to the cluster. In /var/log/glusterfs/glustershd.log every couple of seconds I'm getting this: [2014-06-10 22:23:52.055128] I [rpc-clnt.c:1685:rpc_clnt_reconfig] 0-sr_vol01-client-13: changing port to 49156 (from 0) [2014-06-10 22:23:52.060153] E [socket.c:2161:socket_connect_finish] 0-sr_vol01-client-13: connection to ip-of-one-of-the-gluster-nodes:49156 failed (Connection refused) # gluster volume status sr_vol01 shows that two bricks of the 18 are offline. rebalance fails Iptables was stopped on all nodes If I cd into the two bricks which are offline according to the gluster v status, I can read/write without any problems... The disks are clearly fine. They are mounted, they are available. I cannot find much info online about the error. Does anyone have an idea what could be wrong? How can I get the two bricks back online? Cheers, Olav ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] socket.c:2161:socket_connect_finish (Connection refused)
On 06/11/2014 12:03 PM, Olav Peeters wrote: Thanks Pranith! I see this at the end of the log files of one of the problem bricks (the first two errors are repeated several times): [2014-06-10 09:55:28.354659] E [rpcsvc.c:1206:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x103c59, Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.sr_vol01-server) [2014-06-10 09:55:28.354683] E [server.c:190:server_submit_reply] (-->/usr/lib64/glusterfs/3.5.0/xlator/performance/io-threads.so(iot_finodelk_cbk+0xb9) [0x7f8c8e82f189] (-->/usr/lib64/glusterfs/3.5.0/xlator/debug/io-stats.so(io_stats_finodelk_cbk+0xed) [0x7f8c8e1f22ed] (-->/usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_finodelk_cbk+0xad) [0x7f8c8dfc555d]))) 0-: Reply submission failed pending frames: frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) ... ... frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-06-10 09:55:28configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.5.0 /lib64/libc.so.6(+0x329a0)[0x7f8c94aac9a0] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(grant_blocked_inode_locks+0xc1)[0x7f8c8ea54061] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(pl_inodelk_client_cleanup+0x249)[0x7f8c8ea54569] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(+0x6f0a)[0x7f8c8ea49f0a] /usr/lib64/libglusterfs.so.0(gf_client_disconnect+0x5d)[0x7f8c964d701d] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_connection_cleanup+0x458)[0x7f8c8dfbda48] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_rpc_notify+0x183)[0x7f8c8dfb9713] /usr/lib64/libgfrpc.so.0(rpcsvc_handle_disconnect+0x105)[0x7f8c96261d35] /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x1a0)[0x7f8c96263880] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f8c96264f98] /usr/lib64/glusterfs/3.5.0/rpc-transport/socket.so(+0xa9a1)[0x7f8c914c39a1] /usr/lib64/libglusterfs.so.0(+0x672f7)[0x7f8c964d92f7] /usr/sbin/glusterfsd(main+0x564)[0x4075e4] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f8c94a98d1d] /usr/sbin/glusterfsd[0x404679] - Again no info to be found online about the error. Any idea? This is because of bug 1089470 which is fixed for 3.5.1. Which will be releasing shortly. Pranith Olav On 11/06/14 04:42, Pranith Kumar Karampuri wrote: Olav, Check logs of the bricks to see why the bricks went down. Pranith On 06/11/2014 04:02 AM, Olav Peeters wrote: Hi, I upgraded from glusterfs 3.4 to 3.5 about 8 days ago. Everything was running fine until this morning. In a fuse mount we were having write issues. Creating and deleting files became an issue all of a sudden without any new changes to the cluster. In /var/log/glusterfs/glustershd.log every couple of seconds I'm getting this: [2014-06-10 22:23:52.055128] I [rpc-clnt.c:1685:rpc_clnt_reconfig] 0-sr_vol01-client-13: changing port to 49156 (from 0) [2014-06-10 22:23:52.060153] E [socket.c:2161:socket_connect_finish] 0-sr_vol01-client-13: connection to ip-of-one-of-the-gluster-nodes:49156 failed (Connection refused) # gluster volume status sr_vol01 shows that two bricks of the 18 are offline. rebalance fails Iptables was stopped on all nodes If I cd into the two bricks which are offline according to the gluster v status, I can read/write without any problems... The disks are clearly fine. They are mounted, they are available. I cannot find much info online about the error. Does anyone have an idea what could be wrong? How can I get the two bricks back online? Cheers, Olav ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] socket.c:2161:socket_connect_finish (Connection refused)
Thanks Pranith! I see this at the end of the log files of one of the problem bricks (the first two errors are repeated several times): [2014-06-10 09:55:28.354659] E [rpcsvc.c:1206:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x103c59, Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.sr_vol01-server) [2014-06-10 09:55:28.354683] E [server.c:190:server_submit_reply] (-->/usr/lib64/glusterfs/3.5.0/xlator/performance/io-threads.so(iot_finodelk_cbk+0xb9) [0x7f8c8e82f189] (-->/usr/lib64/glusterfs/3.5.0/xlator/debug/io-stats.so(io_stats_finodelk_cbk+0xed) [0x7f8c8e1f22ed] (-->/usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_finodelk_cbk+0xad) [0x7f8c8dfc555d]))) 0-: Reply submission failed pending frames: frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) ... ... frame : type(0) op(30) frame : type(0) op(30) frame : type(0) op(30) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-06-10 09:55:28configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.5.0 /lib64/libc.so.6(+0x329a0)[0x7f8c94aac9a0] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(grant_blocked_inode_locks+0xc1)[0x7f8c8ea54061] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(pl_inodelk_client_cleanup+0x249)[0x7f8c8ea54569] /usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(+0x6f0a)[0x7f8c8ea49f0a] /usr/lib64/libglusterfs.so.0(gf_client_disconnect+0x5d)[0x7f8c964d701d] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_connection_cleanup+0x458)[0x7f8c8dfbda48] /usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_rpc_notify+0x183)[0x7f8c8dfb9713] /usr/lib64/libgfrpc.so.0(rpcsvc_handle_disconnect+0x105)[0x7f8c96261d35] /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x1a0)[0x7f8c96263880] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f8c96264f98] /usr/lib64/glusterfs/3.5.0/rpc-transport/socket.so(+0xa9a1)[0x7f8c914c39a1] /usr/lib64/libglusterfs.so.0(+0x672f7)[0x7f8c964d92f7] /usr/sbin/glusterfsd(main+0x564)[0x4075e4] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f8c94a98d1d] /usr/sbin/glusterfsd[0x404679] - Again no info to be found online about the error. Any idea? Olav On 11/06/14 04:42, Pranith Kumar Karampuri wrote: Olav, Check logs of the bricks to see why the bricks went down. Pranith On 06/11/2014 04:02 AM, Olav Peeters wrote: Hi, I upgraded from glusterfs 3.4 to 3.5 about 8 days ago. Everything was running fine until this morning. In a fuse mount we were having write issues. Creating and deleting files became an issue all of a sudden without any new changes to the cluster. In /var/log/glusterfs/glustershd.log every couple of seconds I'm getting this: [2014-06-10 22:23:52.055128] I [rpc-clnt.c:1685:rpc_clnt_reconfig] 0-sr_vol01-client-13: changing port to 49156 (from 0) [2014-06-10 22:23:52.060153] E [socket.c:2161:socket_connect_finish] 0-sr_vol01-client-13: connection to ip-of-one-of-the-gluster-nodes:49156 failed (Connection refused) # gluster volume status sr_vol01 shows that two bricks of the 18 are offline. rebalance fails Iptables was stopped on all nodes If I cd into the two bricks which are offline according to the gluster v status, I can read/write without any problems... The disks are clearly fine. They are mounted, they are available. I cannot find much info online about the error. Does anyone have an idea what could be wrong? How can I get the two bricks back online? Cheers, Olav ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] socket.c:2161:socket_connect_finish (Connection refused)
Olav, Check logs of the bricks to see why the bricks went down. Pranith On 06/11/2014 04:02 AM, Olav Peeters wrote: Hi, I upgraded from glusterfs 3.4 to 3.5 about 8 days ago. Everything was running fine until this morning. In a fuse mount we were having write issues. Creating and deleting files became an issue all of a sudden without any new changes to the cluster. In /var/log/glusterfs/glustershd.log every couple of seconds I'm getting this: [2014-06-10 22:23:52.055128] I [rpc-clnt.c:1685:rpc_clnt_reconfig] 0-sr_vol01-client-13: changing port to 49156 (from 0) [2014-06-10 22:23:52.060153] E [socket.c:2161:socket_connect_finish] 0-sr_vol01-client-13: connection to ip-of-one-of-the-gluster-nodes:49156 failed (Connection refused) # gluster volume status sr_vol01 shows that two bricks of the 18 are offline. rebalance fails Iptables was stopped on all nodes If I cd into the two bricks which are offline according to the gluster v status, I can read/write without any problems... The disks are clearly fine. They are mounted, they are available. I cannot find much info online about the error. Does anyone have an idea what could be wrong? How can I get the two bricks back online? Cheers, Olav ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
