Re: [Gluster-users] peer probe failures
can you please share the glusterd log file? On Thu, Jun 15, 2017 at 5:18 PM, Guy Cukierman wrote: > Hi, > > I’m having a similar issue, were you able to solve it? > > Thanks. > > > > > > > > Hey all, > > > > I've got a strange problem going on here. I've installed glusterfs-server > > on ubuntu 16.04: > > glusterfs-client/xenial,now 3.7.6-1ubuntu1 amd64 [installed,automatic] > > glusterfs-common/xenial,now 3.7.6-1ubuntu1 amd64 [installed,automatic] > > glusterfs-server/xenial,now 3.7.6-1ubuntu1 amd64 [installed] > > > > I can successfully probe another peer at this point. Then, after installing > > kubernetes via kargo, peer probing begins failing with a timeout. I've > > tried stopping all kubernetes related services, and flushing all iptables > > rules, however I don't see any packets leaving any interface when > > attempting to peer probe. > > > > from cli.log: > > [2017-04-03 22:20:24.704900] I [MSGID: 101190] > > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread > > with index 1 > > [2017-04-03 22:20:24.704973] T [cli.c:273:cli_rpc_notify] 0-glusterfs: got > > RPC_CLNT_CONNECT > > [2017-04-03 22:20:24.705001] T [cli-quotad-client.c:94:cli_quotad_notify] > > 0-glusterfs: got RPC_CLNT_CONNECT > > [2017-04-03 22:20:24.705014] I [socket.c:2355:socket_event_handler] > > 0-transport: disconnecting now > > [2017-04-03 22:20:24.705204] T [rpc-clnt.c:1404:rpc_clnt_record] > > 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: > > [2017-04-03 22:20:24.705256] T > > [rpc-clnt.c:1261:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen > > 156, payload: 92, rpc hdr: 64 > > [2017-04-03 22:20:24.705662] T [socket.c:2879:socket_connect] (--> > > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[ > 0x7f012fd21953] > > (--> /usr/lib/x86_64-linux-gnu > > /libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f012f69add4] > (--> > > /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x55)[ > 0x7f012f697af5] > > (--> /usr/lib/x8 > > 6_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f012f698338] (--> > > /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_ > notify+0x23)[0x7f012f6945b3] > > ) 0-glusterfs: connect > > () called on transport already connected > > [2017-04-03 22:20:24.705680] D > > [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> > > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[ > 0x7f012fd21953] > > (--> / > > usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ > ping_timer_locked+0x84)[0x7f012f69add4] > > (--> > > /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x55)[ > 0x7f012f > > 697af5] (--> > > /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+ > 0x88)[0x7f012f698338] > > (--> > > /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_ > notify+0x23)[0x7f012f6945b3] > > ))) > > )) 0-: /var/run/gluster/quotad.socket: ping timer event already removed > > [2017-04-03 22:20:24.705710] T [cli-quotad-client.c:100:cli_quotad_notify] > > 0-glusterfs: got RPC_CLNT_DISCONNECT > > [2017-04-03 22:20:24.705718] T [rpc-clnt.c:1598:rpc_clnt_submit] > > 0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, > > Proc: 1) to rpc-transport (glusterfs) > > [2017-04-03 22:20:24.705739] D [rpc-clnt-ping.c:281:rpc_clnt_start_ping] > > 0-glusterfs: ping timeout is 0, returning > > [2017-04-03 22:20:24.705723] D [MSGID: 0] > > [event-epoll.c:591:event_dispatch_epoll_handler] 0-epoll: generation > bumped > > on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7 > > [2017-04-03 22:20:27.614881] T [rpc-clnt.c:418:rpc_clnt_reconnect] > > 0-glusterfs: attempting reconnect > > [2017-04-03 22:20:27.615151] T [socket.c:2879:socket_connect] (--> > > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[ > 0x7f012fd21953] > > (--> > > /usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/rpc-transport/ > socket.so(+0x6c1b)[0x7f012a697c1b] > > (--> > > /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_reconnect+0xb9)[ > 0x7f012f695999] > > (--> > > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_timer_ > proc+0xfc)[0x7f012fd3d70c] > > (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f012f0b86ba] ) > > 0-glusterfs: connect () called on transport already connected > > > > it then repeats the following: > > [2017-04-03 22:20:27.615177] T [rpc-clnt.c:418:rpc_clnt_reconnect] > > 0-glusterfs: attempting reconnect > > [2017-04-03 22:20:27.615188] T [socket.c:2887:socket_connect] 0-glusterfs: > > connecting 0x25d3550, state=0 gen=0 sock=-1 > > [2017-04-03 22:20:27.615200] T > > [name.c:295:af_unix_client_get_remote_sockaddr] 0-glusterfs: using > > connect-path /var/run/gluster/quotad.socket > > [2017-04-03 22:20:27.615218] T [name.c:111:af_unix_client_bind] > > 0-glusterfs: bind-path not specified for unix socket, letting connect to > > assign default value > > [2017-04-03 22:20:27.615329] T [cli-quotad-client.c:94:cli_quotad_notify] > > 0-glusterfs: got RPC_CLNT_CONNECT
Re: [Gluster-users] peer probe failures
Hi, I'm having a similar issue, were you able to solve it? Thanks. Hey all, I've got a strange problem going on here. I've installed glusterfs-server on ubuntu 16.04: glusterfs-client/xenial,now 3.7.6-1ubuntu1 amd64 [installed,automatic] glusterfs-common/xenial,now 3.7.6-1ubuntu1 amd64 [installed,automatic] glusterfs-server/xenial,now 3.7.6-1ubuntu1 amd64 [installed] I can successfully probe another peer at this point. Then, after installing kubernetes via kargo, peer probing begins failing with a timeout. I've tried stopping all kubernetes related services, and flushing all iptables rules, however I don't see any packets leaving any interface when attempting to peer probe. from cli.log: [2017-04-03 22:20:24.704900] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-03 22:20:24.704973] T [cli.c:273:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-04-03 22:20:24.705001] T [cli-quotad-client.c:94:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-04-03 22:20:24.705014] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2017-04-03 22:20:24.705204] T [rpc-clnt.c:1404:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: [2017-04-03 22:20:24.705256] T [rpc-clnt.c:1261:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64 [2017-04-03 22:20:24.705662] T [socket.c:2879:socket_connect] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f012fd21953] (--> /usr/lib/x86_64-linux-gnu /libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f012f69add4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x55)[0x7f012f697af5] (--> /usr/lib/x8 6_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f012f698338] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f012f6945b3] ) 0-glusterfs: connect () called on transport already connected [2017-04-03 22:20:24.705680] D [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f012fd21953] (--> / usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f012f69add4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x55)[0x7f012f 697af5] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f012f698338] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f012f6945b3] ))) )) 0-: /var/run/gluster/quotad.socket: ping timer event already removed [2017-04-03 22:20:24.705710] T [cli-quotad-client.c:100:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT [2017-04-03 22:20:24.705718] T [rpc-clnt.c:1598:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 1) to rpc-transport (glusterfs) [2017-04-03 22:20:24.705739] D [rpc-clnt-ping.c:281:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2017-04-03 22:20:24.705723] D [MSGID: 0] [event-epoll.c:591:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7 [2017-04-03 22:20:27.614881] T [rpc-clnt.c:418:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-04-03 22:20:27.615151] T [socket.c:2879:socket_connect] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f012fd21953] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/rpc-transport/socket.so(+0x6c1b)[0x7f012a697c1b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_reconnect+0xb9)[0x7f012f695999] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_timer_proc+0xfc)[0x7f012fd3d70c] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f012f0b86ba] ) 0-glusterfs: connect () called on transport already connected it then repeats the following: [2017-04-03 22:20:27.615177] T [rpc-clnt.c:418:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-04-03 22:20:27.615188] T [socket.c:2887:socket_connect] 0-glusterfs: connecting 0x25d3550, state=0 gen=0 sock=-1 [2017-04-03 22:20:27.615200] T [name.c:295:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/gluster/quotad.socket [2017-04-03 22:20:27.615218] T [name.c:111:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-04-03 22:20:27.615329] T [cli-quotad-client.c:94:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-04-03 22:20:27.615355] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2017-04-03 22:20:27.615567] D [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f012fd21953] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f012f69add4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x55)[0x7f012f697af5] (--> /usr/lib/x86_64-linux-gnu/libgf
[Gluster-users] peer probe failures
Hey all, I've got a strange problem going on here. I've installed glusterfs-server on ubuntu 16.04: glusterfs-client/xenial,now 3.7.6-1ubuntu1 amd64 [installed,automatic] glusterfs-common/xenial,now 3.7.6-1ubuntu1 amd64 [installed,automatic] glusterfs-server/xenial,now 3.7.6-1ubuntu1 amd64 [installed] I can successfully probe another peer at this point. Then, after installing kubernetes via kargo, peer probing begins failing with a timeout. I've tried stopping all kubernetes related services, and flushing all iptables rules, however I don't see any packets leaving any interface when attempting to peer probe. from cli.log: [2017-04-03 22:20:24.704900] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-03 22:20:24.704973] T [cli.c:273:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-04-03 22:20:24.705001] T [cli-quotad-client.c:94:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-04-03 22:20:24.705014] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2017-04-03 22:20:24.705204] T [rpc-clnt.c:1404:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: [2017-04-03 22:20:24.705256] T [rpc-clnt.c:1261:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64 [2017-04-03 22:20:24.705662] T [socket.c:2879:socket_connect] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f012fd21953] (--> /usr/lib/x86_64-linux-gnu /libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f012f69add4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x55)[0x7f012f697af5] (--> /usr/lib/x8 6_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f012f698338] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f012f6945b3] ) 0-glusterfs: connect () called on transport already connected [2017-04-03 22:20:24.705680] D [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f012fd21953] (--> / usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f012f69add4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x55)[0x7f012f 697af5] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f012f698338] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f012f6945b3] ))) )) 0-: /var/run/gluster/quotad.socket: ping timer event already removed [2017-04-03 22:20:24.705710] T [cli-quotad-client.c:100:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT [2017-04-03 22:20:24.705718] T [rpc-clnt.c:1598:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 1) to rpc-transport (glusterfs) [2017-04-03 22:20:24.705739] D [rpc-clnt-ping.c:281:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2017-04-03 22:20:24.705723] D [MSGID: 0] [event-epoll.c:591:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7 [2017-04-03 22:20:27.614881] T [rpc-clnt.c:418:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-04-03 22:20:27.615151] T [socket.c:2879:socket_connect] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f012fd21953] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/rpc-transport/socket.so(+0x6c1b)[0x7f012a697c1b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_reconnect+0xb9)[0x7f012f695999] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_timer_proc+0xfc)[0x7f012fd3d70c] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f012f0b86ba] ) 0-glusterfs: connect () called on transport already connected it then repeats the following: [2017-04-03 22:20:27.615177] T [rpc-clnt.c:418:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-04-03 22:20:27.615188] T [socket.c:2887:socket_connect] 0-glusterfs: connecting 0x25d3550, state=0 gen=0 sock=-1 [2017-04-03 22:20:27.615200] T [name.c:295:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/gluster/quotad.socket [2017-04-03 22:20:27.615218] T [name.c:111:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-04-03 22:20:27.615329] T [cli-quotad-client.c:94:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-04-03 22:20:27.615355] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2017-04-03 22:20:27.615567] D [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f012fd21953] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f012f69add4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x55)[0x7f012f697af5] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f012f698338] (--> /usr/lib/x86_64-li
