Update The speed is back down, but when the gfs is mounted via nfs instead of through the gluster client, i get good speeds
Thank You Kindly, Kaamesh Bioinformatician Novocraft Technologies Sdn Bhd C-23A-05, 3 Two Square, Section 19, 46300 Petaling Jaya Selangor Darul Ehsan Malaysia Mobile: +60176562635 Ph: +60379600541 Fax: +60379600540 On Thu, Feb 26, 2015 at 8:58 AM, Kaamesh Kamalaaharan <[email protected] > wrote: > Hi guys, > > My glusterfs setup is across 3 servers (gfs1 ,gfs2, hpc1) with 2 bricks > (on gfs1 and gfs2). > > The reason i have 3 servers is that i wanted the gluster to stay online > when one server goes down as i frequently lose my gfs2 server. I have no > idea what causes this and im at a dead end. > > I reinstalled glusterfs-server on my gfs2 server and i found that my > performance dropped to the point my R/W speed is around 1.5 kB/S on my 10G > LAN . Pressing tab when i ls into a folder takes around 20-30 seconds to > autocomplete, sometimes longer. Previously i had blazing fast speeds. When > i take gluster2 off the network, the speed is back up. > > Both my servers are now on the fritz and the bricks keep going offline and > i have no idea what to do. Can anyone help shed some light on this? I have > no idea what i need to provide so ill provide the logs and config. > > > ::EDIT:: > As i was typing, gluster went down again and when i brought it back up, > the speed went back to normal. This didnt happen the last 3 or four times i > restarted the servers.. I would still like to know what is going on... > :::::::: > > > > gluster volume info > > Volume Name: gfsvolume > Type: Replicate > Volume ID: a29bd2fb-b1ef-4481-be10-c2f4faf4059b > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: gfs1:/export/sda/brick > Brick2: gfs2:/export/sda/brick > Options Reconfigured: > performance.quick-read: off > network.ping-timeout: 30 > network.frame-timeout: 90 > performance.cache-max-file-size: 2MB > cluster.server-quorum-type: none > nfs.addr-namelookup: off > nfs.trusted-write: off > performance.write-behind-window-size: 4MB > cluster.data-self-heal-algorithm: diff > performance.cache-refresh-timeout: 60 > performance.cache-size: 6442450944 > cluster.quorum-type: fixed > auth.allow: 172.* > cluster.quorum-count: 1 > cluster.server-quorum-ratio: 50% > > /var/log//glusterfs/ > > > > [2015-02-26 00:13:36.207408] I > [glusterd-volume-ops.c:481:__glusterd_handle_cli_heal_volume] 0-management: > Received heal vol req for volume gfsvolume > > [2015-02-26 00:13:36.211810] E [glusterd-syncop.c:101:gd_collate_errors] > 0-: Commit failed on gfs2. Please check log file for details. > [2015-02-26 00:14:17.939391] W [glusterfsd.c:1095:cleanup_and_exit] > (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f13408f8e6d] > (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f1340fa6b50] (-->/us > r/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f1342832d55]))) 0-: received > signum (15), shutting down > [2015-02-26 00:14:20.127606] I [glusterfsd.c:1959:main] > 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 > (/usr/sbin/glusterd -p /var/run/glusterd.pid) > [2015-02-26 00:14:20.136366] I [glusterd.c:1148:init] 0-management: Using > /var/lib/glusterd as working directory > [2015-02-26 00:14:20.137768] I [socket.c:3561:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-26 00:14:20.137797] I [socket.c:3576:socket_init] > 0-socket.management: using system polling thread > [2015-02-26 00:14:20.139452] W [rdma.c:4194:__gf_rdma_ctx_create] > 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) > [2015-02-26 00:14:20.139483] E [rdma.c:4482:init] 0-rdma.management: > Failed to initialize IB Device > [2015-02-26 00:14:20.139497] E [rpc-transport.c:333:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > [2015-02-26 00:14:20.139568] W [rpcsvc.c:1483:rpcsvc_transport_create] > 0-rpc-service: cannot create listener, initing the transport failed > [2015-02-26 00:14:20.139719] I [socket.c:3561:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-26 00:14:20.139741] I [socket.c:3576:socket_init] > 0-socket.management: using system polling thread > [2015-02-26 00:14:23.351872] I > [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved > op-version: 2 > [2015-02-26 00:14:23.355502] E > [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: > brick-0 > [2015-02-26 00:14:23.355552] E > [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: > brick-1 > [2015-02-26 00:14:23.584714] I > [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect > returned 0 > [2015-02-26 00:14:23.588106] I > [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect > returned 0 > [2015-02-26 00:14:23.588223] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:23.588328] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:23.588347] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:23.589290] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:23.589378] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:23.589397] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:23.589885] I [glusterd.c:138:glusterd_uuid_init] > 0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070 > Final graph: > > +------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option transport.socket.read-fail-log off > 8: option transport.socket.keepalive-interval 2 > 9: option transport.socket.keepalive-time 10 > 10: option transport-type rdma > 11: option working-directory /var/lib/glusterd > 12: end-volume > 13: > > +------------------------------------------------------------------------------+ > [2015-02-26 00:14:23.780644] I > [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC > from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0 > [2015-02-26 00:14:23.798102] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:23.798242] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:23.798276] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:24.800202] E > [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: > Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error: > Permission denied > [2015-02-26 00:14:24.800858] I > [glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered > MOUNTV3 successfully > [2015-02-26 00:14:24.801261] I > [glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered > MOUNTV1 successfully > [2015-02-26 00:14:24.801649] I > [glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered > NFSV3 successfully > [2015-02-26 00:14:24.802028] I > [glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM > v4 successfully > [2015-02-26 00:14:24.802410] I > [glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM > v1 successfully > [2015-02-26 00:14:24.802793] I > [glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL > v3 successfully > [2015-02-26 00:14:24.806276] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:24.806387] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:24.806406] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:25.807341] E > [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: > Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No > such file or directory > [2015-02-26 00:14:25.810987] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:25.811100] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled[2015-02-26 00:14:25.811119] I > [socket.c:3576:socket_init] 0-management: using system polling thread > [2015-02-26 00:14:25.811329] I > [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: > Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c > [2015-02-26 00:14:25.811370] I > [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received > uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1 > [2015-02-26 00:14:25.811391] I > [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my > uuid as Friend > [2015-02-26 00:14:25.811407] I > [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received > uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1 > [2015-02-26 00:14:25.811510] I > [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: > using the op-version 2 > [2015-02-26 00:14:25.982101] I [socket.c:2238:socket_event_handler] > 0-transport: disconnecting now > [2015-02-26 00:14:25.982172] I [socket.c:2238:socket_event_handler] > 0-transport: disconnecting now > [2015-02-26 00:14:25.982265] I > [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC > from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0 > [2015-02-26 00:14:26.003332] I [glusterd-pmap.c:227:pmap_registry_bind] > 0-pmap: adding brick /export/sda/brick on port 49153 > [2015-02-26 00:14:26.159051] I > [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: > Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:26.159112] I > [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received > uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1 > [2015-02-26 00:14:26.159130] I > [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my > uuid as Friend > [2015-02-26 00:14:26.178634] I > [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: > Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c > [2015-02-26 00:14:26.178794] I > [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: > Responded to gfs2 (0), ret: 0 > [2015-02-26 00:14:26.194289] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1 > [2015-02-26 00:14:26.194348] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2 > [2015-02-26 00:14:26.208631] I > [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: > Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c > [2015-02-26 00:14:26.373031] I > [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: > Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:26.373101] I > [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: > using the op-version 2 > [2015-02-26 00:14:26.385510] I > [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: > Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:26.385645] I > [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: > Responded to hpc1 (0), ret: 0 > [2015-02-26 00:14:26.399963] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1 > [2015-02-26 00:14:26.400010] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2 > [2015-02-26 00:14:26.414448] I > [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: > Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c > [2015-02-26 00:14:26.668753] I > [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: > Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:32.167298] W [socket.c:522:__socket_rwv] 0-management: > readv on 172.20.20.22:24007 failed (No data available) > [2015-02-26 00:14:33.590941] E [socket.c:2161:socket_connect_finish] > 0-management: connection to 172.20.20.22:24007 failed (Connection refused) > [2015-02-26 00:14:39.060572] W [glusterfsd.c:1095:cleanup_and_exit] > (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f727130ce6d] > (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f72719bab50] > (-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f7273246d55]))) 0-: > received signum (15), shutting down > [2015-02-26 00:14:42.990079] I [glusterfsd.c:1959:main] > 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 > (/usr/sbin/glusterd -p /var/run/glusterd.pid) > [2015-02-26 00:14:42.994671] I [glusterd.c:1148:init] 0-management: Using > /var/lib/glusterd as working directory > [2015-02-26 00:14:42.996068] I [socket.c:3561:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-26 00:14:42.996098] I [socket.c:3576:socket_init] > 0-socket.management: using system polling thread > [2015-02-26 00:14:42.996892] W [rdma.c:4194:__gf_rdma_ctx_create] > 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) > [2015-02-26 00:14:42.996919] E [rdma.c:4482:init] 0-rdma.management: > Failed to initialize IB Device > [2015-02-26 00:14:42.996932] E [rpc-transport.c:333:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > [2015-02-26 00:14:42.997004] W [rpcsvc.c:1483:rpcsvc_transport_create] > 0-rpc-service: cannot create listener, initing the transport failed > [2015-02-26 00:14:42.997138] I [socket.c:3561:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-26 00:14:42.997156] I [socket.c:3576:socket_init] > 0-socket.management: using system polling thread > [2015-02-26 00:14:46.319737] I > [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved > op-version: 2 > [2015-02-26 00:14:46.323519] E > [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: > brick-0 > [2015-02-26 00:14:46.323567] E > [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: > brick-1 > [2015-02-26 00:14:46.561399] I > [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect > returned 0 > [2015-02-26 00:14:46.564795] I > [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect > returned 0 > [2015-02-26 00:14:46.564919] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:46.565035] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:46.565054] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:46.566216] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:46.566329] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:46.566348] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:46.566832] I [glusterd.c:138:glusterd_uuid_init] > 0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070 > Final graph: > > +------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option transport.socket.read-fail-log off > 8: option transport.socket.keepalive-interval 2 > 9: option transport.socket.keepalive-time 10 > 10: option transport-type rdma > 11: option working-directory /var/lib/glusterd > 12: end-volume > 13: > > +------------------------------------------------------------------------------+ > [2015-02-26 00:14:46.570634] E [socket.c:2161:socket_connect_finish] > 0-management: connection to 172.20.20.22:24007 failed (Connection refused) > [2015-02-26 00:14:46.571244] I > [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: > using the op-version 2 > [2015-02-26 00:14:47.001039] I > [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: > Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:47.001183] I > [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: > Responded to hpc1 (0), ret: 0 > [2015-02-26 00:14:47.015772] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1 > [2015-02-26 00:14:47.015820] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2 > [2015-02-26 00:14:47.030090] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:47.030178] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:47.030196] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:48.031850] E > [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: > Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error: > Permission denied > [2015-02-26 00:14:48.032392] I > [glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered > MOUNTV3 successfully > [2015-02-26 00:14:48.032794] I > [glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered > MOUNTV1 successfully > [2015-02-26 00:14:48.033182] I > [glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered > NFSV3 successfully > [2015-02-26 00:14:48.033571] I > [glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM > v4 successfully > [2015-02-26 00:14:48.033964] I > [glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM > v1 successfully > [2015-02-26 00:14:48.034356] I > [glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL > v3 successfully > [2015-02-26 00:14:48.037872] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:48.037982] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:48.038001] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:49.039002] E > [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: > Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No > such file or directory > [2015-02-26 00:14:49.042564] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:49.042670] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:49.042690] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:49.042958] I > [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: > Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:49.042998] I > [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received > uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1 > [2015-02-26 00:14:49.043013] I > [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my > uuid as Friend > [2015-02-26 00:14:49.043319] I > [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC > from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0 > [2015-02-26 00:14:49.204554] I [socket.c:2238:socket_event_handler] > 0-transport: disconnecting now > [2015-02-26 00:14:49.204657] I [socket.c:2238:socket_event_handler] > 0-transport: disconnecting now > [2015-02-26 00:14:49.204717] I > [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: > Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:49.814028] I [glusterd-pmap.c:227:pmap_registry_bind] > 0-pmap: adding brick /export/sda/brick on port 49153 > [2015-02-26 00:14:53.678347] I > [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: > using the op-version 2 > [2015-02-26 00:14:53.848276] I > [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: > Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c > [2015-02-26 00:14:53.848407] I > [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: > Responded to gfs2 (0), ret: 0 > [2015-02-26 00:14:53.862892] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1 > [2015-02-26 00:14:53.862952] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2 > [2015-02-26 00:14:53.877685] I > [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: > Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:55.217486] I > [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC > from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0 > [2015-02-26 00:14:55.234370] I > [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: > Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c > [2015-02-26 00:14:55.234433] I > [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received > uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1 > [2015-02-26 00:14:55.234451] I > [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my > uuid as Friend > [2015-02-26 00:14:55.234468] I > [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received > uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1 > [2015-02-26 00:15:14.637613] E > [glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management: > Volume gfsvolume already started > [2015-02-26 00:15:14.637661] E [glusterd-syncop.c:912:gd_stage_op_phase] > 0-management: Staging of operation 'Volume Start' failed on localhost : > Volume gfsvolume already started > [2015-02-26 00:24:41.265078] W [socket.c:522:__socket_rwv] 0-management: > readv on 172.20.20.22:24007 failed (No data available) > [2015-02-26 00:24:43.631739] E [socket.c:2161:socket_connect_finish] > 0-management: connection to 172.20.20.22:24007 failed (Connection refused) > [2015-02-26 00:24:56.174790] W [socket.c:522:__socket_rwv] 0-management: > readv on /var/run/91bfef953906a83fc14aa435a886ac4d.socket failed (No data > available) > [2015-02-26 00:24:56.175299] I > [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: > Disconnected from gfs1:/export/sda/brick > [2015-02-26 00:25:24.076560] W > [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx > modification failed > [2015-02-26 00:25:24.078109] I > [glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management: > Received status volume req for volume gfsvolume > [2015-02-26 00:25:31.041343] E > [glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management: > Volume gfsvolume already started > [2015-02-26 00:25:31.041404] E [glusterd-syncop.c:912:gd_stage_op_phase] > 0-management: Staging of operation 'Volume Start' failed on localhost : > Volume gfsvolume already started > [2015-02-26 00:25:43.389260] W [glusterfsd.c:1095:cleanup_and_exit] > (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8f8cfa7e6d] > (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f8f8d655b50] > (-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f8f8eee1d55]))) 0-: > received signum (15), shutting down > [2015-02-26 00:25:45.437325] I [glusterfsd.c:1959:main] > 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 > (/usr/sbin/glusterd -p /var/run/glusterd.pid) > [2015-02-26 00:25:45.487307] I [glusterd.c:1148:init] 0-management: Using > /var/lib/glusterd as working directory > [2015-02-26 00:25:45.488734] I [socket.c:3561:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-26 00:25:45.488765] I [socket.c:3576:socket_init] > 0-socket.management: using system polling thread > [2015-02-26 00:25:45.489557] W [rdma.c:4194:__gf_rdma_ctx_create] > 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) > [2015-02-26 00:25:45.489584] E [rdma.c:4482:init] 0-rdma.management: > Failed to initialize IB Device > [2015-02-26 00:25:45.489597] E [rpc-transport.c:333:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > [2015-02-26 00:25:45.489668] W [rpcsvc.c:1483:rpcsvc_transport_create] > 0-rpc-service: cannot create listener, initing the transport failed > [2015-02-26 00:25:45.489799] I [socket.c:3561:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-26 00:25:45.489818] I [socket.c:3576:socket_init] > 0-socket.management: using system polling thread > [2015-02-26 00:25:48.706235] I > [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved > op-version: 2 > > > > > Please Do let me know if there is anything else i can provide to help. > Apologies if this is a simple fix but i have never found an answer after a > week of searching. > > Thank You Kindly, > Kaamesh >
_______________________________________________ Gluster-users mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-users
