Dear all, I have a 3-node distribute setup with a controller of GlusterFS and upgraded to 3.7.6 today and to CentOS 7.2.
After the ugprade (reboot), I can start the volume fine and see mounted volume as well on the controller. However, a gluster volume info <volname> results in an [root@stor104 glusterfs]# gluster volume status live Commit failed on localhost. Please check the log file for more details. error. Below some information and log extracts. Thank you for any hints on where to start fixing this, Kind regards, — Christophe —————— Here is the gluster info command for the volume. [root@stor104 glusterfs]# gluster volume info live Volume Name: live Type: Distribute Volume ID: 1328637d-7730-4627-8945-bbe43626d527 Status: Started Number of Bricks: 9 Transport-type: tcp Bricks: Brick1: stor104:/zfs/brick0/brick Brick2: stor104:/zfs/brick1/brick Brick3: stor104:/zfs/brick2/brick Brick4: stor106:/zfs/brick0/brick Brick5: stor106:/zfs/brick1/brick Brick6: stor106:/zfs/brick2/brick Brick7: stor105:/zfs/brick0/brick Brick8: stor105:/zfs/brick1/brick Brick9: stor105:/zfs/brick2/brick Options Reconfigured: performance.io<http://performance.io>-thread-count: 8 nfs.disable: on performance.write-behind-window-size: 4MB performance.client-io-threads: on performance.cache-size: 1GB performance.cache-refresh-timeout: 60 performance.cache-max-file-size: 4MB cluster.data-self-heal-algorithm: full diagnostics.client-log-level: ERROR diagnostics.brick-log-level: ERROR cluster.min-free-disk: 1% server.allow-insecure: on Relevant log parts when carrying out the command: ==> /var/log/glusterfs/cli.log <== [2015-12-29 12:51:03.216998] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2015-12-29 12:51:03.226123] I [cli-cmd-volume.c:1926:cli_check_gsync_present] 0-: geo-replication not installed [2015-12-29 12:51:03.226623] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-12-29 12:51:03.226723] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now ==> /var/log/glusterfs/cmd_history.log <== [2015-12-29 12:51:03.236182] : volume status : SUCCESS ==> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log <== [2015-12-29 12:51:03.238562] I [MSGID: 106499] [glusterd-handler.c:4267:__glusterd_handle_status_volume] 0-management: Received status volume req for volume live [2015-12-29 12:51:03.246043] E [MSGID: 106396] [glusterd-op-sm.c:2985:_add_task_to_dict] 0-management: Statedump operation doesn't have a task_id [2015-12-29 12:51:03.246081] E [MSGID: 106060] [glusterd-op-sm.c:3055:glusterd_aggregate_task_status] 0-management: Failed to add task details to dict [2015-12-29 12:51:03.246098] E [MSGID: 106123] [glusterd-syncop.c:1404:gd_commit_op_phase] 0-management: Commit of operation 'Volume Status' failed on localhost ==> /var/log/glusterfs/cmd_history.log <== [2015-12-29 12:51:03.249364] : volume status : FAILED : Commit failed on localhost. Please check the log file for more details. ==> /var/log/glusterfs/cli.log <== [2015-12-29 12:51:03.249647] I [input.c:36:cli_batch] 0-: Exiting with: 0 All bricks are up [root@highlander glusterfs]# pdsh -g live 'df -h | grep brick*' stor106: brick0 50T 33T 18T 66% /zfs/brick0 stor106: brick1 50T 33T 18T 66% /zfs/brick1 stor106: brick2 50T 33T 18T 66% /zfs/brick2 stor105: brick0 40T 23T 18T 57% /zfs/brick0 stor105: brick1 40T 23T 18T 57% /zfs/brick1 stor105: brick2 40T 23T 18T 57% /zfs/brick2 stor104: brick0 40T 23T 18T 57% /zfs/brick0 stor104: brick1 40T 23T 18T 57% /zfs/brick1 stor104: brick2 40T 23T 18T 57% /zfs/brick2 Package details [root@highlander glusterfs]# rpm -qa | grep gluster glusterfs-client-xlators-3.7.6-1.el7.x86_64 glusterfs-server-3.7.6-1.el7.x86_64 samba-vfs-glusterfs-4.2.3-10.el7.x86_64 glusterfs-3.7.6-1.el7.x86_64 glusterfs-cli-3.7.6-1.el7.x86_64 glusterfs-libs-3.7.6-1.el7.x86_64 glusterfs-fuse-3.7.6-1.el7.x86_64 glusterfs-api-3.7.6-1.el7.x86_64 [root@highlander glusterfs]# pdsh -g live 'rpm -qa | grep gluster' stor105: glusterfs-libs-3.7.6-1.el7.x86_64 stor105: glusterfs-api-3.7.6-1.el7.x86_64 stor105: glusterfs-3.7.6-1.el7.x86_64 stor105: glusterfs-fuse-3.7.6-1.el7.x86_64 stor105: glusterfs-cli-3.7.6-1.el7.x86_64 stor105: glusterfs-client-xlators-3.7.6-1.el7.x86_64 stor105: glusterfs-server-3.7.6-1.el7.x86_64 stor104: glusterfs-server-3.7.6-1.el7.x86_64 stor104: glusterfs-libs-3.7.6-1.el7.x86_64 stor104: glusterfs-api-3.7.6-1.el7.x86_64 stor104: glusterfs-3.7.6-1.el7.x86_64 stor104: glusterfs-fuse-3.7.6-1.el7.x86_64 stor104: glusterfs-cli-3.7.6-1.el7.x86_64 stor104: glusterfs-client-xlators-3.7.6-1.el7.x86_64 stor106: glusterfs-3.7.6-1.el7.x86_64 stor106: glusterfs-cli-3.7.6-1.el7.x86_64 stor106: glusterfs-server-3.7.6-1.el7.x86_64 stor106: glusterfs-libs-3.7.6-1.el7.x86_64 stor106: glusterfs-client-xlators-3.7.6-1.el7.x86_64 stor106: glusterfs-api-3.7.6-1.el7.x86_64 stor106: glusterfs-fuse-3.7.6-1.el7.x86_64 More detailled logs: ==> /var/log/glusterfs/cli.log <== [2015-12-29 12:57:23.520821] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2015-12-29 12:57:23.530898] I [cli-cmd-volume.c:1926:cli_check_gsync_present] 0-: geo-replication not installed [2015-12-29 12:57:23.531844] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-12-29 12:57:23.532004] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now ==> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log <== [2015-12-29 12:57:23.534830] D [MSGID: 0] [glusterd-op-sm.c:214:glusterd_generate_txn_id] 0-management: Transaction_id = c2886398-03c1-4d32-924a-9f92367be85c [2015-12-29 12:57:23.534946] D [MSGID: 0] [glusterd-op-sm.c:311:glusterd_set_txn_opinfo] 0-management: Successfully set opinfo for transaction ID : c2886398-03c1-4d32-924a-9f92367be85c [2015-12-29 12:57:23.534975] D [MSGID: 0] [glusterd-op-sm.c:318:glusterd_set_txn_opinfo] 0-management: Returning 0 [2015-12-29 12:57:23.535001] D [MSGID: 0] [glusterd-syncop.c:1767:gd_sync_task_begin] 0-management: Transaction ID : c2886398-03c1-4d32-924a-9f92367be85c [2015-12-29 12:57:23.535031] D [MSGID: 0] [glusterd-syncop.c:1807:gd_sync_task_begin] 0-glusterd: Failed to get volume name [2015-12-29 12:57:23.535078] D [MSGID: 0] [glusterd-op-sm.c:1695:glusterd_op_stage_status_volume] 0-management: Returning: 0 [2015-12-29 12:57:23.535103] D [MSGID: 0] [glusterd-op-sm.c:5515:glusterd_op_stage_validate] 0-management: OP = 18. Returning 0 [2015-12-29 12:57:23.535555] D [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f410a9d5a82] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f410a7a587b] (--> /lib64/libgfrpc.so.0(+0x13e74)[0x7f410a7a5e74] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x34f)[0x7f410a7a1c4f] (--> /usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(gd_syncop_submit_request+0x1a5)[0x7f40ff5d24b5] ))))) 0-: 192.168.123.105:24007: ping timer event already removed [2015-12-29 12:57:23.535935] D [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f410a9d5a82] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f410a7a587b] (--> /lib64/libgfrpc.so.0(+0x13e74)[0x7f410a7a5e74] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x34f)[0x7f410a7a1c4f] (--> /usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(gd_syncop_submit_request+0x1a5)[0x7f40ff5d24b5] ))))) 0-: 192.168.123.1:24007: ping timer event already removed [2015-12-29 12:57:23.536279] D [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f410a9d5a82] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f410a7a587b] (--> /lib64/libgfrpc.so.0(+0x13e74)[0x7f410a7a5e74] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x34f)[0x7f410a7a1c4f] (--> /usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(gd_syncop_submit_request+0x1a5)[0x7f40ff5d24b5] ))))) 0-: 192.168.123.106:24007: ping timer event already removed [2015-12-29 12:57:23.536358] D [MSGID: 0] [glusterd-syncop.c:1312:gd_stage_op_phase] 0-management: Sent stage op req for 'Volume Status' to 3 peers [2015-12-29 12:57:23.538064] D [MSGID: 0] [glusterd-peer-utils.c:200:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster [2015-12-29 12:57:23.539059] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [glusterd-peer-utils.c:200:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster" repeated 2 times between [2015-12-29 12:57:23.538064] and [2015-12-29 12:57:23.538833] [2015-12-29 12:57:23.539057] D [MSGID: 0] [glusterd-op-sm.c:7010:glusterd_op_bricks_select] 0-management: Returning 0 [2015-12-29 12:57:23.539163] D [MSGID: 0] [glusterd-syncop.c:1709:gd_brick_op_phase] 0-management: Sent op req to 0 bricks [2015-12-29 12:57:23.539213] D [MSGID: 0] [glusterd-op-sm.c:3360:glusterd_op_status_volume] 0-management: Returning 0 [2015-12-29 12:57:23.539237] D [MSGID: 0] [glusterd-op-sm.c:5642:glusterd_op_commit_perform] 0-management: Returning 0 [2015-12-29 12:57:23.541960] D [MSGID: 0] [glusterd-peer-utils.c:200:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster [2015-12-29 12:57:23.542525] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [glusterd-peer-utils.c:200:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster" repeated 2 times between [2015-12-29 12:57:23.541960] and [2015-12-29 12:57:23.542186] [2015-12-29 12:57:23.542523] D [MSGID: 0] [glusterd-syncop.c:1449:gd_commit_op_phase] 0-management: Sent commit op req for 'Volume Status' to 3 peers [2015-12-29 12:57:23.542643] D [MSGID: 0] [glusterd-op-sm.c:4487:glusterd_op_modify_op_ctx] 0-management: op_ctx modification not required for status operation being performed [2015-12-29 12:57:23.542680] D [MSGID: 0] [glusterd-op-sm.c:255:glusterd_get_txn_opinfo] 0-management: Successfully got opinfo for transaction ID : c2886398-03c1-4d32-924a-9f92367be85c [2015-12-29 12:57:23.542704] D [MSGID: 0] [glusterd-op-sm.c:259:glusterd_get_txn_opinfo] 0-management: Returning 0 [2015-12-29 12:57:23.542732] D [MSGID: 0] [glusterd-op-sm.c:360:glusterd_clear_txn_opinfo] 0-management: Successfully cleared opinfo for transaction ID : c2886398-03c1-4d32-924a-9f92367be85c [2015-12-29 12:57:23.542756] D [MSGID: 0] [glusterd-op-sm.c:364:glusterd_clear_txn_opinfo] 0-management: Returning 0 ==> /var/log/glusterfs/cmd_history.log <== [2015-12-29 12:57:23.542781] : volume status : SUCCESS ==> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log <== [2015-12-29 12:57:23.542847] D [MSGID: 0] [glusterd-rpc-ops.c:205:glusterd_op_send_cli_response] 0-management: Returning 0 [2015-12-29 12:57:23.545697] I [MSGID: 106499] [glusterd-handler.c:4267:__glusterd_handle_status_volume] 0-management: Received status volume req for volume live [2015-12-29 12:57:23.545818] D [MSGID: 0] [glusterd-op-sm.c:214:glusterd_generate_txn_id] 0-management: Transaction_id = fd3671e4-fa7a-4913-a6dd-b37884a3a715 [2015-12-29 12:57:23.545872] D [MSGID: 0] [glusterd-op-sm.c:311:glusterd_set_txn_opinfo] 0-management: Successfully set opinfo for transaction ID : fd3671e4-fa7a-4913-a6dd-b37884a3a715 [2015-12-29 12:57:23.545905] D [MSGID: 0] [glusterd-op-sm.c:318:glusterd_set_txn_opinfo] 0-management: Returning 0 [2015-12-29 12:57:23.545926] D [MSGID: 0] [glusterd-syncop.c:1767:gd_sync_task_begin] 0-management: Transaction ID : fd3671e4-fa7a-4913-a6dd-b37884a3a715 [2015-12-29 12:57:23.545978] D [MSGID: 0] [glusterd-locks.c:562:glusterd_mgmt_v3_lock] 0-management: Trying to acquire lock of vol live for 305c0f00-0f11-4da3-a470-50b6e6c14976 as live_vol [2015-12-29 12:57:23.546388] D [MSGID: 0] [glusterd-locks.c:618:glusterd_mgmt_v3_lock] 0-management: Lock for vol live successfully held by 305c0f00-0f11-4da3-a470-50b6e6c14976 [2015-12-29 12:57:23.546478] D [MSGID: 0] [glusterd-syncop.c:411:gd_syncop_mgmt_v3_lock] 0-glusterd: Returning 0 [2015-12-29 12:57:23.549943] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [glusterd-syncop.c:411:gd_syncop_mgmt_v3_lock] 0-glusterd: Returning 0" repeated 2 times between [2015-12-29 12:57:23.546478] and [2015-12-29 12:57:23.546565] [2015-12-29 12:57:23.549941] D [MSGID: 0] [glusterd-syncop.c:1205:gd_lock_op_phase] 0-management: Sent lock op req for 'Volume Status' to 3 peers. Returning 0 [2015-12-29 12:57:23.550063] D [MSGID: 0] [glusterd-utils.c:1424:glusterd_volinfo_find] 0-management: Volume live found [2015-12-29 12:57:23.550087] D [MSGID: 0] [glusterd-utils.c:1431:glusterd_volinfo_find] 0-management: Returning 0 [2015-12-29 12:57:23.550121] D [MSGID: 0] [glusterd-utils.c:1424:glusterd_volinfo_find] 0-management: Volume live found [2015-12-29 12:57:23.550140] D [MSGID: 0] [glusterd-utils.c:1431:glusterd_volinfo_find] 0-management: Returning 0 [2015-12-29 12:57:23.550167] D [MSGID: 0] [glusterd-op-sm.c:1695:glusterd_op_stage_status_volume] 0-management: Returning: 0 [2015-12-29 12:57:23.550192] D [MSGID: 0] [glusterd-op-sm.c:5515:glusterd_op_stage_validate] 0-management: OP = 18. Returning 0 [2015-12-29 12:57:23.550309] D [MSGID: 0] [glusterd-syncop.c:1312:gd_stage_op_phase] 0-management: Sent stage op req for 'Volume Status' to 3 peers [2015-12-29 12:57:23.552791] D [MSGID: 0] [glusterd-peer-utils.c:200:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster [2015-12-29 12:57:23.553395] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [glusterd-peer-utils.c:200:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster" repeated 2 times between [2015-12-29 12:57:23.552791] and [2015-12-29 12:57:23.553087] [2015-12-29 12:57:23.553394] D [MSGID: 0] [glusterd-op-sm.c:7010:glusterd_op_bricks_select] 0-management: Returning 0 [2015-12-29 12:57:23.553499] D [MSGID: 0] [glusterd-syncop.c:1709:gd_brick_op_phase] 0-management: Sent op req to 0 bricks [2015-12-29 12:57:23.553535] D [MSGID: 0] [glusterd-utils.c:1424:glusterd_volinfo_find] 0-management: Volume live found [2015-12-29 12:57:23.553556] D [MSGID: 0] [glusterd-utils.c:1431:glusterd_volinfo_find] 0-management: Returning 0 [2015-12-29 12:57:23.553786] D [MSGID: 0] [glusterd-snapshot-utils.c:3595:glusterd_is_snapd_enabled] 0-management: Key features.uss not present in the dict for volume live [2015-12-29 12:57:23.554065] E [MSGID: 106396] [glusterd-op-sm.c:2985:_add_task_to_dict] 0-management: Statedump operation doesn't have a task_id [2015-12-29 12:57:23.554097] E [MSGID: 106060] [glusterd-op-sm.c:3055:glusterd_aggregate_task_status] 0-management: Failed to add task details to dict [2015-12-29 12:57:23.554117] D [MSGID: 0] [glusterd-op-sm.c:3360:glusterd_op_status_volume] 0-management: Returning -1 [2015-12-29 12:57:23.554135] D [MSGID: 0] [glusterd-op-sm.c:5642:glusterd_op_commit_perform] 0-management: Returning -1 [2015-12-29 12:57:23.554155] E [MSGID: 106123] [glusterd-syncop.c:1404:gd_commit_op_phase] 0-management: Commit of operation 'Volume Status' failed on localhost [2015-12-29 12:57:23.554240] D [MSGID: 0] [glusterd-syncop.c:510:gd_syncop_mgmt_v3_unlock] 0-glusterd: Returning 0 [2015-12-29 12:57:23.557147] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [glusterd-syncop.c:510:gd_syncop_mgmt_v3_unlock] 0-glusterd: Returning 0" repeated 2 times between [2015-12-29 12:57:23.554240] and [2015-12-29 12:57:23.554319] [2015-12-29 12:57:23.557146] D [MSGID: 0] [glusterd-syncop.c:1558:gd_unlock_op_phase] 0-management: Sent unlock op req for 'Volume Status' to 3 peers. Returning 0 [2015-12-29 12:57:23.557255] D [MSGID: 0] [glusterd-locks.c:669:glusterd_mgmt_v3_unlock] 0-management: Trying to release lock of vol live for 305c0f00-0f11-4da3-a470-50b6e6c14976 as live_vol [2015-12-29 12:57:23.557286] D [MSGID: 0] [glusterd-locks.c:714:glusterd_mgmt_v3_unlock] 0-management: Lock for vol live successfully released [2015-12-29 12:57:23.557312] D [MSGID: 0] [glusterd-op-sm.c:255:glusterd_get_txn_opinfo] 0-management: Successfully got opinfo for transaction ID : fd3671e4-fa7a-4913-a6dd-b37884a3a715 [2015-12-29 12:57:23.557332] D [MSGID: 0] [glusterd-op-sm.c:259:glusterd_get_txn_opinfo] 0-management: Returning 0 [2015-12-29 12:57:23.557356] D [MSGID: 0] [glusterd-op-sm.c:360:glusterd_clear_txn_opinfo] 0-management: Successfully cleared opinfo for transaction ID : fd3671e4-fa7a-4913-a6dd-b37884a3a715 [2015-12-29 12:57:23.557420] D [MSGID: 0] [glusterd-op-sm.c:364:glusterd_clear_txn_opinfo] 0-management: Returning 0 ==> /var/log/glusterfs/cmd_history.log <== [2015-12-29 12:57:23.557447] : volume status : FAILED : Commit failed on localhost. Please check the log file for more details. Dr Christophe Trefois, Dipl.-Ing. Technical Specialist / Post-Doc UNIVERSITÉ DU LUXEMBOURG LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | House of Biomedicine 6, avenue du Swing L-4367 Belvaux T: +352 46 66 44 6124 F: +352 46 66 44 6949 http://www.uni.lu/lcsb [Facebook]<https://www.facebook.com/trefex> [Twitter] <https://twitter.com/Trefex> [Google Plus] <https://plus.google.com/+ChristopheTrefois/> [Linkedin] <https://www.linkedin.com/in/trefoischristophe> [skype] <http://skype:Trefex?call> ---- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. ----
_______________________________________________ Gluster-users mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-users
