Hello, We have a prod cluster of four machines running riak (1.1.4 2012-06-19) Debian x86_64. Two days ago one of the servers went down because of a hardware failure. I force-removed the machine in question to re-balance the cluster before adding the new machine. Since then the cluster is operating properly, but I noticed some handoffs are stalled now. I had similar situation awhile ago that was solved by simply forcing the handoffs, but this time the same approach didn't work. Any ideas, solutions or just hints are greatly appreciated. Below are cluster statuses. Replaced the IP addresses for security reason.
~# riak-admin member_status Attempting to restart script through sudo -u riak ================================= Membership ================================== Status Ring Pending Node ------------------------------------------------------------------------------- valid 45.3% 34.4% 'r...@aaa.aaa.aaa.aaa' valid 26.6% 32.8% 'r...@bbb.bbb.bbb.bbb' valid 28.1% 32.8% 'r...@ccc.ccc.ccc.ccc' ------------------------------------------------------------------------------- Valid:3 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 ~# riak-admin ring_status Attempting to restart script through sudo -u riak ================================== Claimant =================================== Claimant: 'r...@aaa.aaa.aaa.aaa' Status: up Ring Ready: true ============================== Ownership Handoff ============================== Owner: r...@aaa.aaa.aaa.aaa Next Owner: r...@bbb.bbb.bbb.bbb Index: 22835963083295358096932575511191922182123945984 Waiting on: [riak_kv_vnode] Complete: [riak_pipe_vnode] Index: 570899077082383952423314387779798054553098649600 Waiting on: [riak_kv_vnode] Complete: [riak_pipe_vnode] Index: 1118962191081472546749696200048404186924073353216 Waiting on: [riak_kv_vnode] Complete: [riak_pipe_vnode] Index: 1392993748081016843912887106182707253109560705024 Waiting on: [riak_kv_vnode] Complete: [riak_pipe_vnode] ------------------------------------------------------------------------------- Owner: r...@aaa.aaa.aaa.aaa Next Owner: r...@ccc.ccc.ccc.ccc Index: 114179815416476790484662877555959610910619729920 Waiting on: [riak_kv_vnode] Complete: [riak_pipe_vnode] Index: 662242929415565384811044689824565743281594433536 Waiting on: [riak_kv_vnode] Complete: [riak_pipe_vnode] Index: 1210306043414653979137426502093171875652569137152 Waiting on: [riak_kv_vnode] Complete: [riak_pipe_vnode] ------------------------------------------------------------------------------- ============================== Unreachable Nodes ============================== All nodes are up and reachable Thanks in advance, Ivaylo -- Ivaylo Panitchkov Software developer Hibernum Creations Inc. Ce courriel est confidentiel et peut aussi être protégé par la loi.Si vous avez reçu ce courriel par erreur, veuillez nous en aviser immédiatement en y répondant, puis supprimer ce message de votre système. Veuillez ne pas le copier, l’utiliser pour quelque raison que ce soit ni divulguer son contenu à quiconque. This email is confidential and may also be legally privileged. If you have received this email in error, please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purpose or disclose its content.
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
