> There's a code snippet in riak 1.0.1 or 1.0.2 release notes which addresses > this. Sorry can't find it for you, network here is useless. :(
The code snippet in the release notes deals with a different issue. The case where Riak believes all handoff has already occurred, but refuses to actually change partition ownership. This case is easily determined by seeing cases of "Waiting on: []" in 'riak-admin ring_status'. In other words, waiting on the empty-set, on nothing. The issue reported here is different. Ring status shows 'Waiting on: [riak_kv_vnode]'. This is an issue with the kv-vnode not triggering handoff. The solutions outlined above should work for this simpler case. For the curious, or those that ever run into the 'Waiting on: []" issue, the snippet Kyle mentioned can be found in the "Ownership Handoff Stall" section here: https://github.com/basho/riak/blob/1.0.2-release/RELEASE-NOTES.org -Joe On Tue, Jan 10, 2012 at 9:40 AM, Aphyr <[email protected]> wrote: > There's a code snippet in riak 1.0.1 or 1.0.2 release notes which addresses > this. Sorry can't find it for you, network here is useless. :( > > Ivaylo Panitchkov <[email protected]> wrote: > >> >>Hello All, >> >>We have a cluster of three machines (Debian 6.0, 4GB RAM, >>riak_1.0.2-1_amd64.deb, n_val: 3) that serves an application for a >>while. As we go to production soon added a fourth machine to the cluster >>(exactly the same as the first three) yesterday. The partition handoff >>began in the late afternoon and I had an impression that the transition >>will not take too long as there are only few hundred IMPORTANT records >>in the storage for the moment. Today in the morning checked the >>situation again and realized the partition handoff still runs (or get >>stuck). The Ownership Handoff is still the same since yesterday (at >>least 19 hours till now). Any suggestions to fix the problem are welcome :-) >> >>REMARK: Replaced the IP addresses for security sake >> >> >># riak-admin ringready >>Attempting to restart script through sudo -u riak >>TRUE All nodes agree on the ring >>['[email protected]','[email protected]','[email protected]','[email protected]'] >> >> >># riak-admin transfers >>Attempting to restart script through sudo -u riak >>'[email protected]' waiting to handoff 2 partitions >>'[email protected]' waiting to handoff 2 partitions >>'[email protected]' waiting to handoff 2 partitions >> >> >># riak-admin ring_status >>Attempting to restart script through sudo -u riak >>================================== Claimant >>=================================== >>Claimant: '[email protected]' >>Status: up >>Ring Ready: true >> >>============================== Ownership Handoff >>============================== >>Owner: [email protected] >>Next Owner: [email protected] >> >>Index: 548063113999088594326381812268606132370974703616 >>Waiting on: [riak_kv_vnode] >>Complete: [riak_pipe_vnode] >> >>Index: 1370157784997721485815954530671515330927436759040 >>Waiting on: [riak_kv_vnode] >>Complete: [riak_pipe_vnode] >> >>------------------------------------------------------------------------------- >> >>============================== Unreachable Nodes >>============================== >>All nodes are up and reachable >> >> >># riak-admin member_status >>Attempting to restart script through sudo -u riak >>================================= Membership >>================================== >>Status Ring Pending Node >>------------------------------------------------------------------------------- >>valid 21.9% 25.0% '[email protected]' >>valid 28.1% 25.0% '[email protected]' >>valid 25.0% 25.0% '[email protected]' >>valid 25.0% 25.0% '[email protected]' >>------------------------------------------------------------------------------- >>Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 >> >> >>-- >>Ivaylo Panitchkov >>Software developer >>Hibernum Creations Inc. >> >>Ce courriel est confidentiel et peut aussi être protégé par la loi.Si vous >>avez reçu ce courriel par erreur, veuillez nous en aviser immédiatement en y >>répondant, puis supprimer ce message de votre système. Veuillez ne pas le >>copier, l’utiliser pour quelque raison que ce soit ni divulguer son contenu à >>quiconque. >>This email is confidential and may also be legally privileged. If you have >>received this email in error, please notify us immediately by reply email and >>then delete this message from your system. Please do not copy it or use it >>for any purpose or disclose its content. >> >> >>_______________________________________________ >>riak-users mailing list >>[email protected] >>http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com -- Joseph Blomstedt <[email protected]> Software Engineer Basho Technologies, Inc. http://www.basho.com/ _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
