Giri, I've seen similar issues in the past when someone was adjusting their ttl setting on the memory backend. Because one memory backend has it and the other does not, it fails on handoff. The solution then was to make sure that all memory backend settings are the same and then do a rolling restart of the cluster (ignoring a lot of errors along the way). I am not sure that this is applicable to your case, but it's something to look at.
On Thu, Mar 28, 2013 at 10:22 AM, Giri Iyengar <[email protected]> wrote: > Godefroy: > > Thanks. Your email exchange on the mailing list was what prompted me to > consider switching to Riak 1.3. I do see repair messages in the console logs > and so some healing is happening. However, there are a bunch of hinted > handoffs and ownership handoffs that are simply not proceeding because the > same vnodes keep coming up for transfer and fail. Perhaps there is a manual > way to forcibly repair and push the vnodes around. > > -giri > > > On Thu, Mar 28, 2013 at 1:19 PM, Godefroy de Compreignac > <[email protected]> wrote: >> >> I have exactly the same problem with my cluster. If anyone knows what >> those errors mean... :-) >> >> Godefroy >> >> >> 2013/3/28 Giri Iyengar <[email protected]> >>> >>> Hello, >>> >>> We are running a 6-node Riak 1.3.0 cluster in production. We recently >>> upgraded to 1.3. Prior to this, we were running Riak 1.2 on the same 6-node >>> cluster. >>> >>> We are finding that the nodes are not balanced. For instance: >>> >>> ================================= Membership >>> ================================== >>> Status Ring Pending Node >>> >>> ------------------------------------------------------------------------------- >>> valid 0.0% 0.0% '[email protected]' >>> valid 34.4% 20.3% '[email protected]' >>> valid 21.9% 20.3% '[email protected]' >>> valid 19.5% 20.3% '[email protected]' >>> valid 8.6% 19.5% '[email protected]' >>> valid 15.6% 19.5% '[email protected]' >>> >>> ------------------------------------------------------------------------------- >>> Valid:6 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 >>> >>> >>> When we look at the logs in the largest node ([email protected]), we see >>> error messages that look like this: >>> >>> 2013-03-28 13:04:16.957 [error] >>> <0.10957.1462>@riak_core_handoff_sender:start_fold:226 hinted_handoff >>> transfer of riak_kv_vnode from '[email protected]' >>> 148433760041419827630061740822747494183805648896 to '[email protected]' >>> 148433760041419827630061740822747494183805648896 failed because of >>> error:{badmatch,{error,{worker_crash,{function_clause,[{riak_core_pb,encode,[{ts,{1364,476737,222223}},{{ts,{1364,476737,222223}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,11,69,78,84,73,84,89,95,83,69,83,83,109,0,0,0,36,67,54,57,95,48,48,51,56,100,56,102,50,52,49,52,99,97,97,54,102,99,52,56,53,52,99,99,101,51,98,50,48,102,53,98,52,108,0,0,0,1,104,3,100,0,9,114,95,99,111,110,116,101,110,116,104,9,100,0,4,100,105,99,116,97,5,97,16,97,16,97,8,97,80,97,48,104,16,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,104,1,104,16,106,106,106,106,106,106,106,106,106,106,108,0,0,0,2,108,0,0,0,11,109,0,0,0,12,99,111,110,116,101,110,116,45,116,121,112,101,97,116,97,101,97,120,97,116,97,47,97,112,97,108,97,97,97,105,97,110,106,108,0,0,0,23,109,0,0,0,11,88,45,82,105,97,107,45,86,84,97,103,97,51,97,120,97,105,97,101,97,120,97,66,97,120,97,107,97,119,97,101,97,75,97,117,97,122,97,111,97,55,97,85,97,104,97,85,97,107,97,112,97,120,97,107,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,105,110,100,101,120,106,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,20,88,45,82,105,97,107,45,76,97,115,116,45,77,111,100,105,102,105,101,100,104,3,98,0,0,5,84,98,0,7,70,65,98,0,3,99,115,106,106,108,0,0,0,1,108,0,0,0,6,109,0,0,0,7,99,104,97,114,115,101,116,97,85,97,84,97,70,97,45,97,56,106,106,109,0,0,0,36,52,54,55,98,54,51,98,50,45,50,99,56,52,45,52,56,50,99,45,97,48,99,54,45,56,53,50,100,53,99,57,97,98,98,53,101,106,108,0,0,0,1,104,2,109,0,0,0,8,0,69,155,215,81,84,63,31,104,2,97,1,110,5,0,65,191,200,202,14,106,104,9,100,0,4,100,105,99,116,97,1,97,16,97,16,97,8,97,80,97,48,104,16,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,104,1,104,16,106,106,106,106,106,106,106,106,106,106,106,106,106,106,108,0,0,0,1,108,0,0,0,1,100,0,5,99,108,101,97,110,100,0,4,116,114,117,101,106,106,100,0,9,117,110,100,101,102,105,110,101,100>>}],[{file,"src/riak_core_pb.erl"},{line,40}]},{riak_core_pb,pack,5,...},...]},...}}} >>> [{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,170}]}] >>> 2013-03-28 13:04:16.961 [error] <0.29352.909> CRASH REPORT Process >>> <0.29352.909> with 0 neighbours exited with reason: no function clause >>> matching riak_core_pb:encode({ts,{1364,476737,222223}}, >>> {{ts,{1364,476737,222223}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,11,69,78,...>>}) >>> line 40 in gen_server:terminate/6 line 747 >>> >>> >>> 2013-03-28 13:04:13.888 [error] >>> <0.12680.1435>@riak_core_handoff_sender:start_fold:226 ownership_handoff >>> transfer of riak_kv_vnode from '[email protected]' >>> 11417981541647679048466287755595961091061972992 to '[email protected]' >>> 11417981541647679048466287755595961091061972992 failed because of >>> error:{badmatch,{error,{worker_crash,{function_clause,[{riak_core_pb,encode,[{ts,{1364,458917,232318}},{{ts,{1364,458917,232318}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,11,69,78,84,73,84,89,95,83,69,83,83,109,0,0,0,36,67,54,57,95,48,48,48,54,52,98,99,52,53,51,49,52,55,101,50,101,53,97,102,101,102,49,57,99,50,55,99,97,49,53,54,99,108,0,0,0,1,104,3,100,0,9,114,95,99,111,110,116,101,110,116,104,9,100,0,4,100,105,99,116,97,5,97,16,97,16,97,8,97,80,97,48,104,16,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,104,1,104,16,106,106,106,106,106,106,106,106,106,106,108,0,0,0,2,108,0,0,0,11,109,0,0,0,12,99,111,110,116,101,110,116,45,116,121,112,101,97,116,97,101,97,120,97,116,97,47,97,112,97,108,97,97,97,105,97,110,106,108,0,0,0,23,109,0,0,0,11,88,45,82,105,97,107,45,86,84,97,103,97,54,97,88,97,76,97,66,97,69,97,69,97,116,97,73,97,104,97,118,97,77,97,86,97,48,97,81,97,103,97,110,97,119,97,73,97,51,97,85,97,72,97,53,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,105,110,100,101,120,106,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,20,88,45,82,105,97,107,45,76,97,115,116,45,77,111,100,105,102,105,101,100,104,3,98,0,0,5,84,98,0,7,0,165,98,0,3,138,179,106,106,108,0,0,0,1,108,0,0,0,6,109,0,0,0,7,99,104,97,114,115,101,116,97,85,97,84,97,70,97,45,97,56,106,106,109,0,0,0,36,55,102,98,52,50,54,54,53,45,57,100,56,48,45,52,54,98,97,45,98,53,97,100,45,56,55,52,52,54,54,97,97,50,56,53,99,106,108,0,0,0,1,104,2,109,0,0,0,8,0,69,155,215,81,59,179,219,104,2,97,1,110,5,0,165,121,200,202,14,106,104,9,100,0,4,100,105,99,116,97,1,97,16,97,16,97,8,97,80,97,48,104,16,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,104,1,104,16,106,106,106,106,106,106,106,106,106,106,106,106,106,106,108,0,0,0,1,108,0,0,0,1,100,0,5,99,108,101,97,110,100,0,4,116,114,117,101,106,106,100,0,9,117,110,100,101,102,105,110,101,100>>}],[{file,"src/riak_core_pb.erl"},{line,40}]},{riak_core_pb,pack,5,[{...},...]},...]},...}}} >>> [{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,170}]}] >>> 2013-03-28 13:04:14.255 [error] <0.1120.0> CRASH REPORT Process >>> <0.1120.0> with 0 neighbours exited with reason: no function clause matching >>> riak_core_pb:encode({ts,{1364,458917,232318}}, >>> {{ts,{1364,458917,232318}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,11,69,78,...>>}) >>> line 40 in gen_server:terminate/6 line 747 >>> >>> This has been going on for days and the cluster doesn't seem to be >>> rebalancing itself. We see this issue with both hinted_handoffs and >>> ownership_handoffs. Looks like we have some corrupt data in our cluster. I >>> checked through the leveldb LOGs and did not see any compaction errors. >>> I was hoping that upgrading to 1.3.0 will slowly start repairing the >>> cluster. However, that doesn't seem to be happening. >>> >>> Any help/hints would be much appreciated. >>> >>> -giri >>> -- >>> GIRI IYENGAR, CTO >>> SOCIOCAST >>> Simple. Powerful. Predictions. >>> >>> 36 WEST 25TH STREET, 7TH FLOOR NEW YORK CITY, NY 10010 >>> O: 917.525.2466x104 M: 914.924.7935 F: 347.943.6281 >>> E: [email protected] W: www.sociocast.com >>> >>> Facebook's Ad Guru Joins Sociocast - http://bit.ly/NjPQBQ >>> >>> _______________________________________________ >>> riak-users mailing list >>> [email protected] >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >> > > > > -- > GIRI IYENGAR, CTO > SOCIOCAST > Simple. Powerful. Predictions. > > 36 WEST 25TH STREET, 7TH FLOOR NEW YORK CITY, NY 10010 > O: 917.525.2466x104 M: 914.924.7935 F: 347.943.6281 > E: [email protected] W: www.sociocast.com > > Facebook's Ad Guru Joins Sociocast - http://bit.ly/NjPQBQ > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
