We recently upgraded to Riak 2.0.4 and started seeing the thousands of the
following error in one of our nodes:
2015-02-18 07:48:16.652 [error] <0.23966.264> CRASH REPORT Process
<0.23966.264> with 0 neighbours exited with reason: no match of right hand
value {error,{db_open,"IO error: lock
/srv/riak-0/data/anti_entropy/548063113999088594326381812268606132370974703616/LOCK:
already held by process"}} in hashtree:new_segment_store/2 line 555 in
gen_server:init_it/6 line 328
2015-02-18 07:48:16.652 [info]
<0.27823.1997>@riak_kv_vnode:maybe_create_hashtrees:178
riak_kv/548063113999088594326381812268606132370974703616: unable to start
index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: lock
/srv/riak-0/data/anti_entropy/548063113999088594326381812268606132370974703616/LOCK:
already held by
process"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,555}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,224}]},{riak_kv_index_hashtree,do_new_tree,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,560}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,430}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,255}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}
This correlates with a massive spike in GET requests on the three nodes
that own the affected vnode.
Anyone have any ideas?
Chris
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
