We are seeing similar issues using levelDB with 1.0.0RC1
We run a 6 way cluster on EC2 64bit micro instances.
Initially we had assumed that there may be some issue with small memory
environments.
on 0.14.2 things would seem to balance out and stabilize.
on 1.0.0RC1, once this sort of report starts showing up, it tends to run out of
control until the node crashes.
We also use a multi backend configuration as follows:
{storage_backend, riak_kv_multi_backend},
{multi_backend_default, <<"eleveldb">>},
{multi_backend, [
{<<"eleveldb">>, riak_kv_eleveldb_backend, []},
{<<"sessions">>, riak_kv_bitcask_backend, [
{expiry_secs, 7200},
{data_root, "/riak/data/bitcask_sessions"}]}
]},
We are now storing over 150,000 keys in the levelDB buckets
17:17:03.282 [error] CRASH REPORT Process [] with 0 neighbours crashed with
reason:
{timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,122743301572712549771012593372656581728916209664,'[email protected]','[email protected]',riak_pipe_vnode}]}}
17:17:04.299 [error] Supervisor riak_core_vnode_sup had child undefined started
with {riak_core_vnode,start_link,undefined} at <0.16220.0> exit with reason
{timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,122743301572712549771012593372656581728916209664,'[email protected]','[email protected]',riak_pipe_vnode}]}}
in context child_terminated
17:17:05.814 [info] monitor long_gc <0.44.0>
[{name,lager_crash_log},{initial_call,{lager_crash_log,init,1}},{almost_current_function,{gen,do_call,4}}]
[{timeout,1501},{old_heap_block_size,0},{heap_block_size,1597},{mbuf_size,0},{stack_size,285},{old_heap_size,0},{heap_size,1179}]
17:17:07.349 [error] gen_fsm <0.16589.0> in state active terminated with
reason:
{timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,161992613122126446500115457532517697979441741824,'[email protected]','[email protected]',riak_pipe_vnode}]}}
17:17:07.351 [info] monitor long_gc <0.101.0>
[{name,riak_core_ring_manager},{initial_call,{gen,init_it,7}},{almost_current_function,{erl_syntax,is_tree,1}}]
[{timeout,1510},{old_heap_block_size,0},{heap_block_size,317811},{mbuf_size,0},{stack_size,2363},{old_heap_size,0},{heap_size,275697}]
17:17:10.345 [error] CRASH REPORT Process [] with 0 neighbours crashed with
reason:
{timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,161992613122126446500115457532517697979441741824,'[email protected]','[email protected]',riak_pipe_vnode}]}}
17:17:10.348 [info] monitor long_gc <0.17891.0>
[{initial_call,{erlang,apply,2}},{almost_current_function,{erl_scan,set_attribute,3}}]
[{timeout,1481},{old_heap_block_size,0},{heap_block_size,121393},{mbuf_size,0},{stack_size,99},{old_heap_size,0},{heap_size,116704}]
17:17:11.847 [error] gen_server <0.16590.0> terminated with reason:
{timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,161992613122126446500115457532517697979441741824,'[email protected]','[email protected]',riak_pipe_vnode}]}}
17:17:13.339 [info] monitor long_gc <0.17891.0>
[{initial_call,{erlang,apply,2}},{almost_current_function,{erl_expand_records,expr,2}}]
[{timeout,1473},{old_heap_block_size,0},{heap_block_size,317811},{mbuf_size,0},{stack_size,1364},{old_heap_size,0},{heap_size,151085}]
17:17:13.342 [error] CRASH REPORT Process [] with 0 neighbours crashed with
reason:
{timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,161992613122126446500115457532517697979441741824,'[email protected]','[email protected]',riak_pipe_vnode}]}}
17:17:14.842 [error] Supervisor riak_core_vnode_sup had child undefined started
with {riak_core_vnode,start_link,undefined} at <0.16589.0> exit with reason
{timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,161992613122126446500115457532517697979441741824,'[email protected]','[email protected]',riak_pipe_vnode}]}}
in context child_terminated
17:17:14.857 [error] gen_fsm <0.15083.0> in state active terminated with
reason:
{timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,72076008481650973993443441457199504387328704512,'[email protected]','[email protected]',riak_kv_vnode}]}}
17:17:16.338 [info] monitor long_gc <0.17891.0>
[{initial_call,{erlang,apply,2}},{almost_current_function,{zlib,deflateInit,2}}]
[{timeout,1470},{old_heap_block_size,0},{heap_block_size,75025},{mbuf_size,0},{stack_size,34},{old_heap_size,0},{heap_size,36434}]
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
