Hi Allen, What version of erlang are you running, please? Cheers
Russell > On 8 Oct 2015, at 19:58, Allen Landsidel <landsidel.al...@gmail.com> wrote: > > Background: > Riak 2.1.1 > FreeBSD 9.1. > Servers are all virtualized on VMWare ESX 5.5. Each node is given ~300G of > storage, 4G of RAM, and 2 core SMP. > Storage is via FC SAN. > Access to the cluster from clients is strictly over the HTTP interface and is > funneled through haproxy. > The cluster was five nodes using leveldb with a ring size of 16. > > The cluster is setup this way so that we could start small and add nodes as > needed, with a small memory footprint, rather than preallocating a ton of > memory and disk space that we may never use. The cluster has been running > fine for the past month. > > Today the cluster has begun experiencing strange repeated failures. No > changes in any of the clients have been made, so the problem seems as though > it must be in the data coming in. > > I started a fresh two-node cluster and shortly after data begins flowing into > the cluster, nodes begin crashing. > > The first report in the crash log is below. I'm far from an expert at > understanding these errors, but "invalid_unicode" seems telling. > > If more from the crash log is needed, I can provide it. Once this first > error comes in, the node goes offline and errors just continue to roll in to > the log file, in what looks like an attempt to automatically restart the node. > > -------------------------------------------- > ** Generic server <0.2136.0> terminating > ** Last message in was {vnodeid,10000} > ** When Server state == {state,"/var/db/riak/kv_vnode/0",0,<0.2135.0>,2} > ** Reason for termination == > ** > {{badmatch,{error,{4,file_io_server,invalid_unicode}}},[{riak_kv_vnode_status_mgr,handle_call,3,[{file,"src/riak_kv_vnode_status_mgr.erl"},{line,178}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,607}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,639}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]} > 2015-10-08 18:36:18 =CRASH REPORT==== > crasher: > initial call: riak_kv_vnode_status_mgr:init/1 > pid: <0.2136.0> > registered_name: [] > exception exit: > {{{badmatch,{error,{4,file_io_server,invalid_unicode}}},[{riak_kv_vnode_status_mgr,handle_call,3,[{file,"src/riak_kv_vnode_status_mgr.erl"},{line,178}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,607}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,639}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]},[{gen_server,terminate,7,[{file,"gen_server.erl"},{line,804}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]} > ancestors: [<0.2135.0>,riak_core_vnode_sup,riak_core_sup,<0.169.0>] > messages: [] > links: [<0.2135.0>] > dictionary: [] > trap_exit: false > status: running > heap_size: 610 > stack_size: 27 > reductions: 569 > neighbours: > -------------------------------------------- > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com