Hi Allen,
What version of erlang are you running, please?

Cheers

Russell

> On 8 Oct 2015, at 19:58, Allen Landsidel <landsidel.al...@gmail.com> wrote:
> 
> Background:
> Riak 2.1.1
> FreeBSD 9.1.
> Servers are all virtualized on VMWare ESX 5.5.  Each node is given ~300G of 
> storage, 4G of RAM, and 2 core SMP.
> Storage is via FC SAN.
> Access to the cluster from clients is strictly over the HTTP interface and is 
> funneled through haproxy.
> The cluster was five nodes using leveldb with a ring size of 16.
> 
> The cluster is setup this way so that we could start small and add nodes as 
> needed, with a small memory footprint, rather than preallocating a ton of 
> memory and disk space that we may never use.  The cluster has been running 
> fine for the past month.
> 
> Today the cluster has begun experiencing strange repeated failures.  No 
> changes in any of the clients have been made, so the problem seems as though 
> it must be in the data coming in.
> 
> I started a fresh two-node cluster and shortly after data begins flowing into 
> the cluster, nodes begin crashing.
> 
> The first report in the crash log is below.  I'm far from an expert at 
> understanding these errors, but "invalid_unicode" seems telling.
> 
> If more from the crash log is needed, I can provide it.  Once this first 
> error comes in, the node goes offline and errors just continue to roll in to 
> the log file, in what looks like an attempt to automatically restart the node.
> 
> --------------------------------------------
> ** Generic server <0.2136.0> terminating
> ** Last message in was {vnodeid,10000}
> ** When Server state == {state,"/var/db/riak/kv_vnode/0",0,<0.2135.0>,2}
> ** Reason for termination ==
> ** 
> {{badmatch,{error,{4,file_io_server,invalid_unicode}}},[{riak_kv_vnode_status_mgr,handle_call,3,[{file,"src/riak_kv_vnode_status_mgr.erl"},{line,178}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,607}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,639}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]}
> 2015-10-08 18:36:18 =CRASH REPORT====
>  crasher:
>    initial call: riak_kv_vnode_status_mgr:init/1
>    pid: <0.2136.0>
>    registered_name: []
>    exception exit: 
> {{{badmatch,{error,{4,file_io_server,invalid_unicode}}},[{riak_kv_vnode_status_mgr,handle_call,3,[{file,"src/riak_kv_vnode_status_mgr.erl"},{line,178}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,607}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,639}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]},[{gen_server,terminate,7,[{file,"gen_server.erl"},{line,804}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]}
>    ancestors: [<0.2135.0>,riak_core_vnode_sup,riak_core_sup,<0.169.0>]
>    messages: []
>    links: [<0.2135.0>]
>    dictionary: []
>    trap_exit: false
>    status: running
>    heap_size: 610
>    stack_size: 27
>    reductions: 569
>  neighbours:
> --------------------------------------------
> 
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Reply via email to