Hi Magnus Kessler, The configuration looks good.
[root@server205 bin]# ./riak config effective | grep "_dir" anti_entropy.data_dir = $(platform_data_dir)/anti_entropy bitcask.data_root = $(platform_data_dir)/bitcask leveldb.data_root = $(platform_data_dir)/leveldb log.console.file = $(platform_log_dir)/console.log log.crash.file = $(platform_log_dir)/crash.log log.error.file = $(platform_log_dir)/error.log platform_bin_dir = ./bin platform_data_dir = ./data platform_etc_dir = ./etc platform_lib_dir = ./lib platform_log_dir = ./log ring.state_dir = $(platform_data_dir)/ring search.anti_entropy.data_dir = $(platform_data_dir)/yz_anti_entropy search.root_dir = $(platform_data_dir)/yz search.temp_dir = $(platform_data_dir)/yz_temp Regards, Jerald Emerging Products and Technologies | Avaya Voice +91 44 6108 4258 | ajer...@avaya.com<mailto:ajer...@avaya.com> | Chennai, India From: Magnus Kessler [mailto:mkess...@basho.com] Sent: Thursday, May 04, 2017 12:36 PM To: Arulappan, Jerald (Jerald) **CTR** <ajer...@avaya.com> Cc: riak-users@lists.basho.com Subject: Re: Riak 2.1.4 crashes with Out of Memory Error On 2 May 2017 at 14:56, Arulappan, Jerald (Jerald) <ajer...@avaya.com<mailto:ajer...@avaya.com>> wrote: Hi, I am using a single node riak server 2.1.4 with bitcask as backend for storing files. The riak node stops working after every week. (Looks like when the active anti-entropy process recreates the hash tree) The sylog shows Out of memory Error. But the console.log shows "sst: No such file or directory" Syslog Error: Apr 26 17:39:37 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child Apr 26 17:39:37 TLCCBAPRO2 kernel: Killed process 16987 (sh) total-vm:106168kB, anon-rss:116kB, file-rss:0kB Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 30374 (memsup) total-vm:4112kB, anon-rss:80kB, file-rss:0kB Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 14351 (cpu_sup) total-vm:4112kB, anon-rss:68kB, file-rss:0kB Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 30385 (sh) total-vm:106164kB, anon-rss:136kB, file-rss:416kB Apr 26 17:44:48 TLCCBAPRO2 run_erl[16682]: Erlang closed the connection. Console.log: 2017-04-26 17:37:03.493 [info] <0.625.0>@riak_kv_vnode:maybe_create_hashtrees:227 riak_kv/91343852333181432387730302044767688728495783936: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: ./data/anti_entropy/91343852333181432387730302044767688728495783936/sst_0/001954.sst: No such file or directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}} 2017-04-26 17:37:03.515 [error] <0.30178.2881> CRASH REPORT Process <0.30178.2881> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: ./data/anti_entropy/936274486415109681974235595958868809467081785344/000037.sst: No such file or directory"}} in hashtree:new_segment_store/2 line 675 in gen_server:init_it/6 line 328 2017-04-26 17:37:03.515 [info] <0.623.0>@riak_kv_vnode:maybe_create_hashtrees:227 riak_kv/45671926166590716193865151022383844364247891968: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst: No such file or directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}} 2017-04-26 17:37:03.516 [error] <0.30207.2881> CRASH REPORT Process <0.30207.2881> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst: No such file or directory"}} in hashtree:new_segment_store/2 line 675 in gen_server:init_it/6 line 328 The complete logs are in the attached zip file. Any thoughts on the root cause and possible solution to overcome this is much appreciated. Regards, Jerald Hi Jerald, I suspect that there is a mis-configuration on your setup. Please check by running `riak config effective | grep "_dir"`, what the values of `platform_data_dir` and `anti_entropy.data_dir` are set to. Kind Regards, Magnus -- Magnus Kessler Client Services Engineer Basho Technologies Limited Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
