Hi Dave,
Just to confirm that the ulimit settings "stuck", could you please run riak
attach and execute the following Erlang snippet?
os:cmd('ulimit -n').
The period is significant and please exit using CTRL-C twice.
Thanks!
--
Luke Bakken
CSE
[email protected]
On Mon, Nov 11, 2013 at 11:56 AM, Dave Brady <[email protected]> wrote:
> Hey Everyone,
>
> We have a five-node, 128 partition cluster running 1.4.2 on Debian.
>
> Is there a doc somewhere that explains how to size max_open_files as it
> applies to AAE?
>
> I have max_open_files for eLevelDB set to 3000, as we have about 1500 .sst
> files in one VNode's data directory, and the boxes have plenty of RAM.
>
> I set max_open_files in the AAE section to 3000, too, on whim after we had
> our first issue. Still got these in the logs on a couple of nodes after
> running for less than one day:
>
> ===============
> 2013-11-09 11:37:12.438 [info]
> <0.857.0>@riak_kv_vnode:maybe_create_hashtrees:142
> riak_kv/125597796958124469533129165311555572001681702912: unable to start
> index_hashtree: {error,{{badmatch,{error,{db_open,"IO error:
> /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK:
> Too many open
> files"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,499}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},{riak_kv_index_hashtree,do_new_tree,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,426}]},{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,366}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,226}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}}
> 2013-11-09 11:37:12.441 [error] <0.5209.2422> gen_server <0.5209.2422>
> terminated with reason: no match of right hand value {error,{db_write,"IO
> error:
> /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log:
> Too many open files"}} in hashtree:flush_buffer/1 line 302
> 2013-11-09 11:37:12.441 [error] <0.5209.2422> CRASH REPORT Process
> <0.5209.2422> with 1 neighbours exited with reason: no match of right hand
> value {error,{db_write,"IO error:
> /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log:
> Too many open files"}} in hashtree:flush_buffer/1 line 302 in
> gen_server:terminate/6 line 747
> 2013-11-09 11:37:12.441 [error] <0.19959.2426> CRASH REPORT Process
> <0.19959.2426> with 0 neighbours exited with reason: no match of right hand
> value {error,{db_open,"IO error:
> /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK:
> Too many open files"}} in hashtree:new_segment_store/2 line 499 in
> gen_server:init_it/6 line 328
> ===============
>
> Our init script has "ulimit -n 65536" in it, which I *thought* that would
> be high enough. Maybe not?
>
> I also made the necessary tweaks to /etc/pam.d/common-session*, so that
> /etc/security/limts.conf would be read, and that did not help.
>
> Much obliged for any suggestions!
> --
> Dave Brady
>
>
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
