Hi Matt -
Can you attach to a node in your cluster with "riak attach" and run
this erlang snippet (before the node errors out)?
os:cmd("ulimit -n").
This will show the actual value for ulimit for that running riak node.
I suspect that this value hasn't been updated for the running process,
even though you've modified limits.conf.
Depending on your OS there may be more configuration necessary for the
open files limit to "stick":
http://docs.basho.com/riak/latest/ops/tuning/open-files-limit/#Linux
On Sun, Sep 8, 2013 at 9:15 PM, Matt Black <matt.bl...@jbadigital.com> wrote:
> Hey list,
>
> We're migrating from our currently operational cluster to a new one in a
> different EC2 region. As part of this migration, we'd like to move from
> Bitcask to eLevelDB - mostly for the benefits provided by secondary indexes.
>
> We also use Multi-backend configuration, in order to split our various
> client's data into entirely separate spaces on disk (this is useful for
> legal reasons). I made the simple change from "riak_kv_bitcask_backend" to
> "riak_kv_eleveldb_backend" in our config, and did some calculations for
> "max_open_files", and then started a new node.
>
> It fails with the following error in short order:
>
>> cat error.log
> 2013-09-09 03:49:51.391 [error] <0.715.0>@riak_kv_vnode:init:375 Failed to
> start riak_kv_multi_backend Reason: [{riak_kv_eleveldb_backend,{db_open,"IO
> error:
> /mnt/riak/eleveldb/client1/365375409332725729550921208179070754913983135744/000015.dbtmp:
> Too many open files"}}]
> 2013-09-09 03:49:51.392 [error] <0.940.0>@riak_kv_vnode:init:375 Failed to
> start riak_kv_multi_backend Reason: [{riak_kv_eleveldb_backend,{db_open,"IO
> error:
> /mnt/riak/eleveldb/client2/570899077082383952423314387779798054553098649600/CURRENT:
> Too many open files"}},{riak_kv_eleveldb_backend,{db_open,"IO error:
> /mnt/riak/eleveldb/client3/570899077082383952423314387779798054553098649600/CURRENT:
> Too many open files"}},{riak_kv_eleveldb_backend,{db_open,"IO error:
> /mnt/riak/eleveldb/client4/570899077082383952423314387779798054553098649600/CURRENT:
> Too many open files"}},{riak_kv_eleveldb_backend,{db_open,"IO error:
> /mnt/riak/eleveldb/client5/570899077082383952423314387779798054553098649600/CURRENT:
> Too many open files"}},{riak_kv_eleveldb_backend,{db_open,"IO error:
> /mnt/riak/eleveldb/client6/570899077082383952423314387779798054553098649600/CURRENT:
> Too many open files"}}]
>
> And some relevant app.config:
>
> {multi_backend_default, <<"default">>},
> {multi_backend, [
> %% Default fallback: this is unused
> {<<"default">>, riak_kv_eleveldb_backend, [
> {data_root, "/mnt/riak/eleveldb/default"}
> ]},
> {<<"eleveldb_client1">>, riak_kv_eleveldb_backend, [
> {data_root, "/mnt/riak/eleveldb/client1"}
> ]},
> {<<"eleveldb_client2">>, riak_kv_eleveldb_backend, [
> {data_root, "/mnt/riak/eleveldb/client2"}
> ]},
> {<<"eleveldb_client3">>, riak_kv_eleveldb_backend, [
> {data_root, "/mnt/riak/eleveldb/client3"}
> ]},
> {<<"eleveldb_client4">>, riak_kv_eleveldb_backend, [
> {data_root, "/mnt/riak/eleveldb/client4"}
> ]},
> {<<"eleveldb_client5">>, riak_kv_eleveldb_backend, [
> {data_root, "/mnt/riak/eleveldb/client5"}
> ]},
> {<<"eleveldb_client6">>, riak_kv_eleveldb_backend, [
> {data_root, "/mnt/riak/eleveldb/client6"}
> ]}
> ]},
>
> .. snip ..
>
> %% Default cache size of 8MB
> {cache_size, 8388608},
> %% Maximum number of files open at once per partition
> {max_open_files, 50}
>
> I've set the "riak soft/hard nofile 65536" in /etc/security/limits.conf, so
> presumably this "Too many open files" error is referring to the
> "max_open_files" option as part of eLevelDB config. The RAM in each machine
> is 3.5GB free, so I calculated on a 5 node cluster this 50 max_open_files
> limit.
>
> - Is there something about Multi-backend I haven't taken into account?
> - Do I need a larger max_open_files? (And thus more RAM? :)
>
> Cheers!
> Matt
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
--
Luke Bakken
CSE
lbak...@basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
