Hey list,
We're migrating from our currently operational cluster to a new one in a
different EC2 region. As part of this migration, we'd like to move from
Bitcask to eLevelDB - mostly for the benefits provided by secondary indexes.
We also use Multi-backend configuration, in order to split our various
client's data into entirely separate spaces on disk (this is useful for
legal reasons). I made the simple change from "riak_kv_bitcask_backend" to
"riak_kv_eleveldb_backend" in our config, and did some calculations for
"max_open_files", and then started a new node.
It fails with the following error in short order:
> cat error.log
2013-09-09 03:49:51.391 [error] <0.715.0>@riak_kv_vnode:init:375 Failed to
start riak_kv_multi_backend Reason: [{riak_kv_eleveldb_backend,{db_open,"IO
error:
/mnt/riak/eleveldb/client1/365375409332725729550921208179070754913983135744/000015.dbtmp:
Too many open files"}}]
2013-09-09 03:49:51.392 [error] <0.940.0>@riak_kv_vnode:init:375 Failed to
start riak_kv_multi_backend Reason: [{riak_kv_eleveldb_backend,{db_open,"IO
error:
/mnt/riak/eleveldb/client2/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files"}},{riak_kv_eleveldb_backend,{db_open,"IO error:
/mnt/riak/eleveldb/client3/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files"}},{riak_kv_eleveldb_backend,{db_open,"IO error:
/mnt/riak/eleveldb/client4/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files"}},{riak_kv_eleveldb_backend,{db_open,"IO error:
/mnt/riak/eleveldb/client5/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files"}},{riak_kv_eleveldb_backend,{db_open,"IO error:
/mnt/riak/eleveldb/client6/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files"}}]
And some relevant app.config:
{multi_backend_default, <<"default">>},
{multi_backend, [
%% Default fallback: this is unused
{<<"default">>, riak_kv_eleveldb_backend, [
{data_root, "/mnt/riak/eleveldb/default"}
]},
{<<"eleveldb_client1">>, riak_kv_eleveldb_backend, [
{data_root, "/mnt/riak/eleveldb/client1"}
]},
{<<"eleveldb_client2">>, riak_kv_eleveldb_backend, [
{data_root, "/mnt/riak/eleveldb/client2"}
]},
{<<"eleveldb_client3">>, riak_kv_eleveldb_backend, [
{data_root, "/mnt/riak/eleveldb/client3"}
]},
{<<"eleveldb_client4">>, riak_kv_eleveldb_backend, [
{data_root, "/mnt/riak/eleveldb/client4"}
]},
{<<"eleveldb_client5">>, riak_kv_eleveldb_backend, [
{data_root, "/mnt/riak/eleveldb/client5"}
]},
{<<"eleveldb_client6">>, riak_kv_eleveldb_backend, [
{data_root, "/mnt/riak/eleveldb/client6"}
]}
]},
.. snip ..
%% Default cache size of 8MB
{cache_size, 8388608},
%% Maximum number of files open at once per partition
{max_open_files, 50}
I've set the "riak soft/hard nofile 65536" in /etc/security/limits.conf, so
presumably this "Too many open files" error is referring to the
"max_open_files" option as part of eLevelDB config. The RAM in each machine
is 3.5GB free, so I calculated on a 5 node cluster this 50 max_open_files
limit.
- Is there something about Multi-backend I haven't taken into account?
- Do I need a larger max_open_files? (And thus more RAM? :)
Cheers!
Matt
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
