Re: Multi-backend, eLevelDB, Too many open files
Hi Matt -
Can you attach to a node in your cluster with riak attach and run
this erlang snippet (before the node errors out)?
os:cmd(ulimit -n).
This will show the actual value for ulimit for that running riak node.
I suspect that this value hasn't been updated for the running process,
even though you've modified limits.conf.
Depending on your OS there may be more configuration necessary for the
open files limit to stick:
http://docs.basho.com/riak/latest/ops/tuning/open-files-limit/#Linux
On Sun, Sep 8, 2013 at 9:15 PM, Matt Black matt.bl...@jbadigital.com wrote:
Hey list,
We're migrating from our currently operational cluster to a new one in a
different EC2 region. As part of this migration, we'd like to move from
Bitcask to eLevelDB - mostly for the benefits provided by secondary indexes.
We also use Multi-backend configuration, in order to split our various
client's data into entirely separate spaces on disk (this is useful for
legal reasons). I made the simple change from riak_kv_bitcask_backend to
riak_kv_eleveldb_backend in our config, and did some calculations for
max_open_files, and then started a new node.
It fails with the following error in short order:
cat error.log
2013-09-09 03:49:51.391 [error] 0.715.0@riak_kv_vnode:init:375 Failed to
start riak_kv_multi_backend Reason: [{riak_kv_eleveldb_backend,{db_open,IO
error:
/mnt/riak/eleveldb/client1/365375409332725729550921208179070754913983135744/15.dbtmp:
Too many open files}}]
2013-09-09 03:49:51.392 [error] 0.940.0@riak_kv_vnode:init:375 Failed to
start riak_kv_multi_backend Reason: [{riak_kv_eleveldb_backend,{db_open,IO
error:
/mnt/riak/eleveldb/client2/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}},{riak_kv_eleveldb_backend,{db_open,IO error:
/mnt/riak/eleveldb/client3/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}},{riak_kv_eleveldb_backend,{db_open,IO error:
/mnt/riak/eleveldb/client4/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}},{riak_kv_eleveldb_backend,{db_open,IO error:
/mnt/riak/eleveldb/client5/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}},{riak_kv_eleveldb_backend,{db_open,IO error:
/mnt/riak/eleveldb/client6/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}}]
And some relevant app.config:
{multi_backend_default, default},
{multi_backend, [
%% Default fallback: this is unused
{default, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/default}
]},
{eleveldb_client1, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client1}
]},
{eleveldb_client2, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client2}
]},
{eleveldb_client3, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client3}
]},
{eleveldb_client4, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client4}
]},
{eleveldb_client5, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client5}
]},
{eleveldb_client6, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client6}
]}
]},
.. snip ..
%% Default cache size of 8MB
{cache_size, 8388608},
%% Maximum number of files open at once per partition
{max_open_files, 50}
I've set the riak soft/hard nofile 65536 in /etc/security/limits.conf, so
presumably this Too many open files error is referring to the
max_open_files option as part of eLevelDB config. The RAM in each machine
is 3.5GB free, so I calculated on a 5 node cluster this 50 max_open_files
limit.
- Is there something about Multi-backend I haven't taken into account?
- Do I need a larger max_open_files? (And thus more RAM? :)
Cheers!
Matt
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
--
Luke Bakken
CSE
lbak...@basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Multi-backend, eLevelDB, Too many open files
Hi Luke,
Thanks for the response.
I played around with the various config options - and in the end I had to
include the /etc/defaults/riak file, as well as the
/etc/security/limits.conf config in order to get Riak to stay up. That must
be relatively new - our existing v1.4.0 cluster is operating fine without
it.
Cheers
Matt
On 10 September 2013 01:32, Luke Bakken lbak...@basho.com wrote:
Hi Matt -
Can you attach to a node in your cluster with riak attach and run
this erlang snippet (before the node errors out)?
os:cmd(ulimit -n).
This will show the actual value for ulimit for that running riak node.
I suspect that this value hasn't been updated for the running process,
even though you've modified limits.conf.
Depending on your OS there may be more configuration necessary for the
open files limit to stick:
http://docs.basho.com/riak/latest/ops/tuning/open-files-limit/#Linux
On Sun, Sep 8, 2013 at 9:15 PM, Matt Black matt.bl...@jbadigital.com
wrote:
Hey list,
We're migrating from our currently operational cluster to a new one in a
different EC2 region. As part of this migration, we'd like to move from
Bitcask to eLevelDB - mostly for the benefits provided by secondary
indexes.
We also use Multi-backend configuration, in order to split our various
client's data into entirely separate spaces on disk (this is useful for
legal reasons). I made the simple change from riak_kv_bitcask_backend
to
riak_kv_eleveldb_backend in our config, and did some calculations for
max_open_files, and then started a new node.
It fails with the following error in short order:
cat error.log
2013-09-09 03:49:51.391 [error] 0.715.0@riak_kv_vnode:init:375 Failed
to
start riak_kv_multi_backend Reason:
[{riak_kv_eleveldb_backend,{db_open,IO
error:
/mnt/riak/eleveldb/client1/365375409332725729550921208179070754913983135744/15.dbtmp:
Too many open files}}]
2013-09-09 03:49:51.392 [error] 0.940.0@riak_kv_vnode:init:375 Failed
to
start riak_kv_multi_backend Reason:
[{riak_kv_eleveldb_backend,{db_open,IO
error:
/mnt/riak/eleveldb/client2/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}},{riak_kv_eleveldb_backend,{db_open,IO error:
/mnt/riak/eleveldb/client3/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}},{riak_kv_eleveldb_backend,{db_open,IO error:
/mnt/riak/eleveldb/client4/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}},{riak_kv_eleveldb_backend,{db_open,IO error:
/mnt/riak/eleveldb/client5/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}},{riak_kv_eleveldb_backend,{db_open,IO error:
/mnt/riak/eleveldb/client6/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}}]
And some relevant app.config:
{multi_backend_default, default},
{multi_backend, [
%% Default fallback: this is unused
{default, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/default}
]},
{eleveldb_client1, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client1}
]},
{eleveldb_client2, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client2}
]},
{eleveldb_client3, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client3}
]},
{eleveldb_client4, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client4}
]},
{eleveldb_client5, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client5}
]},
{eleveldb_client6, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client6}
]}
]},
.. snip ..
%% Default cache size of 8MB
{cache_size, 8388608},
%% Maximum number of files open at once per partition
{max_open_files, 50}
I've set the riak soft/hard nofile 65536 in /etc/security/limits.conf,
so
presumably this Too many open files error is referring to the
max_open_files option as part of eLevelDB config. The RAM in each
machine
is 3.5GB free, so I calculated on a 5 node cluster this 50 max_open_files
limit.
- Is there something about Multi-backend I haven't taken into account?
- Do I need a larger max_open_files? (And thus more RAM? :)
Cheers!
Matt
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
--
Luke Bakken
CSE
lbak...@basho.com
___
Multi-backend, eLevelDB, Too many open files
Hey list,
We're migrating from our currently operational cluster to a new one in a
different EC2 region. As part of this migration, we'd like to move from
Bitcask to eLevelDB - mostly for the benefits provided by secondary indexes.
We also use Multi-backend configuration, in order to split our various
client's data into entirely separate spaces on disk (this is useful for
legal reasons). I made the simple change from riak_kv_bitcask_backend to
riak_kv_eleveldb_backend in our config, and did some calculations for
max_open_files, and then started a new node.
It fails with the following error in short order:
cat error.log
2013-09-09 03:49:51.391 [error] 0.715.0@riak_kv_vnode:init:375 Failed to
start riak_kv_multi_backend Reason: [{riak_kv_eleveldb_backend,{db_open,IO
error:
/mnt/riak/eleveldb/client1/365375409332725729550921208179070754913983135744/15.dbtmp:
Too many open files}}]
2013-09-09 03:49:51.392 [error] 0.940.0@riak_kv_vnode:init:375 Failed to
start riak_kv_multi_backend Reason: [{riak_kv_eleveldb_backend,{db_open,IO
error:
/mnt/riak/eleveldb/client2/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}},{riak_kv_eleveldb_backend,{db_open,IO error:
/mnt/riak/eleveldb/client3/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}},{riak_kv_eleveldb_backend,{db_open,IO error:
/mnt/riak/eleveldb/client4/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}},{riak_kv_eleveldb_backend,{db_open,IO error:
/mnt/riak/eleveldb/client5/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}},{riak_kv_eleveldb_backend,{db_open,IO error:
/mnt/riak/eleveldb/client6/570899077082383952423314387779798054553098649600/CURRENT:
Too many open files}}]
And some relevant app.config:
{multi_backend_default, default},
{multi_backend, [
%% Default fallback: this is unused
{default, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/default}
]},
{eleveldb_client1, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client1}
]},
{eleveldb_client2, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client2}
]},
{eleveldb_client3, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client3}
]},
{eleveldb_client4, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client4}
]},
{eleveldb_client5, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client5}
]},
{eleveldb_client6, riak_kv_eleveldb_backend, [
{data_root, /mnt/riak/eleveldb/client6}
]}
]},
.. snip ..
%% Default cache size of 8MB
{cache_size, 8388608},
%% Maximum number of files open at once per partition
{max_open_files, 50}
I've set the riak soft/hard nofile 65536 in /etc/security/limits.conf, so
presumably this Too many open files error is referring to the
max_open_files option as part of eLevelDB config. The RAM in each machine
is 3.5GB free, so I calculated on a 5 node cluster this 50 max_open_files
limit.
- Is there something about Multi-backend I haven't taken into account?
- Do I need a larger max_open_files? (And thus more RAM? :)
Cheers!
Matt
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
