Hi,
We have a simple 3 node test cluster, and we are seeing this cluster fall-over
under quite modest loads, 2-3 times a day with the node reporting out of memory
problems.
Our basic details are:
* Riak 1.4.9
* 3 nodes, each being:
o A virtual RedHat EL
o 2 x 2.2GHz CPU
o 4GB RAM
* In-memory back end
* 128 segment ring
We have tried to limit the in-memory max memory by the following setting in
app.config:
%% Memory Config
{memory_backend, [
{max_memory, 32}, %% 32 megabytes
{ttl, 86400} %% 1 Day in seconds
]},
The behaviour we are seeing is a sudden and rapid increase in memory allocated
to riak, up to 100% available RAM, and then a crash.
This is happening on all 3 nodes (at different times).
When looking through the console.log just prior to the out of memory / crash we
see log entries similar to the following appearing:
2015-02-11 16:50:27.356 [info]
<0.98.0>@riak_core_sysmon_handler:handle_event:92 monitor large_heap <0.238.0>
[{name,riak_core_gossip},{initial_call,{riak_core_gossip,init,1}},{almost_current_function,{riak_core_gossip,update_gossip_vers
ion,1}},{message_queue_len,78}]
[{old_heap_block_size,0},{heap_block_size,47828850},{mbuf_size,0},{stack_size,15},{old_heap_size,0},{heap_size,13944475}]
We can't see any errors at the appropriate times in the error.log
We see the following in erlang.log.1:
===== ALIVE Wed Feb 11 17:00:24 GMT 2015
/usr/lib64/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed.
Erlang has closed
Crash dump was written to: /var/log/riak/erl_crash.dump
eheap_alloc: Cannot allocate 4454408120 bytes of memory
=====
Anyone any ideas?
Simon Hartley
Solutions Architect
Email: simon.hart...@williamhill.com<mailto:simon.hart...@williamhill.com>
Skype: +44 (0)113 397 6747
Skype: sijomons
William Hill Online, St. Johns, Merrion St. Leeds, LS2 8LQ
[Description: Description: Description: cid:image002.png@01CC2FFA.24244CF0]
Confidentiality: The contents of this e-mail and any attachments transmitted
with it are intended to be confidential to the intended recipient; and may be
privileged or otherwise protected from disclosure. If you are not an intended
recipient of this e-mail, do not duplicate or redistribute it by any means.
Please delete it and any attachments and notify the sender that you have
received it in error. This e-mail is sent by a William Hill PLC group company.
The William Hill group companies include, among others, William Hill PLC
(registered number 4212563), William Hill Organization Limited (registered
number 278208), William Hill US HoldCo Inc, WHG (International) Limited
(registered number 99191) and WHG Trading Limited (registered number 101439).
Each of William Hill PLC, William Hill Organization Limited is registered in
England and Wales and has its registered office at Greenside House, 50 Station
Road, Wood Green, London N22 7TP. William Hill U.S. HoldCo, Inc. is 160
Greentree Drive, Suite 101, Dover 19904, Kent, Delaware, United States of
America. Each of WHG (International) Limited and WHG Trading Limited is
registered in Gibraltar and has its registered office at 6/1 Waterport Place,
Gibraltar. Unless specifically indicated otherwise, the contents of this e-mail
are subject to contract; and are not an official statement, and do not
necessarily represent the views, of William Hill PLC, its subsidiaries or
affiliated companies. Please note that neither William Hill PLC, nor its
subsidiaries and affiliated companies can accept any responsibility for any
viruses contained within this e-mail and it is your responsibility to scan any
emails and their attachments. William Hill PLC, its subsidiaries and affiliated
companies may monitor e-mail traffic data and also the content of e-mails for
effective operation of the e-mail system, or for security, purposes..
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
