Thanks for the help. We've decided to expand the cluster and plan a future analysis.
Thanks again! On Aug 19, 2011 1:57 AM, "Nico Meyer" <[email protected]> wrote: > Hi Jaques, > > at least it is within a factor of 1.5 or so, could be worse :-). > > I believe some of the additional overhead is due to some intricacies in > Erlang memory management (or possibly memory management in general, like > fragmentation). There was some discussion about a similar issue on the > Erlang mailing list a while ago > ( http://erlang.2086793.n4.nabble.com/discrepancy-of-memory-usage-figures-td3618939.html), > but no conclusion. > > I just checked one of our riak nodes, and also saw a similar > discrepancy: 'erlang:memory()' (from the erlang shell of the node) > reported about 10Gb used, but ps reported about 13Gb of resident memory > for the beam.smp process. > After restarting the riak node the discrepancy became much smaller, 9Gb > to 10Gb. > > Cheers, > Nico > > > Am 19.08.2011 01:40, schrieb Jacques: >> This is very helpful. Thank you. >> >> We actually have two different key sizes, 9 key + 1 bucket = 10 total >> And 11 key + 1 bucket = 12 total. >> >> Sounds like both of those will get stored on a 64 byte boundary. >> >> That puts us at ~80gb of expected usage. We're currently at 110gb. >> >> Coming from the java world, I'm wondering if there are any >> intermediate objects that need to be collected? >> >> It sounds like you're generally saying that "it is what it is" for now >> and just capacity plan around it. >> >> thanks again, >> Jacques >> >> On Thu, Aug 18, 2011 at 3:31 AM, Nico Meyer <[email protected] >> <mailto:[email protected]>> wrote: >> >> Have you taken into account the size of the bucket name? is it 12 >> bytes for key+bucket? >> >> Also there is some additional overhead that is due to memory >> alignment requirements, depending on your CPU architecture. >> I have done some tests on this for x86_64 a while ago: >> >> size=55: overhead=9 >> size=56: overhead=8 >> size=57: overhead=23 >> size=58: overhead=22 >> size=59: overhead=21 >> size=60: overhead=20 >> size=61: overhead=19 >> size=62: overhead=18 >> size=63: overhead=17 >> size=64: overhead=16 >> size=65: overhead=15 >> size=66: overhead=14 >> size=67: overhead=13 >> size=68: overhead=12 >> size=69: overhead=11 >> size=70: overhead=10 >> size=71: overhead=9 >> size=72: overhead=8 >> Not much you can do about that, unless you want to use unaligned >> memory, which is super complicated (replace malloc with your own >> allocator) and has a huge performance penalty. >> >> So lets assume your 12 bytes are include bucket and key: >> >> 43 byte overhead + 12 byte keydata = 55 bytes, this is the number >> of bytes allocated per entry. In reality malloc(55) allocates 64 >> bytes internally, so you total memory consumption is: >> >> 64byte*3*450Mio = 80 Gb >> >> But if the bucket size is not included in the 12 bytes, and your >> bucket name(s) has at least a size of 2 bytes we get: >> >> (43 + 12 + 2) bytes = 57 bytes, and in this case malloc(57) >> really allocates 80 bytes inernally. Therefore: >> >> 80byte*3*450Mio = 100Gb >> >> >> Cheers, >> Nico >> >> >> Am 17.08.2011 19:43, schrieb Jacques: >>> >>> So if my math is correct, consumption should be around 70-80 for >>> me. Yet we see a usage over 100. I'm also notclear what the step >>> changes are. If you have any insights on these that would be >>> great. Thanks! >>> >>> On Aug 17, 2011 12:07 AM, "Nico Meyer" <[email protected] >>> <mailto:[email protected]>> wrote: >>> > Hi Jacques, >>> > >>> > please read my reply back in May, which should clear things up: >>> > >>> > >>> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-May/004292.html >>> > >>> > Cheers, >>> > Nico >>> > >>> > On 16.08.2011 21:09, Jacques wrote: >>> >> We're utilizing Riak 14.2 and we're seeing higher memory >>> consumption >>> >> than we expect. >>> >> >>> >> We're running on a 4 node cluster with each node housing 32gb >>> of memory >>> >> and are utilizing bitcask with a 3x write replication factor. >>> We're >>> >> seeing faster growth than we expect and also seeing weird >>> bounces update. >>> >> >>> >> You can see an example chart >>> <http://picturepush.com/public/6331935>. >>> >> (note, there are sometimes where we've had to stop the job for >>> short >>> >> periods of time-- you can see these as flat spots). >>> >> >>> >> We are doing a large throttled import that has key sizes of >>> >> approximately 12 bytes. We're currently around 450mm unique >>> items and >>> >> riak memory consumption is ~110gb. The input job is probably >>> 95% new >>> >> puts and 5% overwriting puts. >>> >> >>> >> According to the capacity planner tools, our key space should >>> probably >>> >> be about half what are actual memory consumption is. >>> >> >>> >> As you can see in the chart, we're also seeing jumps in memory >>> size at >>> >> random intervals. What might these be? Nothing interesting in >>> the logs >>> >> that I can see. Regular merges. >>> >> >>> >> A close up <http://picturepush.com/public/6332084> of a recent >>> jump in >>> >> memory consumption for one of the nodes (they all look the >>> same). There >>> >> are no corresponding distinct patterns within the cpu chart. >>> Things are >>> >> pretty flat although we have more wait sometimes than we like >>> (need more >>> >> spindles clearly). >>> >> >>> >> >>> >> Any helpful thoughts? >>> >> >>> >> Thanks, >>> >> Jacques >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> _______________________________________________ >>> >> riak-users mailing list >>> >> [email protected] <mailto:[email protected]> >>> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
