See: http://docs.basho.com/riak/latest/tutorials/choosing-a-backend/LevelDB/
Look for the section titled "Parameter Planning". It has the best content. Keep in mind that leveldb maps most of its files into memory. So the RSS (resident set size) is BOTH the memory you allocate via parameters and the memory mapped data files. Our other thread is likely a disk thrashing problem, but we can help it by tuning the memory. Each vnode opens an independent recovery log. Each data item goes to log before the write operation returns. You are constantly writing to 512 different places on the disk … plus leveldb's regular "compaction" runs … or that at least is my theory of the moment. Matthew On Nov 14, 2012, at 1:49 PM, David Lowell <d...@go2ctv.com> wrote: > What major factors drive memory usage in Riak? As users of Riak 1.2.1 with > eleveldb back-end, I had figured that the dominant source of memory use would > be the eleveldb cache. However, we're using the default 8 MB of cache per > vnode, which would therefore max out at 4 GB of cache for our 512 vnodes > (when all partitions are on a single server). However, we see Riak's beam.smp > process taking much more than 4 GB of memory. Running a heavy workload on top > of riak, we see resident set size of 23 GB on our single server Riak cluster > as I type. > > So I haven't come across much in the Riak docs that describe other factors > that can drive up memory usage. What could be adding up to so many gigs of > memory? Queues? Caches other than leveldb? > > Given the other thread about our issues with mysterious problems in riak > (with some evidence of memory stress), I'd like to understand better what > could be driving what seems to be excessive memory usage by Riak. > > Thanks, > > Dave > > -- > Dave Lowell > d...@connectv.com > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
