Re: ES/Lucene eating up entire memory!

2015-04-03 Thread Aaron Mefford
First of all the what, not so much the why: http://www.elastic.co/guide/en/elasticsearch/guide/master/heap-sizing.html In order of depth, first one provides light coverage of the topic: http://linux-mm.org/Low_On_Memory http://www.linuxhowtos.org/System/Linux%20Memory%20Management.htm http://www

Re: ES/Lucene eating up entire memory!

2015-04-03 Thread Yogesh
Thanks Mark. I separated out the processes on different servers, so now ES has a full server. I made the changes yesterday and it seems to be stable since then. On Wednesday, April 1, 2015 at 2:07:41 AM UTC+5:30, Mark Walkom wrote: > > Also, we don't recommend running ES alongside other apps. As

Re: ES/Lucene eating up entire memory!

2015-04-03 Thread Yogesh
Thanks Aaron. Your post was very informative. Can you recommend any blogposts, articles etc. where I could read more on this topic? Thanks again for your help. On Tuesday, March 31, 2015 at 9:57:58 PM UTC+5:30, Aaron Mefford wrote: > > You need to read up a bit on how memory is allocated in Linu

Re: ES/Lucene eating up entire memory!

2015-03-31 Thread Mark Walkom
Also, we don't recommend running ES alongside other apps. As you can see contention is an issue and you will have to pay the price there. On 1 April 2015 at 03:27, Aaron Mefford wrote: > You need to read up a bit on how memory is allocated in Linux. > > In an ElasticSearch or Database server, th

Re: ES/Lucene eating up entire memory!

2015-03-31 Thread Aaron Mefford
You need to read up a bit on how memory is allocated in Linux. In an ElasticSearch or Database server, this seems to be both, you want that free column to be 0. All available free memory should be used to cache files. In your snapshot you have 35GB of file cache listed under the cached headi

Re: ES/Lucene eating up entire memory!

2015-03-31 Thread Yogesh Kansal
Thanks Uwe. As I mentioned earlier, I did guess that VIRT doesn't indicate RAM consumption. What I am concerned about is the 3rd row which shows memory and indicates that out of the total 50g, 43g is in use. Once this number crosses 45g, my other databases start behaving badly. Problem is, even a

Re: ES/Lucene eating up entire memory!

2015-03-29 Thread Uwe Schindler
You should read: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Maybe this allows you to figure out what's going on! VIRT means nothing about consumption, you should look at RES. Thanks, Uwe Am Sonntag, 29. März 2015 22:23:00 UTC+2 schrieb Yogesh: > > Hi, > > I have a

Re: ES/Lucene eating up entire memory!

2015-03-29 Thread Robert Muir
Do you know what virtual memory is? You have terabytes of it. On Sun, Mar 29, 2015 at 4:22 PM, Yogesh wrote: > Hi, > > I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I run > the Twitter river on it. I've set the ES_HEAP_SIZE to 5g. However, when I > do "top", the ES process

ES/Lucene eating up entire memory!

2015-03-29 Thread Yogesh
Hi, I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I run the Twitter river on it. I've set the ES_HEAP_SIZE to 5g. However, when I do "top", the ES process shows the VIRT memory to be around 34g. That would be I assume the max mapped memory. The %MEM though always hovers