Re: Constantly increasing memory outside of Java heap

You can limit the off-heap space used by setting ES_DIRECT_SIZE. -- Ivan On Tue, Apr 8, 2014 at 1:31 PM, Yitzhak Kesselman wrote: > Hi, > > I have experienced same behavior when I have tried to load large amount of > data... If you clear the file system cache > (here

Re: Constantly increasing memory outside of Java heap

Hi, I have experienced same behavior when I have tried to load large amount of data... If you clear the file system cache (hereis a link to a tool), the memory drops to the defined heap size. However this is still looks as a wron

Re: Constantly increasing memory outside of Java heap

As a follow-up, when the server is nearing maximum memory, the memory use stops increasing. This would indeed support Zachary's caching theory, although I'm still confused as to why it shows up as 'in use' memory rather than 'cached' memory. In any case, it does not block me right now. It's jus

Re: Constantly increasing memory outside of Java heap

There are no other processes running except for ES and the program which posts the updates. The memory is constantly increasing when the updater is running, but is stale (and doesn't release the memory at all, no matter how much is used) whenever ES is idle. On Thursday, March 13, 2014 5:32:43

Re: Constantly increasing memory outside of Java heap

Also, are there other processes running which may be causing the problem? Does the behavior only happen when ES is running? On Thursday, March 13, 2014 8:31:18 PM UTC-4, Zachary Tong wrote: > > Cool, curious to see what happens. As an aside, I would recommend > downgrading to Java 1.7.0_u25.

Re: Constantly increasing memory outside of Java heap

Cool, curious to see what happens. As an aside, I would recommend downgrading to Java 1.7.0_u25. There are known bugs in the most recent Oracle JVM versions which have not been resolved yet. u25 is the most recent safe version. I don't think that's your problem, but it's a good general cons

Re: Constantly increasing memory outside of Java heap

@Mark: The heap is set to 2GB, using mlockall. The problem occurs with both OpenJDK7 and OracleJDK7, both the latest versions. I have one index, which is very small: index: { primary_size_in_bytes: 37710681 size_in_bytes: 37710681 } @Zachary Our systems are set up to alert when memory is about

Re: Constantly increasing memory outside of Java heap

I believe you are just witnessing the OS caching files in memory. Lucene (and therefore by extension Elasticsearch) uses a large number of files to represent segments. TTL + updates will cause even higher file turnover than usual. The OS manages all of this caching and will reclaim it for oth

Re: Constantly increasing memory outside of Java heap

How much heap, what java version, how big are your indexes? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 March 2014 11:11, Jos Kraaijeveld wrote: > I forgot to mention, I'm running ElasticSearch 1.0.1 on Ubun

Re: Constantly increasing memory outside of Java heap

I forgot to mention, I'm running ElasticSearch 1.0.1 on Ubuntu 12.04 with 24GB of available RAM. On Thursday, March 13, 2014 5:07:13 PM UTC-7, Jos Kraaijeveld wrote: > > Hey, > > I've run into an issue which is preventing me from moving forwards with > ES. I've got an application where I keep 'l

Constantly increasing memory outside of Java heap

Hey, I've run into an issue which is preventing me from moving forwards with ES. I've got an application where I keep 'live' documents in ElasticSearch. Each document is a combination from data from multiple sources, which are merged together using doc_as_upsert. Each document has a TTL which i