Cool, curious to see what happens. As an aside, I would recommend downgrading to Java 1.7.0_u25. There are known bugs in the most recent Oracle JVM versions which have not been resolved yet. u25 is the most recent safe version. I don't think that's your problem, but it's a good general consideration anyway.
-Z On Thursday, March 13, 2014 8:23:34 PM UTC-4, Jos Kraaijeveld wrote: > > @Mark: > The heap is set to 2GB, using mlockall. The problem occurs with both > OpenJDK7 and OracleJDK7, both the latest versions. I have one index, which > is very small: > index: > { > primary_size_in_bytes: 37710681 > size_in_bytes: 37710681 > } > > @Zachary Our systems are set up to alert when memory is about to run out. > We use Ganglia to monitor our systems and that represents the memory as > 'used', rather than 'cached'. I will try to just let it run until memory > runs out and report back after that though. > > > > On Thursday, March 13, 2014 5:17:20 PM UTC-7, Zachary Tong wrote: >> >> I believe you are just witnessing the OS caching files in memory. Lucene >> (and therefore by extension Elasticsearch) uses a large number of files to >> represent segments. TTL + updates will cause even higher file turnover >> than usual. >> >> The OS manages all of this caching and will reclaim it for other >> processes when needed. Are you experiencing problems, or just witnessing >> memory usage? I wouldn't be concerned unless there is an actual problem >> that you are seeing. >> >> >> >> On Thursday, March 13, 2014 8:07:13 PM UTC-4, Jos Kraaijeveld wrote: >>> >>> Hey, >>> >>> I've run into an issue which is preventing me from moving forwards with >>> ES. I've got an application where I keep 'live' documents in ElasticSearch. >>> Each document is a combination from data from multiple sources, which are >>> merged together using doc_as_upsert. Each document has a TTL which is >>> updated whenever new data comes in for a document, so documents die >>> whenever no data source has given information about it for a while. The >>> amount of documents generally doesn't exceed 15.000 so it's a fairly small >>> data set. >>> >>> Whenever I leave this running, slowly but surely memory usage on the box >>> creeps up, seemingly unbounded until there is no more resident memory left. >>> The Java process nicely keeps within its set ES_MAX_HEAP bounds, but it >>> seems the mapping from storage on disk to memory is every-increasing, even >>> when the amount of 'live' documents goes to 0. >>> >>> I was wondering if anyone has seen such a memory problem before and >>> whether there are ways to debug memory usage which is unaccounted for by >>> processes in 'top'. >>> >> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a492cb9a-abeb-4b0e-ae97-5db7df843565%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.