Re: Very frequent ES OOM's & potential segment merge problems

2014-06-20 Thread Paul Sabou
java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot complete merge at org.apache.lucene.index.IndexWriter.commitMerge(IndexWriter.java:3546) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4272) at org.apache.lucene.index.Index

Re: Very frequent ES OOM's & potential segment merge problems

2014-06-20 Thread Alexander Reelsen
Hey, can you provide more information about the OOM exception? Also you should use the nodes stats API to monitor your system, so you can maybe easily spot, where this memory consumption stems from. Also, are you just indexing or doing searches/queries/gets as well? --Alex On Thu, Jun 19, 2014

Very frequent ES OOM's & potential segment merge problems

2014-06-19 Thread Paul Sabou
Hi, *Situation:* We are using ES 1.2.1 on a machine with 32GB RAM, fast SSD and 12 cores. The machine runs Ubuntu 14.0.x LTS. The ES process has 12GB of RAM allocated. We have an index in which we inserted 105 million small documents so the ES data folder is around 50GB in size (we see this by