Re: ES OutOfMemory on a 30GB index

2014-05-29 Thread Paul Sanwald
We've narrowed the problem down to a multi_match clause in our query: {"multi_match":{"fields":["attachments.*.bodies"], "query":"foobar"}} This has to do with the way we've structured our index, We are searching an index that contains emails, and we are indexing attachments in the attachments.

Re: ES OutOfMemory on a 30GB index

2014-05-28 Thread Paul Sanwald
Sorry, it's Java 7: jvm: { pid: 20424 version: 1.7.0_09-icedtea vm_name: OpenJDK 64-Bit Server VM vm_version: 23.7-b01 vm_vendor: Oracle Corporation start_time: 1401309063644 mem: { heap_init_in_bytes: 1073741824 heap_max_in_bytes: 10498867200 non_heap_init_in_bytes: 24313856 non_heap_max_in_bytes

Re: ES OutOfMemory on a 30GB index

2014-05-28 Thread Mark Walkom
What java version are you running, it's not in the stats gist. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 29 May 2014 08:33, Paul Sanwald wrote: > I apologize about the signature, it's automatic. I've created

Re: ES OutOfMemory on a 30GB index

2014-05-28 Thread Paul Sanwald
I apologize about the signature, it's automatic. I've created a gist with the cluster node stats: https://gist.github.com/pcsanwald/e11ba02ac591757c8d92 We are using 1.1.0, using aggregations a lot but nothing crazy. We run our app on much much larger indices successfully. But, the problem seems

Re: ES OutOfMemory on a 30GB index

2014-05-28 Thread Mark Walkom
Can you provide some specs on your cluster, OS, RAM, heap, disk, java and ES versions? Are you using parent/child relationships, TTLs, large facet or other queries? (Also, your elaborate legalese signature is kind of moot given you're posting to a public mailing list :p) Regards, Mark Walkom In

ES OutOfMemory on a 30GB index

2014-05-28 Thread Paul Sanwald
Hi Everyone, We are seeing continual OOM exceptions on one of our 1.1.0 elasticsearch clusters, the index is ~30GB, quite small. I'm trying to work out the root cause via heap dump analysis, but not having a lot of luck. I don't want to include a bunch of unnecessary info, but the stacktrace