Re: Memory issues on ES client node

2014-07-09 Thread Venkat Morampudi
I am having connection the dots. How the timeout lead to out of memory exceptions? Thanks, -VM On Jul 7, 2014, at 9:54 AM, joergpra...@gmail.com wrote: > I think this is not a concurrency problem but the cluster wanted to deliver a > huge portion of data (just guessing about such query respon

Re: Memory issues on ES client node

2014-07-07 Thread joergpra...@gmail.com
I think this is not a concurrency problem but the cluster wanted to deliver a huge portion of data (just guessing about such query responses because I do not know anything about the queries on your system). Client timeout of receiving data is around 30 secs IIRC. It only means that it could be pos

Re: Memory issues on ES client node

2014-07-06 Thread Venkat Morampudi
It expected to nodes move huge volumes of data but what I was wondering why the objects are not being garbage collected. Also, there are 242 TransportSearchQueryThenFetchAction$AsyncAction; I don't think that kind of concurrency is not expected. I couldn't yet find from the code which object is

Re: Memory issues on ES client node

2014-07-02 Thread joergpra...@gmail.com
I'm not sure but it looks like a node tries to move some GB of document hits around. This might have triggered timeouts at other places (probably with node disconnects) and maybe the GB chunk is not yet GC collected, so you see this in your heap analyzer tool. It depends on the search results and

Re: Memory issues on ES client node

2014-07-01 Thread Venkat Morampudi
Thanks Mark. I will look into installing the plug-ins that you suggested. -VM On Jul 1, 2014, at 8:11 PM, Mark Walkom wrote: > The plugins I mentioned before give you insight into cluster state, > specifically around GC. They would be worth installing if you want to > troubleshoot this. > >

Re: Memory issues on ES client node

2014-07-01 Thread Mark Walkom
The plugins I mentioned before give you insight into cluster state, specifically around GC. They would be worth installing if you want to troubleshoot this. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 2 July 2014

Re: Memory issues on ES client node

2014-07-01 Thread Venkat Morampudi
Hi Mark, is the any particular aspect of monitoring or monitoring in general? -VM On Tuesday, July 1, 2014 3:40:23 PM UTC-7, Mark Walkom wrote: > > Are you using anything to monitor your cluster, plugins like ElasticHQ or > Marvel? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campai

Re: Memory issues on ES client node

2014-07-01 Thread Venkat Morampudi
Thanks for reply Jörg. I don't have any logs. I will try to enable them it would but it would take some time though. If there anything in particular that we need to enable, please let me know. -VM On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote: > > Do you have anything in your lo

Re: Memory issues on ES client node

2014-07-01 Thread Mark Walkom
Are you using anything to monitor your cluster, plugins like ElasticHQ or Marvel? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 2 July 2014 05:58, joergpra...@gmail.com wrote: > Do you have anything in your logs,

Re: Memory issues on ES client node

2014-07-01 Thread joergpra...@gmail.com
Do you have anything in your logs, i.e. many disconnects/reconnects? Jörg On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi wrote: > In the elastic search deployment we are seeing random client node crashed > due to out of memory exception. I got the memory dump from one of the crash > and anal