Re: Memory issues on ES client node

[Venkat Morampudi]

I am having connection the dots. How the timeout lead to out of memory 
exceptions?

Thanks,

-VM

On Jul 7, 2014, at 9:54 AM, joergpra...@gmail.com wrote:

> I think this is not a concurrency problem but the cluster wanted to deliver a 
> huge portion of data (just guessing about such query responses because I do 
> not know anything about the queries on your system).
> 
> Client timeout of receiving data is around 30 secs IIRC. It only means that 
> it could be possible the client gave up before the transfer of the data was 
> done.
> 
> Jörg
> 
> 
> On Mon, Jul 7, 2014 at 4:49 AM, Venkat Morampudi  
> wrote:
> It expected to nodes move huge volumes of data but what I was wondering why 
> the objects are not being garbage collected. Also, there are 242 
> TransportSearchQueryThenFetchAction$AsyncAction; I don't think that kind of 
> concurrency is not expected. I couldn't yet find from the code which object 
> is holding those objects.
> 
> The timeouts that you are referring to, are these between client node and 
> data nodes or client node and consumer?  Is there any thing the consumer need 
> to do to release objects.
> 
> Thanks for your time,
> -VM
> 
> 
> On Jul 2, 2014, at 7:08 AM, joergpra...@gmail.com wrote:
> 
>> I'm not sure but it looks like a node tries to move some GB of document hits 
>> around. This might have triggered timeouts at other places (probably with 
>> node disconnects) and maybe the GB chunk is not yet GC collected, so you see 
>> this in your heap analyzer tool.
>> 
>> It depends on the search results and search hits you generated if the 
>> heaviness of the search result is expected or not, so it would be useful to 
>> know more about your queries.
>> 
>> Jörg
>> 
>> 
>> On Wed, Jul 2, 2014 at 3:29 AM, Venkat Morampudi  
>> wrote:
>> Thanks for reply Jörg. I don't have any logs. I will try to enable them it 
>> would but it would take some time though. If there anything in particular 
>> that we need to enable, please let me know.
>> 
>> -VM
>> 
>> 
>> On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote:
>> Do you have anything in your logs, i.e. many disconnects/reconnects?
>> 
>> Jörg
>> 
>> 
>> On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi  
>> wrote:
>> In the elastic search deployment we are seeing random client node crashed 
>> due to out of memory exception. I got the memory dump from one of the crash 
>> and analysed using Eclipse memory analyzer. I have attached leak suspect 
>> report. Apparently 242 objects of type 
>> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction
>>  are holding almost 8gb of memory. I have spent some time on source code but 
>> couldn't find anything obvious. 
>> 
>> 
>> I would really appreciate any help with this issue. 
>> 
>> 
>> -VM
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com.
>> 
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com.
>> 
>> For more options, visit https://groups.google.com/d/optout.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to 
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com.
>> 
>> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com.
> 
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "elasticsearch" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/elasti

Re: Memory issues on ES client node

[joergpra...@gmail.com]

I think this is not a concurrency problem but the cluster wanted to deliver
a huge portion of data (just guessing about such query responses because I
do not know anything about the queries on your system).

Client timeout of receiving data is around 30 secs IIRC. It only means that
it could be possible the client gave up before the transfer of the data was
done.

Jörg


On Mon, Jul 7, 2014 at 4:49 AM, Venkat Morampudi 
wrote:

> It expected to nodes move huge volumes of data but what I was wondering
> why the objects are not being garbage collected. Also, there are 242
> TransportSearchQueryThenFetchAction$AsyncAction; I don’t think that kind
> of concurrency is not expected. I couldn’t yet find from the code which
> object is holding those objects.
>
> The timeouts that you are referring to, are these between client node and
> data nodes or client node and consumer?  Is there any thing the consumer
> need to do to release objects.
>
> Thanks for your time,
> -VM
>
>
> On Jul 2, 2014, at 7:08 AM, joergpra...@gmail.com wrote:
>
> I'm not sure but it looks like a node tries to move some GB of document
> hits around. This might have triggered timeouts at other places (probably
> with node disconnects) and maybe the GB chunk is not yet GC collected, so
> you see this in your heap analyzer tool.
>
> It depends on the search results and search hits you generated if the
> heaviness of the search result is expected or not, so it would be useful to
> know more about your queries.
>
> Jörg
>
>
> On Wed, Jul 2, 2014 at 3:29 AM, Venkat Morampudi <
> venkatmoramp...@gmail.com> wrote:
>
>> Thanks for reply Jörg. I don't have any logs. I will try to enable them
>> it would but it would take some time though. If there anything in
>> particular that we need to enable, please let me know.
>>
>> -VM
>>
>>
>> On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote:
>>
>>> Do you have anything in your logs, i.e. many disconnects/reconnects?
>>>
>>> Jörg
>>>
>>>
>>> On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi 
>>> wrote:
>>>
 In the elastic search deployment we are seeing random client node
 crashed due to out of memory exception. I got the memory dump from one of
 the crash and analysed using Eclipse memory analyzer. I have attached leak
 suspect report. Apparently 242 objects of type org.elasticsearch.action.
 search.type.TransportSearchQueryThenFetchAction$AsyncAction are
 holding almost 8gb of memory. I have spent some time on source code but
 couldn't find anything obvious.


 I would really appreciate any help with this issue.


 -VM

 --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearc...@googlegroups.com.

 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%
 40googlegroups.com
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E83376

Re: Memory issues on ES client node

[Venkat Morampudi]

It expected to nodes move huge volumes of data but what I was wondering why the 
objects are not being garbage collected. Also, there are 242 
TransportSearchQueryThenFetchAction$AsyncAction; I don't think that kind of 
concurrency is not expected. I couldn't yet find from the code which object is 
holding those objects.

The timeouts that you are referring to, are these between client node and data 
nodes or client node and consumer?  Is there any thing the consumer need to do 
to release objects.

Thanks for your time,
-VM


On Jul 2, 2014, at 7:08 AM, joergpra...@gmail.com wrote:

> I'm not sure but it looks like a node tries to move some GB of document hits 
> around. This might have triggered timeouts at other places (probably with 
> node disconnects) and maybe the GB chunk is not yet GC collected, so you see 
> this in your heap analyzer tool.
> 
> It depends on the search results and search hits you generated if the 
> heaviness of the search result is expected or not, so it would be useful to 
> know more about your queries.
> 
> Jörg
> 
> 
> On Wed, Jul 2, 2014 at 3:29 AM, Venkat Morampudi  
> wrote:
> Thanks for reply Jörg. I don't have any logs. I will try to enable them it 
> would but it would take some time though. If there anything in particular 
> that we need to enable, please let me know.
> 
> -VM
> 
> 
> On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote:
> Do you have anything in your logs, i.e. many disconnects/reconnects?
> 
> Jörg
> 
> 
> On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi  wrote:
> In the elastic search deployment we are seeing random client node crashed due 
> to out of memory exception. I got the memory dump from one of the crash and 
> analysed using Eclipse memory analyzer. I have attached leak suspect report. 
> Apparently 242 objects of type 
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction
>  are holding almost 8gb of memory. I have spent some time on source code but 
> couldn't find anything obvious. 
> 
> 
> I would really appreciate any help with this issue. 
> 
> 
> -VM
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com.
> 
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com.
> 
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "elasticsearch" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Memory issues on ES client node

[joergpra...@gmail.com]

I'm not sure but it looks like a node tries to move some GB of document
hits around. This might have triggered timeouts at other places (probably
with node disconnects) and maybe the GB chunk is not yet GC collected, so
you see this in your heap analyzer tool.

It depends on the search results and search hits you generated if the
heaviness of the search result is expected or not, so it would be useful to
know more about your queries.

Jörg


On Wed, Jul 2, 2014 at 3:29 AM, Venkat Morampudi 
wrote:

> Thanks for reply Jörg. I don't have any logs. I will try to enable them it
> would but it would take some time though. If there anything in particular
> that we need to enable, please let me know.
>
> -VM
>
>
> On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote:
>
>> Do you have anything in your logs, i.e. many disconnects/reconnects?
>>
>> Jörg
>>
>>
>> On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi 
>> wrote:
>>
>>> In the elastic search deployment we are seeing random client node
>>> crashed due to out of memory exception. I got the memory dump from one of
>>> the crash and analysed using Eclipse memory analyzer. I have attached leak
>>> suspect report. Apparently 242 objects of type org.elasticsearch.action.
>>> search.type.TransportSearchQueryThenFetchAction$AsyncAction are holding
>>> almost 8gb of memory. I have spent some time on source code but couldn't
>>> find anything obvious.
>>>
>>>
>>> I would really appreciate any help with this issue.
>>>
>>>
>>> -VM
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Memory issues on ES client node

[Venkat Morampudi]

Thanks Mark. I will look into installing the plug-ins that you suggested.

-VM

On Jul 1, 2014, at 8:11 PM, Mark Walkom  wrote:

> The plugins I mentioned before give you insight into cluster state, 
> specifically around GC. They would be worth installing if you want to 
> troubleshoot this.
> 
> Regards,
> Mark Walkom
> 
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
> 
> 
> On 2 July 2014 11:30, Venkat Morampudi  wrote:
> Hi Mark, is the any particular aspect of monitoring or monitoring in general?
> 
> -VM
> 
> On Tuesday, July 1, 2014 3:40:23 PM UTC-7, Mark Walkom wrote:
> Are you using anything to monitor your cluster, plugins like ElasticHQ or 
> Marvel?
> 
> Regards,
> Mark Walkom
> 
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
> 
> 
> On 2 July 2014 05:58, joerg...@gmail.com  wrote:
> Do you have anything in your logs, i.e. many disconnects/reconnects?
> 
> Jörg
> 
> 
> On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi  wrote:
> In the elastic search deployment we are seeing random client node crashed due 
> to out of memory exception. I got the memory dump from one of the crash and 
> analysed using Eclipse memory analyzer. I have attached leak suspect report. 
> Apparently 242 objects of type 
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction
>  are holding almost 8gb of memory. I have spent some time on source code but 
> couldn't find anything obvious. 
> 
> 
> I would really appreciate any help with this issue. 
> 
> 
> -VM
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com.
> 
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com.
> 
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "elasticsearch" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/CAEM624YnVud3oFFoMU7-cuqNro6Q1ML00AurMx3RZ8Eaeis0Pw%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/29787865-DCEF-43F5-8DCA-B5DAD4F7324B%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Memory issues on ES client node

[Mark Walkom]

The plugins I mentioned before give you insight into cluster state,
specifically around GC. They would be worth installing if you want to
troubleshoot this.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 2 July 2014 11:30, Venkat Morampudi  wrote:

> Hi Mark, is the any particular aspect of monitoring or monitoring in
> general?
>
> -VM
>
> On Tuesday, July 1, 2014 3:40:23 PM UTC-7, Mark Walkom wrote:
>>
>> Are you using anything to monitor your cluster, plugins like ElasticHQ or
>> Marvel?
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 2 July 2014 05:58, joerg...@gmail.com  wrote:
>>
>>> Do you have anything in your logs, i.e. many disconnects/reconnects?
>>>
>>> Jörg
>>>
>>>
>>> On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi 
>>> wrote:
>>>
 In the elastic search deployment we are seeing random client node
 crashed due to out of memory exception. I got the memory dump from one of
 the crash and analysed using Eclipse memory analyzer. I have attached leak
 suspect report. Apparently 242 objects of type org.elasticsearch.action.
 search.type.TransportSearchQueryThenFetchAction$AsyncAction are
 holding almost 8gb of memory. I have spent some time on source code but
 couldn't find anything obvious.


 I would really appreciate any help with this issue.


 -VM

 --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%
 40googlegroups.com
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7E
>>> SMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624YnVud3oFFoMU7-cuqNro6Q1ML00AurMx3RZ8Eaeis0Pw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Memory issues on ES client node

[Venkat Morampudi]

Hi Mark, is the any particular aspect of monitoring or monitoring in 
general?

-VM

On Tuesday, July 1, 2014 3:40:23 PM UTC-7, Mark Walkom wrote:
>
> Are you using anything to monitor your cluster, plugins like ElasticHQ or 
> Marvel?
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>  
>
> On 2 July 2014 05:58, joerg...@gmail.com   > wrote:
>
>> Do you have anything in your logs, i.e. many disconnects/reconnects?
>>
>> Jörg
>>
>>
>> On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi > > wrote:
>>
>>> In the elastic search deployment we are seeing random client node 
>>> crashed due to out of memory exception. I got the memory dump from one of 
>>> the crash and analysed using Eclipse memory analyzer. I have attached leak 
>>> suspect report. Apparently 242 objects of 
>>> type 
>>> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction
>>>  
>>> are holding almost 8gb of memory. I have spent some time on source code but 
>>> couldn't find anything obvious. 
>>>
>>>
>>> I would really appreciate any help with this issue. 
>>>
>>>
>>> -VM
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Memory issues on ES client node

[Venkat Morampudi]

Thanks for reply Jörg. I don't have any logs. I will try to enable them it 
would but it would take some time though. If there anything in particular 
that we need to enable, please let me know.

-VM

On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote:
>
> Do you have anything in your logs, i.e. many disconnects/reconnects?
>
> Jörg
>
>
> On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi  > wrote:
>
>> In the elastic search deployment we are seeing random client node crashed 
>> due to out of memory exception. I got the memory dump from one of the crash 
>> and analysed using Eclipse memory analyzer. I have attached leak suspect 
>> report. Apparently 242 objects of 
>> type 
>> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction
>>  
>> are holding almost 8gb of memory. I have spent some time on source code but 
>> couldn't find anything obvious. 
>>
>>
>> I would really appreciate any help with this issue. 
>>
>>
>> -VM
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Memory issues on ES client node

[Mark Walkom]

Are you using anything to monitor your cluster, plugins like ElasticHQ or
Marvel?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 2 July 2014 05:58, joergpra...@gmail.com  wrote:

> Do you have anything in your logs, i.e. many disconnects/reconnects?
>
> Jörg
>
>
> On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi <
> venkatmoramp...@gmail.com> wrote:
>
>> In the elastic search deployment we are seeing random client node crashed
>> due to out of memory exception. I got the memory dump from one of the crash
>> and analysed using Eclipse memory analyzer. I have attached leak suspect
>> report. Apparently 242 objects of
>> type 
>> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction
>> are holding almost 8gb of memory. I have spent some time on source code but
>> couldn't find anything obvious.
>>
>>
>> I would really appreciate any help with this issue.
>>
>>
>> -VM
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624bvS7Y9HoFmMG8NeQM-hWxwqPm9eb6viFANnsnsU_POFw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Memory issues on ES client node

[joergpra...@gmail.com]

Do you have anything in your logs, i.e. many disconnects/reconnects?

Jörg


On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi 
wrote:

> In the elastic search deployment we are seeing random client node crashed
> due to out of memory exception. I got the memory dump from one of the crash
> and analysed using Eclipse memory analyzer. I have attached leak suspect
> report. Apparently 242 objects of
> type 
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction
> are holding almost 8gb of memory. I have spent some time on source code but
> couldn't find anything obvious.
>
>
> I would really appreciate any help with this issue.
>
>
> -VM
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.