Re: Getting Timeout Exceptions with Elasticsearch
How much data is in the cluster? How big are your nodes?
You're running a very old version of Java, recommended is 1.7u55 or
greater, and Oracle not OpenJDK.
On 13 November 2014 06:49, shriyansh jain shriyanshaj...@gmail.com wrote:
Hi all,
I am getting timeout exceptions with elastic search while deleting an
index and while looking for logs in kibana its taking lot of time to load
logs.
Following are the log details and setup info.
API Call for deleting an Index:
curl -XDELETE 'http://localhost:9200/logstash-2014.08.21'
{error:ProcessClusterEventTimeoutException[failed to process cluster
event (delete-index [logstash-2014.08.21]) within 30s],status:503}
Logs from master node: (setup is cluster of 2 nodes)
http://pastebin.com/hUZEf26d
Logs from other node
http://pastebin.com/HxWppiYL
curl -XGET 'localhost:9200'
{
status : 200,
name : Node1,
version : {
number : 1.2.2,
build_hash : 9902f08efc3ad14ce27882b991c4c56b920c9872,
build_timestamp : 2014-07-09T12:02:32Z,
build_snapshot : false,
lucene_version : 4.8
},
tagline : You Know, for Search
}
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
cluster_name : x,
status : green,
timed_out : false,
number_of_nodes : 2,
number_of_data_nodes : 2,
active_primary_shards : 270,
active_shards : 540,
relocating_shards : 0,
initializing_shards : 0,
unassigned_shards : 0
}
Java version on the system
java version 1.7.0_09-icedtea
OpenJDK Runtime Environment (rhel-2.3.4.1.el6_3-x86_64)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
I am using HQ plugin to visualize the elasticsearch plugin, but that
plugin is taking lot of time to load the cluster info.
what might be causing the timeouts to happen. Any inputs will be greatly
appreciated.
please let me know if any other details are required.
Thanks!
Shriyansh
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/89298f22-ec17-49b2-a7fe-c6f110341146%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/89298f22-ec17-49b2-a7fe-c6f110341146%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DNiOEhZ3-d%3DGJoxY1JEi%3DLyF_HxiLuCrGDip3rykENqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: Getting Timeout Exceptions with Elasticsearch
Hi Mark, The data is around id 150GB. The master node holds : 16 GB RAM with ES_HEAP_SIZE = 8GB The other node holds : 16 GB RAM with ES_HEAP_SIZE = 8 GB I will get the version updated with Oracle Java, will that solve the issue.? Thanks! Shriyansh -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8fc2f714-2c1a-43d4-a67c-2b6869007712%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Getting Timeout Exceptions with Elasticsearch
It won't solve it but it may help. You have some pretty large GC's happening, some getting close to the 30s default ping timeout. You can try increasing discovery.zen.fd.ping_timeout but you may want to consider adding another node in to generally help relieve heap pressure and improve performance, 3 nodes is much better than 2! On 13 November 2014 07:08, shriyansh jain shriyanshaj...@gmail.com wrote: Hi Mark, The data is around id 150GB. The master node holds : 16 GB RAM with ES_HEAP_SIZE = 8GB The other node holds : 16 GB RAM with ES_HEAP_SIZE = 8 GB I will get the version updated with Oracle Java, will that solve the issue.? Thanks! Shriyansh -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8fc2f714-2c1a-43d4-a67c-2b6869007712%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/8fc2f714-2c1a-43d4-a67c-2b6869007712%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZn0ZmJjkR_K5JtqPvU7bPbwB4DdRuf_4ugSxAN%2BN2Adqg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Getting Timeout Exceptions with Elasticsearch
I have stopped collecting all the data from logstash( as I am using ELK stack), but it still I am getting pretty large GC's (as you mentioned). Can you please point me out what might be the reason for that.? Thank you for the suggestion and pointing me out the error, I appreciate it and will consider increasing the number of nodes of the cluster. Thanks! Shriyansh -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8e005f9a-bb48-4a50-9c94-2c14861dad7d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Getting Timeout Exceptions with Elasticsearch
I am also getting Out of Memory exception now in the logs [2014-11-12 13:08:23,157][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by a user handler while handling an exception event ([id: 0x40f3788c, /10.18.144.231:52924 = /10.18.144.229:9300] EXCEPTION: java.lang.OutOfMemoryError: Java heap space) java.lang.OutOfMemoryError: Java heap space what will be way out to get out of this situation.? Thanks! Shriyansh -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/de51ffc7-7d40-4ced-b37c-84ee68ca7d7c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Getting Timeout Exceptions with Elasticsearch
You either need to add more nodes, reduce your dataset, increase the heap or close some old indexes. On 13 November 2014 08:11, shriyansh jain shriyanshaj...@gmail.com wrote: I am also getting Out of Memory exception now in the logs [2014-11-12 13:08:23,157][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by a user handler while handling an exception event ([id: 0x40f3788c, /10.18.144.231:52924 = /10.18.144.229:9300] EXCEPTION: java.lang.OutOfMemoryError: Java heap space) java.lang.OutOfMemoryError: Java heap space what will be way out to get out of this situation.? Thanks! Shriyansh -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/de51ffc7-7d40-4ced-b37c-84ee68ca7d7c%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/de51ffc7-7d40-4ced-b37c-84ee68ca7d7c%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DaT2YoyTgDaL7nTgoqraGuKG9KGPCP3gqHcVFHDMJbiA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Getting Timeout Exceptions with Elasticsearch
Thank you mark for pointing me out with the situation. As I am trying to delete the index I am getting timeouts for that. But you suggested me to increase discovery.zen.fd.ping_timeout to get rid of that. I am not able to see any such variable in elasticsearch.yml file. Can you please point me how to increase discovery.zen.fd.ping_timeout. Thanks! Shriyansh -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db816351-88d2-4e18-9863-3e719f2af1ed%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Getting Timeout Exceptions with Elasticsearch
You can add that in yourself. On 13 November 2014 08:26, shriyansh jain shriyanshaj...@gmail.com wrote: Thank you mark for pointing me out with the situation. As I am trying to delete the index I am getting timeouts for that. But you suggested me to increase discovery.zen.fd.ping_timeout to get rid of that. I am not able to see any such variable in elasticsearch.yml file. Can you please point me how to increase discovery.zen.fd.ping_timeout. Thanks! Shriyansh -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db816351-88d2-4e18-9863-3e719f2af1ed%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/db816351-88d2-4e18-9863-3e719f2af1ed%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZkspduJTuezUj_%3DhojkhPcO0CjsWVmYxo4HVWsoFR1JTQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Getting Timeout Exceptions with Elasticsearch
Thanks! -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b9bf42ad-f55f-4091-99c0-e34125eccaf0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Getting Timeout Exceptions with Elasticsearch
will it need a restart to being that info effect.? Thank! Shriyansh On Wednesday, November 12, 2014 1:30:57 PM UTC-8, Mark Walkom wrote: You can add that in yourself. On 13 November 2014 08:26, shriyansh jain shriyan...@gmail.com javascript: wrote: Thank you mark for pointing me out with the situation. As I am trying to delete the index I am getting timeouts for that. But you suggested me to increase discovery.zen.fd.ping_timeout to get rid of that. I am not able to see any such variable in elasticsearch.yml file. Can you please point me how to increase discovery.zen.fd.ping_timeout. Thanks! Shriyansh -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db816351-88d2-4e18-9863-3e719f2af1ed%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/db816351-88d2-4e18-9863-3e719f2af1ed%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a1b4fb14-a053-4c9b-ac5f-c6bb22fe3fdb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
