Re: Timeout notification from cluster service
Please check the status of indexes when you get these log messages. I found these log messages appear in when index status in not green. You can use following query to check : curl -s –XGET http://:9200/_cat/indices?pretty=true <http://10.72.12.22:9200/_cat/indices?pretty=true> | grep -v green | wc -l On Thursday, August 14, 2014 10:30:05 AM UTC+5:30, Artur S. (Digit) wrote: > > I see exactly the same in my logs: > [2014-08-14 04:57:43,913][DEBUG][action.index ] [data-es-001] > observer: timeout notification from cluster service. timeout setting [1m], > time since start [1m] > [2014-08-14 04:57:56,869][DEBUG][action.index ] [data-es-001] > observer timed out. notifying listener. timeout setting [1m], time since > start [1m] > > > Those lines keep appearing since we upgraded to 1.3.1. Also, Since the > upgrade last week our cluster died 3 times already. I'm not sure if that's > related... > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4b106709-0f8b-4932-8fba-5b9d541cf77e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Timeout notification from cluster service
Please check the status of indexes when you get these log messages. I found these log messages appear in when index status in not green. You can use following query to check : curl -s –XGET http://10.72.12.22:9200/_cat/indices?pretty=true | grep -v green | wc -l On Thursday, August 14, 2014 10:30:05 AM UTC+5:30, Artur S. (Digit) wrote: > > I see exactly the same in my logs: > [2014-08-14 04:57:43,913][DEBUG][action.index ] [data-es-001] > observer: timeout notification from cluster service. timeout setting [1m], > time since start [1m] > [2014-08-14 04:57:56,869][DEBUG][action.index ] [data-es-001] > observer timed out. notifying listener. timeout setting [1m], time since > start [1m] > > > Those lines keep appearing since we upgraded to 1.3.1. Also, Since the > upgrade last week our cluster died 3 times already. I'm not sure if that's > related... > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b4f5d04f-1674-4045-b5ba-72a2fc90db22%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Timeout notification from cluster service
I see exactly the same in my logs: [2014-08-14 04:57:43,913][DEBUG][action.index ] [data-es-001] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2014-08-14 04:57:56,869][DEBUG][action.index ] [data-es-001] observer timed out. notifying listener. timeout setting [1m], time since start [1m] Those lines keep appearing since we upgraded to 1.3.1. Also, Since the upgrade last week our cluster died 3 times already. I'm not sure if that's related... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1c76479d-6c28-4d73-bab2-99e9a450f79c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Timeout notification from cluster service
I am running a 4 node cluster running in EC2 and for the past few days, I have noticed that some nodes occasionally timeout on a request resulting in the following: ConnectionError(HTTPConnectionPool(host='HOST', port=9200): Read timed out. (read timeout=10)) caused by: ReadTimeoutError(HTTPConnectionPool(host='HOST', port=9200): Read timed out. (read timeout=10)) >From the error, I figured it was something to do with my application keeping the elasticsearch instance open to long, so I decided to spawn it fresh every time a request was made. Still, this issue appear to pop up without much explanation. I figure it could be due to the nodes communicating, so I checked my log and saw the following: [2014-07-08 16:19:29,965][DEBUG][action.index ] [Node Name] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] I see this in all of my log files, but googling around doesn't seem to lead me to a direct cause. I would guess whatever is causing the above log entry is either corresponding to the timeouts I see or is the reason they are occurring. Any ideas? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3229d028-01b1-4015-b9ae-b7e2d86faf11%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
