Hi Janet, Was there anything in the master logs, i.e., gnslogstash10 in your example?
Cheers, Boaz On Monday, November 10, 2014 11:47:22 PM UTC+1, Janet Sullivan wrote: > > I’m also using unicast discovery, as multicast doesn’t work on Azure. I > ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4 > wouldn’t accept shards with the new lucene version. I ended up rebuilding > the cluster, and I’m going to have to backfill from text logs. A fresh 1.4 > cluster works fine, but after two days I couldn’t get the upgraded cluster > to work. I’m glad to hear someone else had a similar issue. > > On Nov 10, 2014, at 7:05 AM, Valentin <plet...@gmail.com> wrote: > > I had similar issues when upgrading from 1.3.4 to 1.4 > from my elasticsearch.yml > > discovery.zen.ping.multicast.enabled: false > > discovery.zen.ping.unicast.hosts:..... > > I could get it up and running after restarting the whole cluster (which > was bad since I'm using it for realtime logging). > > On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote: >> >> Hi, >> >> The logs you mentioned indicate that the nodes try to join the cluster >> but it takes too long for a complete verification cycle (connect back to >> node and publish cluster state to it) takes too long. It seems there is >> something going on your masters. >> >> Can you check the logs over there? Also are you using multicast or >> unicast discovery? >> >> On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote: >>> >>> More hours of working – even when I get a 1.4.0 cluster up, masters >>> wouldn’t fail over – when I took master1 down, neither master2 or master3 >>> would promote themselves. In 1.4.0-beta it fails over quickly. >>> >>> >>> *From:* elasti...@googlegroups.com [mailto:elasti...@googlegroups.com] *On >>> Behalf Of *Janet Sullivan >>> *Sent:* Saturday, November 08, 2014 11:11 PM >>> *To:* elasti...@googlegroups.com >>> *Subject:* RE: Nodes not joining after 1.4.0 upgrade >>> >>> >>> OK, it also happens to some degree with 1.4.0-beta, although overall >>> it’s much better on beta. I wasn’t able to get my 12 node cluster up on >>> 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up. >>> >>> >>> *From:* elasti...@googlegroups.com [mailto:ela...@googlegroups.com] *On >>> Behalf Of *Janet Sullivan >>> *Sent:* Saturday, November 08, 2014 9:26 PM >>> *To:* elasti...@googlegroups.com >>> *Subject:* RE: Nodes not joining after 1.4.0 upgrade >>> >>> >>> But it DOES happen with 1.3.5. Hmmm…. >>> >>> >>> *From:* elasti...@googlegroups.com [mailto:ela...@googlegroups.com] *On >>> Behalf Of *Janet Sullivan >>> *Sent:* Saturday, November 08, 2014 9:24 PM >>> *To:* elasti...@googlegroups.com >>> *Subject:* RE: Nodes not joining after 1.4.0 upgrade >>> >>> >>> Note: This issue doesn’t happen with 1.4.0-beta1 >>> >>> >>> *From:* elasti...@googlegroups.com [mailto:ela...@googlegroups.com] *On >>> Behalf Of *Janet Sullivan >>> *Sent:* Saturday, November 08, 2014 8:46 PM >>> *To:* elasti...@googlegroups.com >>> *Subject:* Nodes not joining after 1.4.0 upgrade >>> >>> >>> I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of >>> them, I had nodes that spewed the following, and were slow to join, if they >>> joined at all: >>> >>> >>> [2014-11-09 04:33:45,995][INFO ][discovery.zen ] >>> [gnslogstash3] failed to send join request to master >>> [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, >>> >>> master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for >>> task.]] >>> >>> [2014-11-09 04:34:49,776][INFO ][discovery.zen ] >>> [gnslogstash3] failed to send join request to master >>> [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, >>> >>> master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for >>> task.]] >>> >>> [2014-11-09 04:35:53,571][INFO ][discovery.zen ] >>> [gnslogstash3] failed to send join request to master >>> [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, >>> >>> master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for >>> task.]] >>> >>> [2014-11-09 04:36:57,353][INFO ][discovery.zen ] >>> [gnslogstash3] failed to send join request to master >>> [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, >>> >>> master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for >>> task.]] >>> >>> [2014-11-09 04:38:01,120][INFO ][discovery.zen ] >>> [gnslogstash3] failed to send join request to master >>> [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, >>> >>> master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for >>> task.]] >>> >>> [2014-11-09 04:39:04,885][INFO ][discovery.zen ] >>> [gnslogstash3] failed to send join request to master >>> [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, >>> >>> master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for >>> task.]] >>> >>> [2014-11-09 04:40:08,657][INFO ][discovery.zen ] >>> [gnslogstash3] failed to send join request to master >>> [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, >>> >>> master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for >>> task.]] >>> >>> >>> I’m able to telnet to port 9300 on gnslogstash10 in this example from >>> gnslogstash3 with no issue, but this cluster doesn’t want to bring all its >>> nodes up. The more nodes added, the more likely a join will fail. In this >>> example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearc...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com >>> >>> <https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearc...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com >>> >>> <https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearc...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com >>> >>> <https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearc...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com >>> >>> <https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com > > <https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4ddc8151-5c10-4e35-871d-655316ef2d48%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
