Heya, We will release aws plugin 2.4.1 in some minutes. It fixes this rolling upgrade issue.
Note that some WARN messages could appear in old nodes logs until the full rolling upgrade is done. Thank you all for reporting this! Le dimanche 23 novembre 2014 03:10:42 UTC+1, Ivan Brusic a écrit : > > Great work everyone. Feel better about upgrading now. > On Nov 22, 2014 4:42 PM, "Boaz Leskes" <b.les...@gmail.com> wrote: > >> Hi Christian, Daniel, >> >> I believe I found the issue - it has to do with the cloud plugins (both >> AWS and GCE) and the way they create the node list for the unicast based >> discovery. Effectively they mislead it to think that that all nodes on the >> cluster are version 1.4.0 which is not correct. >> >> I opened issues for this so it will be corrected soon: >> https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/143 , >> https://github.com/elasticsearch/elasticsearch-cloud-gce/issues/41 >> >> Cheers, >> Boaz >> >> >> On Saturday, November 22, 2014 7:04:33 PM UTC+2, Jörg Prante wrote: >>> >>> As said, the change is due to unicast action, which was split in 1.4.0 >>> to an old and a new action, see this commit: >>> >>> https://github.com/elasticsearch/elasticsearch/commit/ >>> e5de47d928582694c7729d199390086983779e6e >>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch%2Fcommit%2Fe5de47d928582694c7729d199390086983779e6e&sa=D&sntz=1&usg=AFQjCNFQkgiVz8SfE_dZ5Sa5K7TqYCIQ6g> >>> >>> I am not sure if this is a bug. It seems like a feature to prevent >>> multiple masters by accident. >>> >>> The strategy as described above by Christian Hedegaard should work, it >>> is still to be considered a work-around: >>> >>> - setting up all new 1.4 nodes as not master eligible ("data only") >>> >>> - joining them to a 1.3.x cluster while master still is on a 1.3 node >>> should work >>> >>> - then, shutting down all 1.3 nodes (except the master) should relocate >>> the shards >>> >>> - bringing down the final 1.3 master should "stall" master election (I >>> would also configure a large timeout for master election). This is >>> critical, no index/mapping creations/deletions or cluster state modifying >>> actions should be executed now. >>> >>> - adding a 1.4 master eligible node should now overtake the cluster (I >>> would start it with the data folder from the final 1.3 master where the >>> last cluster state is persisted) and the critical phase is over. >>> >>> - from then, more 1.4 master eligible nodes should be possible to add >>> >>> - finally, the minimum master nodes setting should be configured >>> >>> Jörg >>> >>> >>> On Fri, Nov 21, 2014 at 1:56 AM, Christian Hedegaard < >>> chedega...@red5studios.com> wrote: >>> >>>> FYI, I have found a solution that works (at least for me). >>>> >>>> >>>> >>>> I’ve got a small cluster for testing, only 4 v1.3.5 nodes. What I’ve >>>> done is bring up 4X new v1.4.0 nodes as data-only machines. In the yaml I >>>> added a line to point the nodes via unicast explicitly to the current >>>> master: >>>> >>>> discovery.zen.ping.unicast.hosts: ["10.210.9.224:9300"] >>>> >>>> >>>> >>>> When I restarted elasticsearch with that setting, with cloud-aws >>>> installed and configured on version 2.4.0, the new nodes found the cluster >>>> and properly joined it. >>>> >>>> >>>> >>>> I will now start nuking the old v1.3.5 nodes to migrate the data off of >>>> them. Before the final 1.3.5 node is nuked, I will change the config on >>>> one >>>> of the v1.4.0 nodes to allow it as master and restart it. >>>> >>>> >>>> >>>> I’m not sure if the master stuff is needed or not, but I was very >>>> afraid of a split-brain problem. I have another 4-node testing cluster >>>> that >>>> I will be able to try this upgrade again with in a more controlled manner. >>>> >>>> >>>> >>>> I’m NOT looking forward to upgrading our current production cluster >>>> this way (15 data-only nodes, 3 master-only nodes). >>>> >>>> >>>> >>>> So it would appear that the problem is somewhere in the unicast >>>> discovery code. The question is who’s to blame? Elasticsearch or the >>>> cloud-aws plugin? >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *From:* Boaz Leskes [mailto:b.les...@gmail.com] >>>> *Sent:* Wednesday, November 19, 2014 2:27 PM >>>> *To:* elasticsearch@googlegroups.com >>>> *Cc:* Christian Hedegaard >>>> *Subject:* Re: 1.4.0 data node can't join existing 1.3.4 cluster >>>> >>>> >>>> >>>> Hi Christian, >>>> >>>> >>>> >>>> I'm not sure what thread you refer to exactly, but this shouldn't >>>> happen. Can you describe the problem you have some more? Anything in the >>>> nodes? (both the 1.4 node and the master) >>>> >>>> >>>> >>>> Cheers, >>>> >>>> Boaz >>>> >>>> >>>> On Wednesday, November 19, 2014 2:39:57 AM UTC+1, Christian Hedegaard >>>> wrote: >>>> >>>> I found this thread while trying to research the same issue and it >>>> looks like there is currently no resolution. We like to keep up on our >>>> elasticsearch upgrades as often as possible and do rolling upgrades to >>>> keep >>>> our clusters up. When testing I’m having the same issue, I cannot add a >>>> 1.4.0 box to the existing 1.3.4 cluster. >>>> >>>> >>>> >>>> Is there a fix for this anticipated? >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to elasticsearch+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit https://groups.google.com/d/ >>>> msgid/elasticsearch/5CF8216AA982AF47A8E6DEACA629D2 >>>> 2B4EBF409B%40s-us-ex-6.US.R5S.com >>>> <https://groups.google.com/d/msgid/elasticsearch/5CF8216AA982AF47A8E6DEACA629D22B4EBF409B%40s-us-ex-6.US.R5S.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/0bc369d9-1cd1-47ef-ba14-12fa29f5fd4b%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/0bc369d9-1cd1-47ef-ba14-12fa29f5fd4b%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c6a4d157-8f10-485d-a52d-a6cc192e08ef%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
