In retrospect I should've (might still be able to one of these days) open sourced the tool we used to migrate mesos masters. That said, overall the process suggested so far is correct.
To validate the new host joining, you can tail the master log file for "Successfully joined the Paxos group <https://github.com/apache/mesos/blob/3539b7a0e15b594148308319bf052d28b1429b98/src/log/recover.cpp#L578>" to confirm the replicated log recovery has completed for that machine. Once that happens, feel free to move onto adding the next host. On Tue, Nov 24, 2015 at 9:40 PM, Du, Fan <fan...@intel.com> wrote: > > > On 2015/11/24 9:47, Chengwei Yang wrote: > >> Hi all, >> >> We're using mesos in product on CentOS 6 and plan to upgrade CentOS to >> 7.1, to >> avoid affect any tasks running on mesos. We're about to replace all >> mesos-masters in fly. >> >> The procedure listed below: >> >> 0. 3 mesos-masters running on CentOS 6 >> 1. shutdown 1 mesos-master(CentOS 6) and bring up 1 mesos-master(CentOS 7) >> wait the new master synced for some time(is there any simple way to >> know when?) >> > > Login the upgraded master mesos web page, it will redirect you to the > Leader master > if upgraded master join the cluster successfully. > > Or a script friendly way, you can query zookeeper server the role of the > upgraded > master server by: > > echo stat | nc UPGRADED_MASTER_ZOOKEEPER_IP 2181 > > it will report something like this: > > Latency min/avg/max: 0/0/6 > Received: 105 > Sent: 105 > Connections: 2 > Outstanding: 0 > Zxid: 0x30000001c > Mode: *follower* > Node count: 13 > > > > 2. repeat step 1 >> >> NOTE: we plan to shutdown non-leader first, and shutdown the >> leader(CentOS 6) >> last. >> >> Can we do this in such way? Or any other better suggestions? >> >>
