[jira] [Commented] (MESOS-890) Figure out a way to migrate a live Mesos cluster to a different ZooKeeper cluster
[ https://issues.apache.org/jira/browse/MESOS-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13925447#comment-13925447 ] Vinod Kone commented on MESOS-890: -- Having slaves and schedulers be able to do leader detection across multiple zookeeper clusters requires significant amount of code change in mesos. The arguments against it are the same as what Raul pointed in #2. Figure out a way to migrate a live Mesos cluster to a different ZooKeeper cluster - Key: MESOS-890 URL: https://issues.apache.org/jira/browse/MESOS-890 Project: Mesos Issue Type: Improvement Components: master Reporter: Raul Gutierrez Segales Assignee: Raul Gutierrez Segales I've been chatting with [~vinodkone] about approaching a live ZK cluster migration. Here are the options we came up with. For the descriptions we treat `zk1` as the current working cluster, `obs` as a bunch of ZooKeeper Observers [1] and `zk2` as the new cluster to which we need to migrate. Approach #1: Using Observers With this option we need to: * add obs to zk1 * restart slaves to have them use obs to find their master * restart the framework having it use obs to find the mesos master * restart the mesos masters having them use obs to perform their election * we then stop all ZK obs and remove their data (since they will need to sync up with an entirely new cluster, we need to lose the old data) * we restart ZK obs having them be part of zk2 * at this point the slaves, the framework and the masters can reach the ZK obs again and an election happens * optionally you can restart slaves, the framework and masters again using zk2 instead of the ZK obs if you wanted to decommission them. This assumes that we can do the last three steps in 75 secs (75 secs being the slave health check timeout). This is a reasonable assumption if the data size in zk2 is small enough to ensure that the ZK obs can sync up quickly with zk2. If zk2 is a new cluster with no data then this should be very fast. The good things of this approach are: * no mesos code change * it is very easy to rollback half way through, if need be The hard issues are: * Manipulating the ZK obs (i.e.: stopping, removing the data from zk1 and starting again) needs to be done with care. Messing up configs or not removing the data from zk1 on any of the ZK obs will cause problems * we need to restart all slaves to have them use the ZK obs instead of connecting to zk1 directly. But with slave recovery this isn't an issue, just an extra step. * same thing for the framework and the masters Approach #2: Dual publishing from mesos masters With this option we would augment the election handling code in mesos masters to have it deal with the notion of a primary and secondary ZK clusters. Master registration and election would then work as follows: * create an ephemeral|sequential znode in zk1 (i.e.: /path/to/znode/mesos_23) * create an ephemeral, but not sequential, znode in zk2 with the exact same path as what was created in zk1 (i.e.: /path/to/znode/mesos_23) * make sure both sessions, in zk1 and zk2, are always in the same state (i.e.: if one expires, the other one should be closed, etc.) For now, lets omit a few implementation details which might need extra care and assume we can make this work consistently in such a way that zk2 reflects accurately elections that happen in zk1. This means that regardless of being connected to zk1 or zk2, you always get the same master. Once we have this the migration steps would be: * restart slaves to have them use zk2 where masters can be found by virtue of what we implemented above * restart the framework so that it finds the mesos master in zk2 * stop all mesos masters (they all need to be stopped before moving to the next step) * start all mesos masters using zk2 as its primary and only cluster Again, this assumes we can do the last two steps in 75 secs (or if we needed to, we could bump the slave health check timeout). Which, again, sounds achievable given that masters have no state and their start-up time is very short. The good things of this approach are: - no tinkering with extra ZK servers nor with ZK configs The hard issues are: - extra code needs to be added to the election handling bits of mesos master to address a very rare, but probable, use-case of cluster migration. It might take a bit of time to get that code right. - it's easier to end up with a bad state if any of the mesos masters ends up with a bad config or is restarted earlier and ends up publishing differently than the other masters. This could lead to elections with differing results. Thoughts? [1]
[jira] [Commented] (MESOS-890) Figure out a way to migrate a live Mesos cluster to a different ZooKeeper cluster
[ https://issues.apache.org/jira/browse/MESOS-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13914707#comment-13914707 ] Bill Farner commented on MESOS-890: --- Ideally the approach we come up with doesn't require others to have a ZooKeeper expert to replicate. Has the inverse of #2 been considered? i.e. have the slaves look for the master in both locations. This requires more deploy steps, but AFAICT they are relatively easy to reason about. Figure out a way to migrate a live Mesos cluster to a different ZooKeeper cluster - Key: MESOS-890 URL: https://issues.apache.org/jira/browse/MESOS-890 Project: Mesos Issue Type: Improvement Components: master Reporter: Raul Gutierrez Segales Assignee: Raul Gutierrez Segales I've been chatting with [~vinodkone] about approaching a live ZK cluster migration. Here are the options we came up with. For the descriptions we treat `zk1` as the current working cluster, `obs` as a bunch of ZooKeeper Observers [1] and `zk2` as the new cluster to which we need to migrate. Approach #1: Using Observers With this option we need to: * add obs to zk1 * restart slaves to have them use obs to find their master * restart the framework having it use obs to find the mesos master * restart the mesos masters having them use obs to perform their election * we then stop all ZK obs and remove their data (since they will need to sync up with an entirely new cluster, we need to lose the old data) * we restart ZK obs having them be part of zk2 * at this point the slaves, the framework and the masters can reach the ZK obs again and an election happens * optionally you can restart slaves, the framework and masters again using zk2 instead of the ZK obs if you wanted to decommission them. This assumes that we can do the last three steps in 75 secs (75 secs being the slave health check timeout). This is a reasonable assumption if the data size in zk2 is small enough to ensure that the ZK obs can sync up quickly with zk2. If zk2 is a new cluster with no data then this should be very fast. The good things of this approach are: * no mesos code change * it is very easy to rollback half way through, if need be The hard issues are: * Manipulating the ZK obs (i.e.: stopping, removing the data from zk1 and starting again) needs to be done with care. Messing up configs or not removing the data from zk1 on any of the ZK obs will cause problems * we need to restart all slaves to have them use the ZK obs instead of connecting to zk1 directly. But with slave recovery this isn't an issue, just an extra step. * same thing for the framework and the masters Approach #2: Dual publishing from mesos masters With this option we would augment the election handling code in mesos masters to have it deal with the notion of a primary and secondary ZK clusters. Master registration and election would then work as follows: * create an ephemeral|sequential znode in zk1 (i.e.: /path/to/znode/mesos_23) * create an ephemeral, but not sequential, znode in zk2 with the exact same path as what was created in zk1 (i.e.: /path/to/znode/mesos_23) * make sure both sessions, in zk1 and zk2, are always in the same state (i.e.: if one expires, the other one should be closed, etc.) For now, lets omit a few implementation details which might need extra care and assume we can make this work consistently in such a way that zk2 reflects accurately elections that happen in zk1. This means that regardless of being connected to zk1 or zk2, you always get the same master. Once we have this the migration steps would be: * restart slaves to have them use zk2 where masters can be found by virtue of what we implemented above * restart the framework so that it finds the mesos master in zk2 * stop all mesos masters (they all need to be stopped before moving to the next step) * start all mesos masters using zk2 as its primary and only cluster Again, this assumes we can do the last two steps in 75 secs (or if we needed to, we could bump the slave health check timeout). Which, again, sounds achievable given that masters have no state and their start-up time is very short. The good things of this approach are: - no tinkering with extra ZK servers nor with ZK configs The hard issues are: - extra code needs to be added to the election handling bits of mesos master to address a very rare, but probable, use-case of cluster migration. It might take a bit of time to get that code right. - it's easier to end up with a bad state if any of the mesos masters ends up with a bad config or is restarted earlier and ends up publishing differently than the other masters. This could lead to elections
[jira] [Commented] (MESOS-890) Figure out a way to migrate a live Mesos cluster to a different ZooKeeper cluster
[ https://issues.apache.org/jira/browse/MESOS-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913694#comment-13913694 ] David Robinson commented on MESOS-890: -- I think option #1 is the better approach. Writing a heap of code to correctly handle option #2 seems like a lot of effort for something which should very rarely happen. IIRC the 75 second constraint can be adjusted in Aurora too. [~rgs], how difficult is #1 from the ZK perspective? I'm happy to test this in a dev/test cluster. Figure out a way to migrate a live Mesos cluster to a different ZooKeeper cluster - Key: MESOS-890 URL: https://issues.apache.org/jira/browse/MESOS-890 Project: Mesos Issue Type: Improvement Components: master Reporter: Raul Gutierrez Segales Assignee: Raul Gutierrez Segales I've been chatting with [~vinodkone] about approaching a live ZK cluster migration. Here are the options we came up with. For the descriptions we treat `zk1` as the current working cluster, `obs` as a bunch of ZooKeeper Observers [1] and `zk2` as the new cluster to which we need to migrate. Approach #1: Using Observers With this option we need to: * add obs to zk1 * restart slaves to have them use obs to find their master * restart the framework having it use obs to find the mesos master * restart the mesos masters having them use obs to perform their election * we then stop all ZK obs and remove their data (since they will need to sync up with an entirely new cluster, we need to lose the old data) * we restart ZK obs having them be part of zk2 * at this point the slaves, the framework and the masters can reach the ZK obs again and an election happens * optionally you can restart slaves, the framework and masters again using zk2 instead of the ZK obs if you wanted to decommission them. This assumes that we can do the last three steps in 75 secs (75 secs being the slave health check timeout). This is a reasonable assumption if the data size in zk2 is small enough to ensure that the ZK obs can sync up quickly with zk2. If zk2 is a new cluster with no data then this should be very fast. The good things of this approach are: * no mesos code change * it is very easy to rollback half way through, if need be The hard issues are: * Manipulating the ZK obs (i.e.: stopping, removing the data from zk1 and starting again) needs to be done with care. Messing up configs or not removing the data from zk1 on any of the ZK obs will cause problems * we need to restart all slaves to have them use the ZK obs instead of connecting to zk1 directly. But with slave recovery this isn't an issue, just an extra step. * same thing for the framework and the masters Approach #2: Dual publishing from mesos masters With this option we would augment the election handling code in mesos masters to have it deal with the notion of a primary and secondary ZK clusters. Master registration and election would then work as follows: * create an ephemeral|sequential znode in zk1 (i.e.: /path/to/znode/mesos_23) * create an ephemeral, but not sequential, znode in zk2 with the exact same path as what was created in zk1 (i.e.: /path/to/znode/mesos_23) * make sure both sessions, in zk1 and zk2, are always in the same state (i.e.: if one expires, the other one should be closed, etc.) For now, lets omit a few implementation details which might need extra care and assume we can make this work consistently in such a way that zk2 reflects accurately elections that happen in zk1. This means that regardless of being connected to zk1 or zk2, you always get the same master. Once we have this the migration steps would be: * restart slaves to have them use zk2 where masters can be found by virtue of what we implemented above * restart the framework so that it finds the mesos master in zk2 * stop all mesos masters (they all need to be stopped before moving to the next step) * start all mesos masters using zk2 as its primary and only cluster Again, this assumes we can do the last two steps in 75 secs (or if we needed to, we could bump the slave health check timeout). Which, again, sounds achievable given that masters have no state and their start-up time is very short. The good things of this approach are: - no tinkering with extra ZK servers nor with ZK configs The hard issues are: - extra code needs to be added to the election handling bits of mesos master to address a very rare, but probable, use-case of cluster migration. It might take a bit of time to get that code right. - it's easier to end up with a bad state if any of the mesos masters ends up with a bad config or is restarted earlier and ends up publishing differently than
[jira] [Commented] (MESOS-890) Figure out a way to migrate a live Mesos cluster to a different ZooKeeper cluster
[ https://issues.apache.org/jira/browse/MESOS-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13871445#comment-13871445 ] Raul Gutierrez Segales commented on MESOS-890: -- How long can slaves be master-less? Figure out a way to migrate a live Mesos cluster to a different ZooKeeper cluster - Key: MESOS-890 URL: https://issues.apache.org/jira/browse/MESOS-890 Project: Mesos Issue Type: Improvement Components: master Reporter: Raul Gutierrez Segales I've been chatting with [~vinodkone] about approaching a live ZK cluster migration. Here are the options we came up with. For the descriptions we treat `zk1` as the current working cluster, `obs` as a bunch of ZooKeeper Observers [1] and `zk2` as the new cluster to which we need to migrate. Approach #1: Using Observers With this option we need to: * add obs to zk1 * restart slaves to have them use obs to find their master * restart the framework having it use obs to find the mesos master * restart the mesos masters having them use obs to perform their election * we then stop all ZK obs and remove their data (since they will need to sync up with an entirely new cluster, we need to lose the old data) * we restart ZK obs having them be part of zk2 * at this point the slaves, the framework and the masters can reach the ZK obs again and an election happens * optionally you can restart slaves, the framework and masters again using zk2 instead of the ZK obs if you wanted to decommission them. This assumes that we can do the last three steps in 75 secs (75 secs being the slave health check timeout). This is a reasonable assumption if the data size in zk2 is small enough to ensure that the ZK obs can sync up quickly with zk2. If zk2 is a new cluster with no data then this should be very fast. The good things of this approach are: * no mesos code change * it is very easy to rollback half way through, if need be The hard issues are: * Manipulating the ZK obs (i.e.: stopping, removing the data from zk1 and starting again) needs to be done with care. Messing up configs or not removing the data from zk1 on any of the ZK obs will cause problems * we need to restart all slaves to have them use the ZK obs instead of connecting to zk1 directly. But with slave recovery this isn't an issue, just an extra step. * same thing for the framework and the masters Approach #2: Dual publishing from mesos masters With this option we would augment the election handling code in mesos masters to have it deal with the notion of a primary and secondary ZK clusters. Master registration and election would then work as follows: * create an ephemeral|sequential znode in zk1 (i.e.: /path/to/znode/mesos_23) * create an ephemeral, but not sequential, znode in zk2 with the exact same path as what was created in zk1 (i.e.: /path/to/znode/mesos_23) * make sure both sessions, in zk1 and zk2, are always in the same state (i.e.: if one expires, the other one should be closed, etc.) For now, lets omit a few implementation details which might need extra care and assume we can make this work consistently in such a way that zk2 reflects accurately elections that happen in zk1. This means that regardless of being connected to zk1 or zk2, you always get the same master. Once we have this the migration steps would be: * restart slaves to have them use zk2 where masters can be found by virtue of what we implemented above * restart the framework so that it finds the mesos master in zk2 * stop all mesos masters (they all need to be stopped before moving to the next step) * start all mesos masters using zk2 as its primary and only cluster Again, this assumes we can do the last two steps in 75 secs (or if we needed to, we could bump the slave health check timeout). Which, again, sounds achievable given that masters have no state and their start-up time is very short. The good things of this approach are: - no tinkering with extra ZK servers nor with ZK configs The hard issues are: - extra code needs to be added to the election handling bits of mesos master to address a very rare, but probable, use-case of cluster migration. It might take a bit of time to get that code right. - it's easier to end up with a bad state if any of the mesos masters ends up with a bad config or is restarted earlier and ends up publishing differently than the other masters. This could lead to elections with differing results. Thoughts? [1] http://zookeeper.apache.org/doc/trunk/zookeeperObservers.html -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MESOS-890) Figure out a way to migrate a live Mesos cluster to a different ZooKeeper cluster
[ https://issues.apache.org/jira/browse/MESOS-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13871471#comment-13871471 ] Vinod Kone commented on MESOS-890: -- forever. But frameworks like Aurora might transition tasks to LOST if they don't hear updates from them for a while. Figure out a way to migrate a live Mesos cluster to a different ZooKeeper cluster - Key: MESOS-890 URL: https://issues.apache.org/jira/browse/MESOS-890 Project: Mesos Issue Type: Improvement Components: master Reporter: Raul Gutierrez Segales I've been chatting with [~vinodkone] about approaching a live ZK cluster migration. Here are the options we came up with. For the descriptions we treat `zk1` as the current working cluster, `obs` as a bunch of ZooKeeper Observers [1] and `zk2` as the new cluster to which we need to migrate. Approach #1: Using Observers With this option we need to: * add obs to zk1 * restart slaves to have them use obs to find their master * restart the framework having it use obs to find the mesos master * restart the mesos masters having them use obs to perform their election * we then stop all ZK obs and remove their data (since they will need to sync up with an entirely new cluster, we need to lose the old data) * we restart ZK obs having them be part of zk2 * at this point the slaves, the framework and the masters can reach the ZK obs again and an election happens * optionally you can restart slaves, the framework and masters again using zk2 instead of the ZK obs if you wanted to decommission them. This assumes that we can do the last three steps in 75 secs (75 secs being the slave health check timeout). This is a reasonable assumption if the data size in zk2 is small enough to ensure that the ZK obs can sync up quickly with zk2. If zk2 is a new cluster with no data then this should be very fast. The good things of this approach are: * no mesos code change * it is very easy to rollback half way through, if need be The hard issues are: * Manipulating the ZK obs (i.e.: stopping, removing the data from zk1 and starting again) needs to be done with care. Messing up configs or not removing the data from zk1 on any of the ZK obs will cause problems * we need to restart all slaves to have them use the ZK obs instead of connecting to zk1 directly. But with slave recovery this isn't an issue, just an extra step. * same thing for the framework and the masters Approach #2: Dual publishing from mesos masters With this option we would augment the election handling code in mesos masters to have it deal with the notion of a primary and secondary ZK clusters. Master registration and election would then work as follows: * create an ephemeral|sequential znode in zk1 (i.e.: /path/to/znode/mesos_23) * create an ephemeral, but not sequential, znode in zk2 with the exact same path as what was created in zk1 (i.e.: /path/to/znode/mesos_23) * make sure both sessions, in zk1 and zk2, are always in the same state (i.e.: if one expires, the other one should be closed, etc.) For now, lets omit a few implementation details which might need extra care and assume we can make this work consistently in such a way that zk2 reflects accurately elections that happen in zk1. This means that regardless of being connected to zk1 or zk2, you always get the same master. Once we have this the migration steps would be: * restart slaves to have them use zk2 where masters can be found by virtue of what we implemented above * restart the framework so that it finds the mesos master in zk2 * stop all mesos masters (they all need to be stopped before moving to the next step) * start all mesos masters using zk2 as its primary and only cluster Again, this assumes we can do the last two steps in 75 secs (or if we needed to, we could bump the slave health check timeout). Which, again, sounds achievable given that masters have no state and their start-up time is very short. The good things of this approach are: - no tinkering with extra ZK servers nor with ZK configs The hard issues are: - extra code needs to be added to the election handling bits of mesos master to address a very rare, but probable, use-case of cluster migration. It might take a bit of time to get that code right. - it's easier to end up with a bad state if any of the mesos masters ends up with a bad config or is restarted earlier and ends up publishing differently than the other masters. This could lead to elections with differing results. Thoughts? [1] http://zookeeper.apache.org/doc/trunk/zookeeperObservers.html -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MESOS-890) Figure out a way to migrate a live Mesos cluster to a different ZooKeeper cluster
[ https://issues.apache.org/jira/browse/MESOS-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863256#comment-13863256 ] Benjamin Mahler commented on MESOS-890: --- It looks like the 75 second constraint can be removed from approach 1 by re-ordering the steps listed above. That is, you could stop the old masters before performing the ZK observer reconfiguration. Now, all the slaves will simply wait idle for a new master. If these masters are also part of the new configuration, then they could be started again after the ZK observers are up pointed to zk2. Figure out a way to migrate a live Mesos cluster to a different ZooKeeper cluster - Key: MESOS-890 URL: https://issues.apache.org/jira/browse/MESOS-890 Project: Mesos Issue Type: Improvement Components: master Reporter: Raul Gutierrez Segales I've been chatting with [~vinodkone] about approaching a live ZK cluster migration. Here are the options we came up with. For the descriptions we treat `zk1` as the current working cluster, `obs` as a bunch of ZooKeeper Observers [1] and `zk2` as the new cluster to which we need to migrate. Approach #1: Using Observers With this option we need to: * add obs to zk1 * restart slaves to have them use obs to find their master * restart the framework having it use obs to find the mesos master * restart the mesos masters having them use obs to perform their election * we then stop all ZK obs and remove their data (since they will need to sync up with an entirely new cluster, we need to lose the old data) * we restart ZK obs having them be part of zk2 * at this point the slaves, the framework and the masters can reach the ZK obs again and an election happens * optionally you can restart slaves, the framework and masters again using zk2 instead of the ZK obs if you wanted to decommission them. This assumes that we can do the last three steps in 75 secs (75 secs being the slave health check timeout). This is a reasonable assumption if the data size in zk2 is small enough to ensure that the ZK obs can sync up quickly with zk2. If zk2 is a new cluster with no data then this should be very fast. The good things of this approach are: * no mesos code change * it is very easy to rollback half way through, if need be The hard issues are: * Manipulating the ZK obs (i.e.: stopping, removing the data from zk1 and starting again) needs to be done with care. Messing up configs or not removing the data from zk1 on any of the ZK obs will cause problems * we need to restart all slaves to have them use the ZK obs instead of connecting to zk1 directly. But with slave recovery this isn't an issue, just an extra step. * same thing for the framework and the masters Approach #2: Dual publishing from mesos masters With this option we would augment the election handling code in mesos masters to have it deal with the notion of a primary and secondary ZK clusters. Master registration and election would then work as follows: * create an ephemeral|sequential znode in zk1 (i.e.: /path/to/znode/mesos_23) * create an ephemeral, but not sequential, znode in zk2 with the exact same path as what was created in zk1 (i.e.: /path/to/znode/mesos_23) * make sure both sessions, in zk1 and zk2, are always in the same state (i.e.: if one expires, the other one should be closed, etc.) For now, lets omit a few implementation details which might need extra care and assume we can make this work consistently in such a way that zk2 reflects accurately elections that happen in zk1. This means that regardless of being connected to zk1 or zk2, you always get the same master. Once we have this the migration steps would be: * restart slaves to have them use zk2 where masters can be found by virtue of what we implemented above * restart the framework so that it finds the mesos master in zk2 * stop all mesos masters (they all need to be stopped before moving to the next step) * start all mesos masters using zk2 as its primary and only cluster Again, this assumes we can do the last two steps in 75 secs (or if we needed to, we could bump the slave health check timeout). Which, again, sounds achievable given that masters have no state and their start-up time is very short. The good things of this approach are: - no tinkering with extra ZK servers nor with ZK configs The hard issues are: - extra code needs to be added to the election handling bits of mesos master to address a very rare, but probable, use-case of cluster migration. It might take a bit of time to get that code right. - it's easier to end up with a bad state if any of the mesos masters ends up with a bad config or is restarted earlier and ends up
