Hi, Akka cluster uses gossip protocol for Master election. The approach in Spark right now is to use Zookeeper for high availability.
Interestingly Cassandra and Redis clusters are both using Gossip protocol. I am not sure what is the default behavior right now. If the master dies and zookeeper selects a new master, the whole depedency graph will be re-executed or only the unfinished stages will be restarted ? Also why the zookeeper based HA was preferred in Spark ? I was wondering if there is JIRA to add gossip protocol for Spark Master election ? In the code I see zookeeper, filesystem, custom and default is MonarchyLeader. So looks like Spark is designed to add new leaderElectionAgent. Thanks. Deb
