Hi,
I want to enable Spark Master HA in spark. Documentation specifies that we
can do this with the help of Zookeepers. But what I am worried is how to
configure one master with the other and similarly how do workers know that
the have two masters? where do you specify the multi-master
The only way i know to do this is to use mesos with zookeepers. you specify
zookeeper url as spark url that contains multiple zookeeper hosts. Multiple
mesos masters are then elected thru zookeeper leader election until current
leader dies; at which point mesos will elect another master (if still
Thanks Dmitriy. But I want multi master support when running spark
standalone. Also I want to know if this multi master thing works if I use
spark-shell.
On Wed, Apr 9, 2014 at 3:26 PM, Dmitriy Lyubimov dlie...@gmail.com wrote:
The only way i know to do this is to use mesos with zookeepers.
It is as Jagat said. The Masters do not need to know about one another, as
ZooKeeper manages their implicit communication. As for Workers (and
applications, such as spark-shell), once a Worker is registered with
*some *Master,
its metadata is stored in ZooKeeper such that if another Master is
