Hi guys, That's great to hear that this is available in Spark 1.3! .. I will play around with this feature and let you know the results for integrating Hazelcast. Also, may I know the tentative release date for Spark 1.3? ..
Cheers, Anjana. On Mon, Feb 2, 2015 at 3:07 AM, Aaron Davidson <ilike...@gmail.com> wrote: > For the specific question of supplementing Standalone Mode with a custom > leader election protocol, this was actually already committed in master and > will be available in Spark 1.3: > > https://github.com/apache/spark/pull/771/files > > You can specify spark.deploy.recoveryMode = "CUSTOM" > and spark.deploy.recoveryMode.factory to a class which > implements StandaloneRecoveryModeFactory. See the current implementations > of FileSystemRecoveryModeFactory and ZooKeeperRecoveryModeFactory. > > I will update the JIRA you linked to be more current. > > On Sat, Jan 31, 2015 at 12:55 AM, Anjana Fernando <laferna...@gmail.com> > wrote: > >> Hi everyone, >> >> I've been experimenting, and somewhat of a newbie for Spark. I was >> wondering, if there is any way, that I can use a custom cluster manager >> implementation with Spark. Basically, as I understood, at the moment, the >> inbuilt modes supported are with standalone, Mesos and Yarn. My >> requirement is basically a simple clustering solution with high >> availability of the master. I don't want to use a separate Zookeeper >> cluster, since this would complicate my deployment, but rather, I would >> like to use something like Hazelcast, which has a peer-to-peer cluster >> coordination implementation. >> >> I found that, there is already this JIRA [1], which requests for a custom >> persistence engine, I guess for storing state information. So basically, >> what I would want to do is, use Hazelcast to use for leader election, to >> make an existing node the master, and to lookup the state information from >> the distributed memory. Appreciate any help on how to archive this. And if >> it useful for a wider audience, hopefully I can contribute this back to >> the >> project. >> >> [1] https://issues.apache.org/jira/browse/SPARK-1180 >> >> Cheers, >> Anjana. >> > >
