Asynchrony is not supported directly - spark's programming model is naturally BSP. I have seen cases where people have instantiated actors with akka on worker nodes to enable message passing, or even used spark's own ActorSystem to do this. But, I do not recommend this, since you lose a bunch of benefits of spark - e.g. fault tolerance.
Instead, I would think about whether your algorithm can be cast as a BSP one, or think about how frequently you really need to synchronize state among your workers. It may be that having the occasional synchronization barrier is OK. On Wed, Sep 3, 2014 at 7:28 AM, laxmanvemula <laxman8...@gmail.com> wrote: > Hi, > > I would like to implement an asynchronous distributed optimization > algorithm > where workers communicate among one another. It is similar to belief > propagation where each worker is a vertex in the graph. Can some one let me > know if this is possible using spark? > > Thanks, > Laxman > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Message-Passing-among-workers-tp13355.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >