I ve been looking around for some examples of information of how can the driver and the executors exchange information asynchronously, but have not found much apart from the ActorWordCount.scala streaming example that uses Akka.
Is there any "in-band" (within Spark) method that such communication can be effected, or is out-of-band use of Akka the only bet ? (Something equiv in python?). Ideally I do not want to have to send messages to IP addresses just to worker and driver IDs that spark may keep track of. The problem at hand is the following: I would like that while workers number-crunch based on their RDD partition data, they asynchronously communicate some intermediate results back to the driver. The driver then may update and re-broadcast some new state var back to all the workers, which they can use within the same stage of computation. TIA Manolis. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
