Job History Logs for spark jobs submitted on YARN

I observe that YARN jobs history logs are created in /user/history/done (*.jhist files) for all the mapreduce jobs like hive, pig etc. But for spark jobs submitted in yarn-cluster mode, the logs are not being created. I would like to see resource utilization by spark jobs. Is there any other

Message Passing among workers

Hi, I would like to implement an asynchronous distributed optimization algorithm where workers communicate among one another. It is similar to belief propagation where each worker is a vertex in the graph. Can some one let me know if this is possible using spark? Thanks, Laxman -- View this