Job History Logs for spark jobs submitted on YARN

[laxmanvemula]

I observe that YARN jobs history logs are created in /user/history/done
(*.jhist files) for all the mapreduce jobs like hive, pig etc. But for spark
jobs submitted in yarn-cluster mode, the logs are not being created.

I would like to see resource utilization by spark jobs. Is there any other
place where I can find the resource utilization by spark jobs (CPU, Memory
etc). Or is there any configuration to be set so that the job history logs
are created just like other mapreduce jobs.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Job-History-Logs-for-spark-jobs-submitted-on-YARN-tp25946.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Message Passing among workers

[laxmanvemula]

Hi,

I would like to implement an asynchronous distributed optimization algorithm
where workers communicate among one another. It is similar to belief
propagation where each worker is a vertex in the graph. Can some one let me
know if this is possible using spark?

Thanks,
Laxman



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Message-Passing-among-workers-tp13355.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org