Hi all, I recently sent to the dev mailing list about this contribution, but I thought it might be useful to post it here, since I have seen a lot of people asking about OS-level metrics of Spark. This is the result of the work we have been doing recently in IBM Research around Spark. Essentially, we have extended Spark metrics system to utilize Hyperic Sigar library to capture OS-level metrics and modified the Web UI to visualize those metrics per application. The above functionalities can be configured in the metrics.properties and spark-defaults.conf files. We have recorded a small demo that shows those capabilities which you can find here : https://ibm.app.box.com/s/vyaedlyb444a4zna1215c7puhxliqxdg There is a blog post which gives more details on the functionality here: www.spark.tc/sparkoscope-enabling-spark-optimization-through-cross-stack-monitoring-and-visualization-2/ and also there is a public repo where anyone can try it: https://github.com/ibm-research-ireland/sparkoscope
Hope someone finds it useful! Thanks a lot! Yiannis
