Hi Chen, Have you looked at JobLogger.scala? You can register it with SparkContext and it will log task metrics onTaskEnd. Unfortunately, there is currently not a configurable option to enable this. We are looking into a more systematic way to persist the UI information in a more flexible format (JSON).
Andrew 2014-02-11 11:14 GMT-08:00 Chen Jin <karen...@gmail.com>: > Hi All, > > Apart from getting the shuffle read/write size from application UI, is > there any way to enable the metrics in the log file? > > Thank you very much, > > -chen >
