Hi, If the stats are in web UI, they should be flying over the wire and so you can catch the events by implementing SparkListener [1] -- a developer API for custom Spark listeners. That's how web UI gets the data and History Server. I think the stats are sent as accumulator updates in onExecutorMetricsUpdate.
[1] http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.scheduler.SparkListener Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Tue, Aug 30, 2016 at 11:17 PM, Ai Deng <wx.london....@gmail.com> wrote: > Hi there, > > I think the metrics inside of the different SparkPlan (like "numOutputRows" > in FilterExec) are useful to build any Dev dashboard or business monitoring. > Are there a easy way or exist solution to expose and persist these metrics > out of Spark UI (ex: send to Graphite)? Currently they are all "private" > inside Spark library. > > And the main benefit is you can get these metrics for "free" without change > your Spark application. > > Thanks and regards, > > Ai > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/reuse-the-Spark-SQL-internal-metrics-tp27626.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
