Re: How to get the batch information from Streaming UI

You would want to add a listener to your Spark Streaming context. Have a look at the StatsReportListener [1]. [1] http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.scheduler.StatsReportListener 2016-05-17 7:18 GMT+10:00 Samuel Zhou : > Hi, >

How to get the batch information from Streaming UI

Hi, Does anyone know how to get the batch information(like batch time, input size, processing time, status) from Streaming UI by using Scala/Java API ? Because I want to put the information in log files and the streaming jobs are managed by YARN. Thanks, Samuel