Looks like metrics are not a hot topic to discuss - yet so important to sleep well when jobs are running in production.
I've created Spark-4537 <https://issues.apache.org/jira/browse/SPARK-4537> to track this issue. -kr, Gerard. On Thu, Nov 20, 2014 at 9:25 PM, Gerard Maas <gerard.m...@gmail.com> wrote: > As the Spark Streaming tuning guide indicates, the key indicators of a > healthy streaming job are: > - Processing Time > - Total Delay > > The Spark UI page for the Streaming job [1] shows these two indicators but > the metrics source for Spark Streaming (StreamingSource.scala) [2] does > not. > > Any reasons for that? I would like to monitor job performance through an > external monitor (Ganglia in our case) and I've connected already the > currently published metrics. > > -kr, Gerard. > > > [1] > https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala#L127 > > [2] > https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala >
