Re: Accumulator in SparkUI for streaming

[Tim Smith]

So somehow Spark Streaming doesn't support display of named accumulators in
the WebUI?


On Tue, Feb 24, 2015 at 7:58 AM, Petar Zecevic petar.zece...@gmail.com
wrote:


 Interesting. Accumulators are shown on Web UI if you are using the
 ordinary SparkContext (Spark 1.2). It just has to be named (and that's what
 you did).

 scala val acc = sc.accumulator(0, test accumulator)
 acc: org.apache.spark.Accumulator[Int] = 0
 scala val rdd = sc.parallelize(1 to 1000)
 rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at
 parallelize at console:12
 scala rdd.foreach(x = acc += 1)
 scala acc.value
 res1: Int = 1000

 The Stage details page shows:




 On 20.2.2015. 9:25, Tim Smith wrote:

  On Spark 1.2:

  I am trying to capture # records read from a kafka topic:

  val inRecords = ssc.sparkContext.accumulator(0, InRecords)

  ..

  kInStreams.foreach( k =
 {

   k.foreachRDD ( rdd =  inRecords += rdd.count().toInt  )
   inRecords.value


  Question is how do I get the accumulator to show up in the UI? I tried
 inRecords.value but that didn't help. Pretty sure it isn't showing up in
 Stage metrics.

  What's the trick here? collect?

  Thanks,

  Tim

Re: Accumulator in SparkUI for streaming

[Petar Zecevic]

Interesting. Accumulators are shown on Web UI if you are using the 
ordinary SparkContext (Spark 1.2). It just has to be named (and that's 
what you did).


scala val acc = sc.accumulator(0, test accumulator)
acc: org.apache.spark.Accumulator[Int] = 0
scala val rdd = sc.parallelize(1 to 1000)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at 
parallelize at console:12

scala rdd.foreach(x = acc += 1)
scala acc.value
res1: Int = 1000

The Stage details page shows:




On 20.2.2015. 9:25, Tim Smith wrote:

On Spark 1.2:

I am trying to capture # records read from a kafka topic:

val inRecords = ssc.sparkContext.accumulator(0, InRecords)

..

kInStreams.foreach( k =
{

 k.foreachRDD ( rdd =  inRecords += rdd.count().toInt  )
 inRecords.value


Question is how do I get the accumulator to show up in the UI? I tried 
inRecords.value but that didn't help. Pretty sure it isn't showing 
up in Stage metrics.


What's the trick here? collect?

Thanks,

Tim

Re: Accumulator in SparkUI for streaming

[Tathagata Das]

Unless I am unaware some latest changes, the SparkUI shows stages, and
jobs, not accumulator results. And the UI not designed to be pluggable for
showing user-defined stuff.

TD

On Fri, Feb 20, 2015 at 12:25 AM, Tim Smith secs...@gmail.com wrote:

 On Spark 1.2:

 I am trying to capture # records read from a kafka topic:

 val inRecords = ssc.sparkContext.accumulator(0, InRecords)

 ..

 kInStreams.foreach( k =
 {

  k.foreachRDD ( rdd =  inRecords += rdd.count().toInt  )
  inRecords.value


 Question is how do I get the accumulator to show up in the UI? I tried
 inRecords.value but that didn't help. Pretty sure it isn't showing up in
 Stage metrics.

 What's the trick here? collect?

 Thanks,

 Tim

Accumulator in SparkUI for streaming

[Tim Smith]

On Spark 1.2:

I am trying to capture # records read from a kafka topic:

val inRecords = ssc.sparkContext.accumulator(0, InRecords)

..

kInStreams.foreach( k =
{

 k.foreachRDD ( rdd =  inRecords += rdd.count().toInt  )
 inRecords.value


Question is how do I get the accumulator to show up in the UI? I tried
inRecords.value but that didn't help. Pretty sure it isn't showing up in
Stage metrics.

What's the trick here? collect?

Thanks,

Tim