Interesting. Accumulators are shown on Web UI if you are using the
ordinary SparkContext (Spark 1.2). It just has to be named (and that's
what you did).
scala> val acc = sc.accumulator(0, "test accumulator")
acc: org.apache.spark.Accumulator[Int] = 0
scala> val rdd = sc.parallelize(1 to 1000)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at
parallelize at <console>:12
scala> rdd.foreach(x => acc += 1)
scala> acc.value
res1: Int = 1000
The Stage details page shows:
On 20.2.2015. 9:25, Tim Smith wrote:
On Spark 1.2:
I am trying to capture # records read from a kafka topic:
val inRecords = ssc.sparkContext.accumulator(0, "InRecords")
..
kInStreams.foreach( k =>
{
k.foreachRDD ( rdd => inRecords += rdd.count().toInt )
inRecords.value
Question is how do I get the accumulator to show up in the UI? I tried
"inRecords.value" but that didn't help. Pretty sure it isn't showing
up in Stage metrics.
What's the trick here? collect?
Thanks,
Tim
