So somehow Spark Streaming doesn't support display of named accumulators in
the WebUI?
On Tue, Feb 24, 2015 at 7:58 AM, Petar Zecevic <petar.zece...@gmail.com>
wrote:
>
> Interesting. Accumulators are shown on Web UI if you are using the
> ordinary SparkContext (Spark 1.2). It just has to be named (and that's what
> you did).
>
> scala> val acc = sc.accumulator(0, "test accumulator")
> acc: org.apache.spark.Accumulator[Int] = 0
> scala> val rdd = sc.parallelize(1 to 1000)
> rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at
> parallelize at <console>:12
> scala> rdd.foreach(x => acc += 1)
> scala> acc.value
> res1: Int = 1000
>
> The Stage details page shows:
>
>
>
>
> On 20.2.2015. 9:25, Tim Smith wrote:
>
> On Spark 1.2:
>
> I am trying to capture # records read from a kafka topic:
>
> val inRecords = ssc.sparkContext.accumulator(0, "InRecords")
>
> ..
>
> kInStreams.foreach( k =>
> {
>
> k.foreachRDD ( rdd => inRecords += rdd.count().toInt )
> inRecords.value
>
>
> Question is how do I get the accumulator to show up in the UI? I tried
> "inRecords.value" but that didn't help. Pretty sure it isn't showing up in
> Stage metrics.
>
> What's the trick here? collect?
>
> Thanks,
>
> Tim
>
>
>
