So somehow Spark Streaming doesn't support display of named accumulators in
the WebUI?
On Tue, Feb 24, 2015 at 7:58 AM, Petar Zecevic petar.zece...@gmail.com
wrote:
Interesting. Accumulators are shown on Web UI if you are using the
ordinary SparkContext (Spark 1.2). It just has to be named
Interesting. Accumulators are shown on Web UI if you are using the
ordinary SparkContext (Spark 1.2). It just has to be named (and that's
what you did).
scala val acc = sc.accumulator(0, test accumulator)
acc: org.apache.spark.Accumulator[Int] = 0
scala val rdd = sc.parallelize(1 to 1000)
Unless I am unaware some latest changes, the SparkUI shows stages, and
jobs, not accumulator results. And the UI not designed to be pluggable for
showing user-defined stuff.
TD
On Fri, Feb 20, 2015 at 12:25 AM, Tim Smith secs...@gmail.com wrote:
On Spark 1.2:
I am trying to capture # records
On Spark 1.2:
I am trying to capture # records read from a kafka topic:
val inRecords = ssc.sparkContext.accumulator(0, InRecords)
..
kInStreams.foreach( k =
{
k.foreachRDD ( rdd = inRecords += rdd.count().toInt )
inRecords.value
Question
