I am trying to implement counters in Spark and I guess Accumulators are the way to do it.
My motive is to update a counter in map function and access/reset it in the driver code. However the /println/ statement at the end still yields value 0(It should 9). Am I doing something wrong? def main(args : Array[String]){ val conf = new SparkConf().setAppName("SortedNeighbourhoodMatching") val sc = new SparkContext(conf) var counter = sc.accumulable(0, "Counter") var inputFilePath = args(0) val inputRDD = sc.textFile(inputFilePath) inputRDD.map { x => { counter += 1 } } println(counter.value) } -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Counters-in-Spark-tp21646.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org