this is more-or-less the best you can do now, but as has been pointed out, accumulators don't quite fit the bill for counters. There is an open issue to do something better, but no progress on that so far
https://issues.apache.org/jira/browse/SPARK-603 On Fri, Feb 13, 2015 at 11:12 AM, Mark Hamstra <m...@clearstorydata.com> wrote: > Except that transformations don't have an exactly-once guarantee, so this > way of doing counters may produce different answers across various forms of > failures and speculative execution. > > On Fri, Feb 13, 2015 at 8:56 AM, Sean McNamara < > sean.mcnam...@webtrends.com> wrote: > >> .map is just a transformation, so no work will actually be performed >> until something takes action against it. Try adding a .count(), like so: >> >> inputRDD.map { x => { >> counter += 1 >> } }.count() >> >> In case it is helpful, here are the docs on what exactly the >> transformations and actions are: >> http://spark.apache.org/docs/1.2.0/programming-guide.html#transformations >> http://spark.apache.org/docs/1.2.0/programming-guide.html#actions >> >> Cheers, >> >> Sean >> >> >> On Feb 13, 2015, at 9:50 AM, nitinkak001 <nitinkak...@gmail.com> wrote: >> >> I am trying to implement counters in Spark and I guess Accumulators are >> the >> way to do it. >> >> My motive is to update a counter in map function and access/reset it in >> the >> driver code. However the /println/ statement at the end still yields value >> 0(It should 9). Am I doing something wrong? >> >> def main(args : Array[String]){ >> >> val conf = new SparkConf().setAppName("SortedNeighbourhoodMatching") >> val sc = new SparkContext(conf) >> var counter = sc.accumulable(0, "Counter") >> var inputFilePath = args(0) >> val inputRDD = sc.textFile(inputFilePath) >> >> inputRDD.map { x => { >> counter += 1 >> } } >> println(counter.value) >> } >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Counters-in-Spark-tp21646.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >> >
