Hi Areg, Check out http://spark.apache.org/docs/latest/programming-guide.html#accumulators
val sum = sc.accumulator(0) // accumulator created from an initial value in the driver The accumulator variable is created in the driver. Tasks running on the cluster can then add to it. However, they cannot read its value. Only the driver program can read the accumulator’s value, using its value method. sum.value // in the driver > myRdd.map(x => sum += x) > where is this function being run This is being run by the tasks in the workers. The driver accumulates the data from the various workers and mergers them to get the final result as Haripriya mentioned. Thanks, Jayant On Fri, Oct 10, 2014 at 7:46 AM, HARIPRIYA AYYALASOMAYAJULA < aharipriy...@gmail.com> wrote: > If you use parallelize, the data is distributed across multiple nodes > available and sum is computed individually within each partition and later > merged. The driver manages the entire process. Is my understanding correct? > Can someone please correct me if I am wrong? > > On Fri, Oct 10, 2014 at 9:37 AM, Areg Baghdasaryan (BLOOMBERG/ 731 LEX -) > <abaghdasa...@bloomberg.net> wrote: > >> Hello, >> I was wondering on what does the Spark accumulator do under the covers. >> I’ve implemented my own associative addInPlace function for the >> accumulator, where is this function being run? Let’s say you call something >> like myRdd.map(x => sum += x) is “sum” being accumulated locally in any >> way, for each element or partition or node? Is “sum” a broadcast variable? >> Or does it only exist on the driver node? How does the driver node get >> access to the “sum”? >> Thanks, >> Areg >> > > > > -- > Regards, > Haripriya Ayyalasomayajula >
