Re: How to add an accumulator for a Set in Spark

Btw, here is a great article about accumulators and all their related traps! http://imranrashid.com/posts/Spark-Accumulators/ (I'm not the author) On 16 March 2016 at 18:24, swetha kasireddy wrote: > OK. I did take a look at them. So once I have an accumulater for a >

Re: How to add an accumulator for a Set in Spark

OK. I did take a look at them. So once I have an accumulater for a HashSet, how can I check if a particular key is already present in the HashSet accumulator? I don't see any .contains method there. My requirement is that I need to keep accumulating the keys in the HashSet across all the tasks in

Re: How to add an accumulator for a Set in Spark

Have you looked at using Accumulable interface, Take a look at Spark documentation at http://spark.apache.org/docs/latest/programming-guide.html#accumulators it gives example of how to use vector type for accumalator, which might be very close to what you need -- View this message in context:

Re: How to add an accumulator for a Set in Spark

Please take a look at: core/src/test/scala/org/apache/spark/AccumulatorSuite.scala FYI On Tue, Mar 15, 2016 at 4:29 PM, SRK wrote: > Hi, > > How do I add an accumulator for a Set in Spark? > > Thanks! > > > > -- > View this message in context: >

How to add an accumulator for a Set in Spark

Hi, How do I add an accumulator for a Set in Spark? Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-add-an-accumulator-for-a-Set-in-Spark-tp26510.html Sent from the Apache Spark User List mailing list archive at Nabble.com.