Re: Batch of updates

I don't think accumulators come into play here. Use foreachPartition, not mapPartitions. On Wed, Oct 29, 2014 at 12:43 AM, Flavio Pompermaier wrote: > Sorry but I wasn't able to code my stuff using accumulators as you suggested > :( > In my use case I have to to add elements to an array/list and

Re: Batch of updates

Sorry but I wasn't able to code my stuff using accumulators as you suggested :( In my use case I have to to add elements to an array/list and then, every 100 element commit the batch to a solr index and then clear it. In the cleanup code I have to commit the uncommited (remainder) elements. In the

Re: Batch of updates

to a spark one but I have some > doubts.. > My application basically buffers a batch of updates and every 100 elements > it flushes the batch to a server. This is very easy in mapreduce but I don't > know how you can do that in scala.. > > For example, if I do: > > myRdd.

Re: Batch of updates

Hi Flavio, Doing batch += ... shouldn't work. It will create new batch for each element in the myRDD (also val initializes an immutable variable, var is for mutable variables). You can use something like accumulators . val a

Batch of updates

Hi to all, I'm trying to convert my old mapreduce job to a spark one but I have some doubts.. My application basically buffers a batch of updates and every 100 elements it flushes the batch to a server. This is very easy in mapreduce but I don't know how you can do that in scala.. For e