Re: How to use collections inside foreach block

Hi Rishi and Ted, Thank you for the response. Now I'm using Accumulators and getting results. I have a another query, how to start parallel the code. Example :- var listOfIds is a ListBuffer with 2 records I'm creating batches. For each batch size is 500. It means, total batches are : 40.

Re: How to use collections inside foreach block

Your list is defined on the driver, whereas function specified in forEach will be evaluated on each executor. You might want to add an accumulator or handle a Sequence of list from each partition. On Wed, Dec 9, 2015 at 11:19 AM, Madabhattula Rajesh Kumar < mrajaf...@gmail.com> wrote: > Hi, > >

Re: How to use collections inside foreach block

To add onto what Rishi said, you can use foreachPartition() on result where you can save values to DB. Cheers On Wed, Dec 9, 2015 at 12:51 AM, Rishi Mishra wrote: > Your list is defined on the driver, whereas function specified in forEach > will be evaluated on each

How to use collections inside foreach block

Hi, I have a below query. Please help me to solve this I have a 2 ids. I want to join these ids to table. This table contains some blob data. So i can not join these 2000 ids to this table in one step. I'm planning to join this table in a chunks. For example, each step I will join 5000 ids.