Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()

Hi, in case the key value store is large then can you give redis a try? SPARK does work quite well with redis. Regards, Gourav Sengupta On Sat, Apr 7, 2018 at 6:29 PM, klrmowse wrote: > okie, well... > > i'm working with a pair rdd > > i need to

Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()

You can just call rdd.flatMap(_._2).collect Get Outlook for Android<https://aka.ms/ghei36> From: klrmowse <klrmo...@gmail.com> Sent: Saturday, April 7, 2018 1:29:34 PM To: user@spark.apache.org Subject: Re: [Spark 2.x Core] Adding to ArrayList inside

Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()

okie, well... i'm working with a pair rdd i need to extract the values and store them somehow (maybe a simple Array??), which i later parallelize and reuse since adding to a list is a no-no, what, if any, are the other options? (Java Spark, btw) thanks -- Sent

Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()

What are you trying to achieve ? You should not use global variables in a spark application. Especially not adding to a list - that makes in most cases no sense. If you want to put everything into a file then you should repartition to 1 . > On 7. Apr 2018, at 19:07, klrmowse

[Spark 2.x Core] Adding to ArrayList inside rdd.foreach()

it gives null pointer exception... is there a workaround for adding to an arrayList during .foreach of an rdd? thank you -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: