Re: Running foreach on a list of rdds in parallel

2015-07-16 Thread Davies Liu 
sc.union(rdds).saveAsTextFile()

On Wed, Jul 15, 2015 at 10:37 PM, Brandon White bwwintheho...@gmail.com wrote:
 Hello,

 I have a list of rdds

 List(rdd1, rdd2, rdd3,rdd4)

 I would like to save these rdds in parallel. Right now, it is running each
 operation sequentially. I tried using a rdd of rdd but that does not work.

 list.foreach { rdd =
   rdd.saveAsTextFile(/tmp/cache/)
 }

 Any ideas?

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Running foreach on a list of rdds in parallel

2015-07-15 Thread Brandon White 
Hello,

I have a list of rdds

List(rdd1, rdd2, rdd3,rdd4)

I would like to save these rdds in parallel. Right now, it is running each
operation sequentially. I tried using a rdd of rdd but that does not work.

list.foreach { rdd =
  rdd.saveAsTextFile(/tmp/cache/)
}

Any ideas?