subject:"RDD from partitions"

Re: RDD from partitions

2015-08-28 Thread Rishitesh Mishra

Hi Jem, A simple way to get this is to use MapPartitionedRDD. Please see the below code. For this you need to know your parent RDD's partition numbers that you want to exclude. One drawback here is the new RDD will also invoke similar number of tasks as parent RDDs as both the RDDs have same

Re: RDD from partitions

2015-08-28 Thread Jem Tucker

Hey Rishitesh, Thats perfect thanks so much! Dont know why i didnt think of using mapPartitions like this Thanks, Jem On Fri, Aug 28, 2015 at 10:35 AM Rishitesh Mishra rishi80.mis...@gmail.com wrote: Hi Jem, A simple way to get this is to use MapPartitionedRDD. Please see the below code.