Re: How to return a pair RDD from an RDD that has foreachPartition applied?

Looks like I can use mapPartitions but can it be done using forEachPartition? On Tue, Nov 17, 2015 at 11:51 PM, swetha wrote: > Hi, > > How to return an RDD of key/value pairs from an RDD that has > foreachPartition applied. I have my code something like the

Re: How to return a pair RDD from an RDD that has foreachPartition applied?

I think you can use mapPartitions that returns PairRDDs followed by forEachPartition for saving it On Wed, Nov 18, 2015 at 9:31 AM swetha kasireddy wrote: > Looks like I can use mapPartitions but can it be done using > forEachPartition? > > On Tue, Nov 17, 2015 at

How to return a pair RDD from an RDD that has foreachPartition applied?

Hi, How to return an RDD of key/value pairs from an RDD that has foreachPartition applied. I have my code something like the following. It looks like an RDD that has foreachPartition can have only the return type as Unit. How do I apply foreachPartition and do a save and at the same return a pair