I'm not sure exactly what you're trying to do, but take a look at rdd.toLocalIterator if you haven't already.
On Tue, Dec 30, 2014 at 6:16 AM, Sean Owen <[email protected]> wrote: > collect()-ing a partition still implies copying it to the driver, but > you're suggesting you can't collect() the whole data set to the > driver. What do you mean: collect() 1 partition? or collect() some > smaller result from each partition? > > On Tue, Dec 30, 2014 at 11:54 AM, DEVAN M.S. <[email protected]> wrote: > > Hi all, > > i have one large data-set. when i am getting the number of partitions its > > showing 43. > > We can't collect() the large data-set in to memory so i am thinking like > > this, collect() each partitions so that it will be small in size. > > > > Any thoughts ? > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
