collect()-ing a partition still implies copying it to the driver, but you're suggesting you can't collect() the whole data set to the driver. What do you mean: collect() 1 partition? or collect() some smaller result from each partition?
On Tue, Dec 30, 2014 at 11:54 AM, DEVAN M.S. <[email protected]> wrote: > Hi all, > i have one large data-set. when i am getting the number of partitions its > showing 43. > We can't collect() the large data-set in to memory so i am thinking like > this, collect() each partitions so that it will be small in size. > > Any thoughts ? > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
