collect()-ing a partition still implies copying it to the driver, but
you're suggesting you can't collect() the whole data set to the
driver. What do you mean: collect() 1 partition? or collect() some
smaller result from each partition?

On Tue, Dec 30, 2014 at 11:54 AM, DEVAN M.S. <[email protected]> wrote:
> Hi all,
> i have one large data-set. when i am getting the number of partitions its
> showing 43.
> We can't collect() the large data-set in to  memory so i am thinking like
> this, collect() each partitions so that it will be small in size.
>
> Any thoughts ?
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to