It does not necessarily shuffle, yes. I believe it will not if you are strictly reducing the number of partitions, and do not force a shuffle. So I think the answer is 'yes'.
If you have a huge number of small files, you can also consider wholeTextFiles, which gives you entire files of content in each element of the RDD. It is not necessarily helpful, but thought I'd mention it, as it could be of interest depending on what you do. On Fri, Jan 23, 2015 at 2:14 AM, Kane Kim <kane.ist...@gmail.com> wrote: > Does it avoid reshuffling? I have 300 thousands output files. If I > coalesce to the number of cores in the cluster would it keep data > local? (I have 100 nodes, 4 cores each, does it mean if I > coalesce(400) it will use all cores and data will stay local)? > > On Thu, Jan 22, 2015 at 3:26 PM, Sean Owen <so...@cloudera.com> wrote: >> One output file is produced per partition. If you want fewer, use >> coalesce() before saving the RDD. >> >> On Thu, Jan 22, 2015 at 10:46 PM, Kane Kim <kane.ist...@gmail.com> wrote: >>> How I can reduce number of output files? Is there a parameter to >>> saveAsTextFile? >>> >>> Thanks. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
