EDIT: not mapper but a task for HadoopRDD maybe as far as I know. I think the most clear way is just to run a job on multiple files with the API and check the number of tasks in the job. On 27 Apr 2016 12:06 a.m., "Hyukjin Kwon" <gurwls...@gmail.com> wrote:
wholeTextFile() API uses WholeTextFileInputFormat, https://github.com/apache/spark/blob/d6dc12ef0146ae409834c78737c116050961f350/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala, which returns false for isSplittable. In this case, only single mapper appears for the entire file as far as I know. And also https://spark.apache.org/docs/1.6.0/programming-guide.html If the file is single file, then this would not be distributed. On 26 Apr 2016 11:52 p.m., "Ted Yu" <yuzhih...@gmail.com> wrote: > Please take a look at: > core/src/main/scala/org/apache/spark/SparkContext.scala > > * Do `val rdd = sparkContext.wholeTextFile("hdfs://a-hdfs-path")`, > * > * <p> then `rdd` contains > * {{{ > * (a-hdfs-path/part-00000, its content) > * (a-hdfs-path/part-00001, its content) > * ... > * (a-hdfs-path/part-nnnnn, its content) > * }}} > ... > * @param minPartitions A suggestion value of the minimal splitting > number for input data. > > def wholeTextFiles( > path: String, > minPartitions: Int = defaultMinPartitions): RDD[(String, String)] = > withScope { > > On Tue, Apr 26, 2016 at 7:43 AM, Vadim Vararu <vadim.var...@adswizz.com> > wrote: > >> Hi guys, >> >> I'm trying to read many filed from s3 using >> JavaSparkContext.wholeTextFiles(...). Is that executed in a distributed >> manner? Please give me a link to the place in documentation where it's >> specified. >> >> Thanks, Vadim. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
