[ https://issues.apache.org/jira/browse/SPARK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14513194#comment-14513194 ]
Apache Spark commented on SPARK-7155: ------------------------------------- User 'yongtang' has created a pull request for this issue: https://github.com/apache/spark/pull/5708 > SparkContext's newAPIHadoopFile does not support comma-separated list of > files, but the other API hadoopFile does. > ------------------------------------------------------------------------------------------------------------------ > > Key: SPARK-7155 > URL: https://issues.apache.org/jira/browse/SPARK-7155 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.3.1 > Environment: Ubuntu 14.04 > Reporter: Yong Tang > > SparkContext's newAPIHadoopFile() does not support comma-separated list of > files. For example, the following: > sc.newAPIHadoopFile("/root/file1.txt,/root/file2.txt", > classOf[TextInputFormat], classOf[LongWritable], classOf[Text]) > will throw > org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does > not exist: file:/root/file1.txt,/root/file2.txt > However, the other API hadoopFile() is able to process comma-separated list > of files correctly. > In addition, since sc.textFile() uses hadoopFile(), it is also able to > process comma-separated list of files correctly. > The problem is that newAPIHadoopFile() use addInputPath() to add the file > path into NewHadoopRDD. See Ln 928-931, master branch: > val job = new NewHadoopJob(conf) > NewFileInputFormat.addInputPath(job, new Path(path)) > val updatedConf = job.getConfiguration > new NewHadoopRDD(this, fClass, kClass, vClass, updatedConf).setName(path) > Change addInputPath(job, new Path(path)) to addInputPaths(job, path) will > resolve this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org