[ https://issues.apache.org/jira/browse/MAPREDUCE-5912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14030522#comment-14030522 ]
Hudson commented on MAPREDUCE-5912: ----------------------------------- FAILURE: Integrated in Hadoop-Yarn-trunk #582 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/582/]) MAPREDUCE-5912. Task.calculateOutputSize does not handle Windows files after MAPREDUCE-5196. Contributed by Remus Rusanu. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1602282) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java > Task.calculateOutputSize does not handle Windows files after MAPREDUCE-5196 > --------------------------------------------------------------------------- > > Key: MAPREDUCE-5912 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5912 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client > Affects Versions: 3.0.0 > Reporter: Remus Rusanu > Assignee: Remus Rusanu > Fix For: 3.0.0 > > Attachments: MAPREDUCE-5912.1.patch > > > {code} > @@ -1098,8 +1120,8 @@ private long calculateOutputSize() throws IOException { > if (isMapTask() && conf.getNumReduceTasks() > 0) { > try { > Path mapOutput = mapOutputFile.getOutputFile(); > - FileSystem localFS = FileSystem.getLocal(conf); > - return localFS.getFileStatus(mapOutput).getLen(); > + FileSystem fs = mapOutput.getFileSystem(conf); > + return fs.getFileStatus(mapOutput).getLen(); > } catch (IOException e) { > LOG.warn ("Could not find output size " , e); > } > {code} > causes Windows local output files to be routed through HDFS: > {code} > 2014-06-02 00:14:53,891 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.IllegalArgumentException: Pathname > /c:/Hadoop/Data/Hadoop/local/usercache/HadoopUser/appcache/application_1401693085139_0001/output/attempt_1401693085139_0001_m_000000_0/file.out > from > c:/Hadoop/Data/Hadoop/local/usercache/HadoopUser/appcache/application_1401693085139_0001/output/attempt_1401693085139_0001_m_000000_0/file.out > is not a valid DFS filename. > at > org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:187) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:101) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1024) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1020) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1020) > at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1124) > at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1102) > at org.apache.hadoop.mapred.Task.done(Task.java:1048) > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)