[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636399#comment-13636399 ] Hudson commented on MAPREDUCE-4898: --- Integrated in Hadoop-Mapreduce-trunk #1404 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1404/]) MAPREDUCE-4898. FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1. (rkanter via tucu) (Revision 1469523) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469523 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputFormat.java > FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath > incompatible with MR1 > -- > > Key: MAPREDUCE-4898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-4898.patch > > > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}} > throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it > throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making > them incompatible. > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}} > doesn't throw any exceptions but now it throws an {{IOException}}, making > them incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636342#comment-13636342 ] Hudson commented on MAPREDUCE-4898: --- Integrated in Hadoop-Hdfs-trunk #1377 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1377/]) MAPREDUCE-4898. FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1. (rkanter via tucu) (Revision 1469523) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469523 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputFormat.java > FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath > incompatible with MR1 > -- > > Key: MAPREDUCE-4898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-4898.patch > > > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}} > throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it > throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making > them incompatible. > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}} > doesn't throw any exceptions but now it throws an {{IOException}}, making > them incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636262#comment-13636262 ] Hudson commented on MAPREDUCE-4898: --- Integrated in Hadoop-Yarn-trunk #188 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/188/]) MAPREDUCE-4898. FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1. (rkanter via tucu) (Revision 1469523) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469523 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputFormat.java > FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath > incompatible with MR1 > -- > > Key: MAPREDUCE-4898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-4898.patch > > > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}} > throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it > throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making > them incompatible. > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}} > doesn't throw any exceptions but now it throws an {{IOException}}, making > them incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635540#comment-13635540 ] Hudson commented on MAPREDUCE-4898: --- Integrated in Hadoop-trunk-Commit #3629 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3629/]) MAPREDUCE-4898. FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1. (rkanter via tucu) (Revision 1469523) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469523 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputFormat.java > FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath > incompatible with MR1 > -- > > Key: MAPREDUCE-4898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-4898.patch > > > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}} > throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it > throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making > them incompatible. > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}} > doesn't throw any exceptions but now it throws an {{IOException}}, making > them incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635517#comment-13635517 ] Alejandro Abdelnur commented on MAPREDUCE-4898: --- Thanks for verifying this Robert. +1 > FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath > incompatible with MR1 > -- > > Key: MAPREDUCE-4898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-4898.patch > > > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}} > throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it > throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making > them incompatible. > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}} > doesn't throw any exceptions but now it throws an {{IOException}}, making > them incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635338#comment-13635338 ] Robert Kanter commented on MAPREDUCE-4898: -- I created a job like you said and ran it on Hadoop 1 and Hadoop 2 with and without setting the working dir. The behavior is the same in both Hadoop 1 and Hadoop 2: without setting the working dir, it resolves a relative path to the homedir; with setting the working dir, it resolves a relative path to that dir (even in Hadoop 1). So no behavioral change here :) > FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath > incompatible with MR1 > -- > > Key: MAPREDUCE-4898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-4898.patch > > > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}} > throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it > throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making > them incompatible. > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}} > doesn't throw any exceptions but now it throws an {{IOException}}, making > them incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621395#comment-13621395 ] Alejandro Abdelnur commented on MAPREDUCE-4898: --- While the patch looks good, we may have introduced a behavioral change in the {{setOutputPath()}} method. In Hadoop 1, {{setOutputPath()}} sets whatever path it is received in the conf, without any modification. In Hadoop 2, {{setOutputPath()}} makeQualified-s the received path and then sets it in the conf. This means that in Hadoop 1 a relative path will always resolved to the homedir while in Hadoop 2 a relative path will be resolved to the working dir (by default set to the homedir). Can we please verify this hypothesis? This could be done by having a simple MR job where in the submission we set (or not) the workingdirectory in the FS, and use a relative path for the output dir. Then run the job once setting workingdir to /tmp, once not setting it at all. Run this in Hadoop 1 and in Hadoop 2. Find out where the job output ends in all cases. If different, we need to decide if we want to revert to Hadoop 1 behavior or not. > FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath > incompatible with MR1 > -- > > Key: MAPREDUCE-4898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-4898.patch > > > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}} > throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it > throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making > them incompatible. > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}} > doesn't throw any exceptions but now it throws an {{IOException}}, making > them incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538414#comment-13538414 ] Karthik Kambatla commented on MAPREDUCE-4898: - Thanks for taking a closer look, Robert. +1 > FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath > incompatible with MR1 > -- > > Key: MAPREDUCE-4898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.0.3-alpha > > Attachments: MAPREDUCE-4898.patch > > > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}} > throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it > throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making > them incompatible. > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}} > doesn't throw any exceptions but now it throws an {{IOException}}, making > them incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538403#comment-13538403 ] Robert Kanter commented on MAPREDUCE-4898: -- I just ran all of the tests under {{hadoop-mapreduce-project}} successfully. I also just checked where those two methods are used and there's no issues because: 1) Whenever {{checkOutputSpecs}} is used, the code handles an {{IOException}} instead of either of the two types of {{FileAlreadyExistsException}}, and both types of {{FileAlreadyExistsException}} inherit from {{IOException}} 2) Whenever {{setOutputPath}} is used, there's always other method calls that can throw {{IOException}}s anyway > FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath > incompatible with MR1 > -- > > Key: MAPREDUCE-4898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.0.3-alpha > > Attachments: MAPREDUCE-4898.patch > > > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}} > throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it > throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making > them incompatible. > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}} > doesn't throw any exceptions but now it throws an {{IOException}}, making > them incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537639#comment-13537639 ] Karthik Kambatla commented on MAPREDUCE-4898: - Thanks Robert. The patch looks good, the tests in particular. I am surprised these changes didn't result in other changes due to calls to the methods whose signature the patch modifies. Given Jenkins doesn't complain, it should be okay. Do you want to double-check just in case. > FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath > incompatible with MR1 > -- > > Key: MAPREDUCE-4898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.0.3-alpha > > Attachments: MAPREDUCE-4898.patch > > > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}} > throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it > throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making > them incompatible. > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}} > doesn't throw any exceptions but now it throws an {{IOException}}, making > them incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537638#comment-13537638 ] Hadoop QA commented on MAPREDUCE-4898: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562027/MAPREDUCE-4898.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3154//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3154//console This message is automatically generated. > FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath > incompatible with MR1 > -- > > Key: MAPREDUCE-4898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.0.3-alpha > > Attachments: MAPREDUCE-4898.patch > > > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}} > throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it > throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making > them incompatible. > In MR1, > {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}} > doesn't throw any exceptions but now it throws an {{IOException}}, making > them incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira