[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999008#comment-13999008 ] Hudson commented on MAPREDUCE-5814: --- SUCCESS: Integrated in Hadoop-trunk-Commit #5605 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5605/]) MAPREDUCE-5814. fat jar with *-default.xml may fail when mapreduce.job.classloader=true. Contributed by Gera Shegalov (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594326) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ApplicationClassLoader.java > fat jar with *-default.xml may fail when mapreduce.job.classloader=true. > > > Key: MAPREDUCE-5814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Fix For: 3.0.0, 2.5.0 > > Attachments: MAPREDUCE-5814.v01.patch, MAPREDUCE-5814.v02.patch, > MAPREDUCE-5814.v03.patch > > > We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had > to run with {{mapreduce.job.classloader=true}} because it needed a more > recent guava as a dependency. The job failed because the cluster's > {{*-default.xml}} files were overshadowed by the ones in the fat jar. We > propose to treat these default config files like the system packages > {{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had > {{mapreduce.job.user.classpath.first}} set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998178#comment-13998178 ] Hudson commented on MAPREDUCE-5814: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1779 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1779/]) MAPREDUCE-5814. fat jar with *-default.xml may fail when mapreduce.job.classloader=true. Contributed by Gera Shegalov (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594326) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ApplicationClassLoader.java > fat jar with *-default.xml may fail when mapreduce.job.classloader=true. > > > Key: MAPREDUCE-5814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Fix For: 3.0.0, 2.5.0 > > Attachments: MAPREDUCE-5814.v01.patch, MAPREDUCE-5814.v02.patch, > MAPREDUCE-5814.v03.patch > > > We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had > to run with {{mapreduce.job.classloader=true}} because it needed a more > recent guava as a dependency. The job failed because the cluster's > {{*-default.xml}} files were overshadowed by the ones in the fat jar. We > propose to treat these default config files like the system packages > {{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had > {{mapreduce.job.user.classpath.first}} set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998216#comment-13998216 ] Hudson commented on MAPREDUCE-5814: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #1753 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1753/]) MAPREDUCE-5814. fat jar with *-default.xml may fail when mapreduce.job.classloader=true. Contributed by Gera Shegalov (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594326) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ApplicationClassLoader.java > fat jar with *-default.xml may fail when mapreduce.job.classloader=true. > > > Key: MAPREDUCE-5814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Fix For: 3.0.0, 2.5.0 > > Attachments: MAPREDUCE-5814.v01.patch, MAPREDUCE-5814.v02.patch, > MAPREDUCE-5814.v03.patch > > > We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had > to run with {{mapreduce.job.classloader=true}} because it needed a more > recent guava as a dependency. The job failed because the cluster's > {{*-default.xml}} files were overshadowed by the ones in the fat jar. We > propose to treat these default config files like the system packages > {{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had > {{mapreduce.job.user.classpath.first}} set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996736#comment-13996736 ] Hadoop QA commented on MAPREDUCE-5814: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12644307/MAPREDUCE-5814.v03.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4600//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4600//console This message is automatically generated. > fat jar with *-default.xml may fail when mapreduce.job.classloader=true. > > > Key: MAPREDUCE-5814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: MAPREDUCE-5814.v01.patch, MAPREDUCE-5814.v02.patch, > MAPREDUCE-5814.v03.patch > > > We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had > to run with {{mapreduce.job.classloader=true}} because it needed a more > recent guava as a dependency. The job failed because the cluster's > {{*-default.xml}} files were overshadowed by the ones in the fat jar. We > propose to treat these default config files like the system packages > {{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had > {{mapreduce.job.user.classpath.first}} set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996769#comment-13996769 ] Jason Lowe commented on MAPREDUCE-5814: --- +1 lgtm. Committing this. > fat jar with *-default.xml may fail when mapreduce.job.classloader=true. > > > Key: MAPREDUCE-5814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: MAPREDUCE-5814.v01.patch, MAPREDUCE-5814.v02.patch, > MAPREDUCE-5814.v03.patch > > > We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had > to run with {{mapreduce.job.classloader=true}} because it needed a more > recent guava as a dependency. The job failed because the cluster's > {{*-default.xml}} files were overshadowed by the ones in the fat jar. We > propose to treat these default config files like the system packages > {{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had > {{mapreduce.job.user.classpath.first}} set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990904#comment-13990904 ] Jason Lowe commented on MAPREDUCE-5814: --- The explicit listing of the various default.xml files seems like a reasonable approach. Has there been any manual testing of the patch? Since it reformats the system classes for readability it looks like it will not work for org.apache.hadoop. and mapred-default.xml. Those will now have a newline character and multiple spaces prepended to their names and therefore won't ever match. We either need to remove whitespace from the value or change MRApps#setJobClassLoader to use conf.getTrimmedStrings instead of conf.getStrings. > fat jar with *-default.xml may fail when mapreduce.job.classloader=true. > > > Key: MAPREDUCE-5814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: MAPREDUCE-5814.v01.patch > > > We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had > to run with {{mapreduce.job.classloader=true}} because it needed a more > recent guava as a dependency. The job failed because the cluster's > {{*-default.xml}} files were overshadowed by the ones in the fat jar. We > propose to treat these default config files like the system packages > {{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had > {{mapreduce.job.user.classpath.first}} set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990105#comment-13990105 ] Sangjin Lee commented on MAPREDUCE-5814: Ping: this could use some love. :) Could someone look at this, and commit the patch if you think the patch is good to go? Thanks in advance. > fat jar with *-default.xml may fail when mapreduce.job.classloader=true. > > > Key: MAPREDUCE-5814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: MAPREDUCE-5814.v01.patch > > > We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had > to run with {{mapreduce.job.classloader=true}} because it needed a more > recent guava as a dependency. The job failed because the cluster's > {{*-default.xml}} files were overshadowed by the ones in the fat jar. We > propose to treat these default config files like the system packages > {{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had > {{mapreduce.job.user.classpath.first}} set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13955582#comment-13955582 ] Sangjin Lee commented on MAPREDUCE-5814: +1 LGTM > fat jar with *-default.xml may fail when mapreduce.job.classloader=true. > > > Key: MAPREDUCE-5814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: MAPREDUCE-5814.v01.patch > > > We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had > to run with {{mapreduce.job.classloader=true}} because it needed a more > recent guava as a dependency. The job failed because the cluster's > {{*-default.xml}} files were overshadowed by the ones in the fat jar. We > propose to treat these default config files like the system packages > {{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had > {{mapreduce.job.user.classpath.first}} set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950331#comment-13950331 ] Sangjin Lee commented on MAPREDUCE-5814: Sounds good. I agree users adding *-site.xml files in their jar is such an edge case that I don't think we need to be defensive about that type of scenarios as well. The *-default.xml files can definitely get in via build, but not *-site.xml... > fat jar with *-default.xml may fail when mapreduce.job.classloader=true. > > > Key: MAPREDUCE-5814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: MAPREDUCE-5814.v01.patch > > > We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had > to run with {{mapreduce.job.classloader=true}} because it needed a more > recent guava as a dependency. The job failed because the cluster's > {{*-default.xml}} files were overshadowed by the ones in the fat jar. We > propose to treat these default config files like the system packages > {{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had > {{mapreduce.job.user.classpath.first}} set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13949774#comment-13949774 ] Gera Shegalov commented on MAPREDUCE-5814: -- Sangjin, I was thinking of this... but *-site.xml unlike job.xml are not meant to be shipped, they are meant to be installed. default.xml is packaged with hadoop, and can therefore make it into a fat jar. It's a good idea as a defensive measure. The only reason that prevented me from doing this is that the list of such files can be quite long, and include all kinds of log4j and other properties. But they can be legally shipped too. So far I have settled for the least requirement. Looking forward to the pros and cons. > fat jar with *-default.xml may fail when mapreduce.job.classloader=true. > > > Key: MAPREDUCE-5814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: MAPREDUCE-5814.v01.patch > > > We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had > to run with {{mapreduce.job.classloader=true}} because it needed a more > recent guava as a dependency. The job failed because the cluster's > {{*-default.xml}} files were overshadowed by the ones in the fat jar. We > propose to treat these default config files like the system packages > {{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had > {{mapreduce.job.user.classpath.first}} set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13949751#comment-13949751 ] Sangjin Lee commented on MAPREDUCE-5814: How about *-site.xml files? Do we support clients sending their site.xml files as part of app submission and overriding the cluster's site.xml files? > fat jar with *-default.xml may fail when mapreduce.job.classloader=true. > > > Key: MAPREDUCE-5814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: MAPREDUCE-5814.v01.patch > > > We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had > to run with {{mapreduce.job.classloader=true}} because it needed a more > recent guava as a dependency. The job failed because the cluster's > {{*-default.xml}} files were overshadowed by the ones in the fat jar. We > propose to treat these default config files like the system packages > {{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had > {{mapreduce.job.user.classpath.first}} set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13949280#comment-13949280 ] Hadoop QA commented on MAPREDUCE-5814: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637135/MAPREDUCE-5814.v01.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4463//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4463//console This message is automatically generated. > fat jar with *-default.xml may fail when mapreduce.job.classloader=true. > > > Key: MAPREDUCE-5814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: MAPREDUCE-5814.v01.patch > > > We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had > to run with {{mapreduce.job.classloader=true}} because it needed a more > recent guava as a dependency. The job failed because the cluster's > {{*-default.xml}} files were overshadowed by the ones in the fat jar. We > propose to treat these default config files like the system packages > {{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had > {{mapreduce.job.user.classpath.first}} set. -- This message was sent by Atlassian JIRA (v6.2#6252)