[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563149#comment-14563149 ] Hudson commented on YARN-3626: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2157 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2157/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.15.patch, YARN-3626.16.patch, > YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563081#comment-14563081 ] Hudson commented on YARN-3626: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #209 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/209/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.15.patch, YARN-3626.16.patch, > YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562994#comment-14562994 ] Hudson commented on YARN-3626: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #199 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/199/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.15.patch, YARN-3626.16.patch, > YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562971#comment-14562971 ] Hudson commented on YARN-3626: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2139 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2139/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.15.patch, YARN-3626.16.patch, > YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562757#comment-14562757 ] Hudson commented on YARN-3626: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #941 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/941/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.15.patch, YARN-3626.16.patch, > YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562674#comment-14562674 ] Hudson commented on YARN-3626: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #211 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/211/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.15.patch, YARN-3626.16.patch, > YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561790#comment-14561790 ] Hudson commented on YARN-3626: -- FAILURE: Integrated in Hadoop-trunk-Commit #7909 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7909/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.15.patch, YARN-3626.16.patch, > YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560210#comment-14560210 ] Craig Welch commented on YARN-3626: --- The checkstyle is insignificant, the rest is all good. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.15.patch, YARN-3626.16.patch, > YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560092#comment-14560092 ] Vinod Kumar Vavilapalli commented on YARN-3626: --- bq. Sure, annotated also Tx, that is what I meant.. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.15.patch, YARN-3626.16.patch, > YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560080#comment-14560080 ] Hadoop QA commented on YARN-3626: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 15m 17s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 44s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 10m 1s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 2m 34s | The applied patch generated 1 new checkstyle issues (total was 241, now 238). | | {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that end in whitespace. | | {color:green}+1{color} | install | 1m 36s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 4m 23s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | mapreduce tests | 9m 59s | Tests passed in hadoop-mapreduce-client-app. | | {color:green}+1{color} | mapreduce tests | 0m 45s | Tests passed in hadoop-mapreduce-client-common. | | {color:green}+1{color} | yarn tests | 0m 23s | Tests passed in hadoop-yarn-api. | | {color:green}+1{color} | yarn tests | 6m 10s | Tests passed in hadoop-yarn-server-nodemanager. | | | | 60m 8s | | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12735424/YARN-3626.16.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / 7dba700 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt | | hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt | | hadoop-mapreduce-client-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt | | hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/testrun_hadoop-yarn-api.txt | | hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8090/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8090/console | This message was automatically generated. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.15.patch, YARN-3626.16.patch, > YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559939#comment-14559939 ] Chris Nauroth commented on YARN-3626: - Thanks, Craig! We could potentially stick the {{@Private}} annotation directly onto {{ApplicationConstants#CLASSPATH_PREPEND_DISTCACHE}}. I'll let Vinod chime in on whether or not this was the intent of his feedback. +1 from me, pending Jenkins run. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.15.patch, YARN-3626.4.patch, YARN-3626.6.patch, > YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559935#comment-14559935 ] Craig Welch commented on YARN-3626: --- [~vinodkv] indicated it's private & Windows specific, all for doing something better overall long term aka [YARN-3685] [~cnauroth] switched to simpler valueOf > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.15.patch, YARN-3626.4.patch, YARN-3626.6.patch, > YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558495#comment-14558495 ] Chris Nauroth commented on YARN-3626: - Hi Craig. This looks good to me. I have just one minor nitpick. I think the logic in {{ContainerLaunch}} for setting {{preferLocalizedJars}} could be simplified to this: {code} boolean preferLocalizedJars = Boolean.valueOf(classpathPrependDistCache); {code} {{Boolean#valueOf}} is null-safe. Thanks! > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14552686#comment-14552686 ] Craig Welch commented on YARN-3626: --- Checkstyle looks insignificant. [~cnauroth], [~vinodkv], I've changed the approach to use the environment instead of configuration as suggested, can one of you review pls? > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14551790#comment-14551790 ] Hadoop QA commented on YARN-3626: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 14m 42s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 33s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 2m 22s | The applied patch generated 1 new checkstyle issues (total was 240, now 238). | | {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that end in whitespace. | | {color:green}+1{color} | install | 1m 34s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 4m 21s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | mapreduce tests | 9m 26s | Tests passed in hadoop-mapreduce-client-app. | | {color:green}+1{color} | mapreduce tests | 0m 45s | Tests passed in hadoop-mapreduce-client-common. | | {color:green}+1{color} | yarn tests | 0m 30s | Tests passed in hadoop-yarn-api. | | {color:green}+1{color} | yarn tests | 6m 5s | Tests passed in hadoop-yarn-server-nodemanager. | | | | 58m 2s | | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12734054/YARN-3626.14.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / ce53c8e | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt | | hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt | | hadoop-mapreduce-client-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt | | hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/testrun_hadoop-yarn-api.txt | | hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8015/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8015/console | This message was automatically generated. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.14.patch, YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546527#comment-14546527 ] Hadoop QA commented on YARN-3626: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 14m 44s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 35s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 1m 51s | The applied patch generated 1 new checkstyle issues (total was 240, now 238). | | {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that end in whitespace. | | {color:green}+1{color} | install | 1m 36s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 3m 26s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. | | {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in hadoop-mapreduce-client-common. | | {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in hadoop-yarn-api. | | {color:green}+1{color} | yarn tests | 6m 1s | Tests passed in hadoop-yarn-server-nodemanager. | | | | 47m 15s | | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12733294/YARN-3626.11.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / ac742c7 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/7957/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt | | hadoop-mapreduce-client-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/7957/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt | | hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/7957/artifact/patchprocess/testrun_hadoop-yarn-api.txt | | hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7957/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7957/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7957/console | This message was automatically generated. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.11.patch, > YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546400#comment-14546400 ] Chris Nauroth commented on YARN-3626: - I see now. Thanks for the clarification. In that case, I agree with the new proposal. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch, > YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546393#comment-14546393 ] Vinod Kumar Vavilapalli commented on YARN-3626: --- bq. I thought this implemented "per app" behavior, because it could vary between MR app submission instances. It would not be a requirement to put YARN_APPLICATION_CLASSPATH_PREPEND_DISTCACHE into the server configs and have the client and server share configs. YARN doesn't have a notion of app-configs, it doesn't know app's config files etc. So, the app cannot set a config property that it expects the server to respect. No idea, how your original patch apparently worked. May be we are missing something. [~cwelch], what I was proposing was something in the lines of (a) user sets MR user-classpath-before config (b) MR converts that into a special env for YARN and (c) YARN looks at the ENV to figure out how to the order the classpath. Overall, it is terrible that we are talking classpaths in YARN, but that's for another JIRA. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch, > YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546374#comment-14546374 ] Craig Welch commented on YARN-3626: --- Right, going back to [~cnauroth], [~vinodkv], we chatted and you asserted that the original approach can't work, but it seemed to, it's not entirely clear to me why it shouldn't... > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch, > YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546370#comment-14546370 ] Craig Welch commented on YARN-3626: --- bq. Can't we do the above? We definitely cannot insert mapreduce incantations like job.jar into YARN. That's why I took the config based approach, which apparently is invalid... but it also worked, which is quite confusing. I'm going to go back and validate our reasoning for believing it shoudn't. bq. Can't we do the above? We definitely cannot insert mapreduce incantations like job.jar into YARN. I suppose we can if it would work. It needs to be something which can be propagated from Oozie, which adds additional complexity. Ideally, we need something the MrApps can set based on the presence of the mapred configuration so that it propagates through. Do we have an example of this being done elsewhere? > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch, > YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546365#comment-14546365 ] Chris Nauroth commented on YARN-3626: - I don't fully understand the objection to the former patch that had been committed. bq. The new configuration added is supposed to be per app, but it is now a server side configuration. There was a new YARN configuration property for triggering this behavior, but the MR application would toggle on that YARN property only if the MR job submission had {{MAPREDUCE_JOB_USER_CLASSPATH_FIRST}} on. From {{MRApps}}: {code} boolean userClassesTakesPrecedence = conf.getBoolean(MRJobConfig.MAPREDUCE_JOB_USER_CLASSPATH_FIRST, false); if (userClassesTakesPrecedence) { conf.set(YarnConfiguration.YARN_APPLICATION_CLASSPATH_PREPEND_DISTCACHE, "true"); } {code} I thought this implemented "per app" behavior, because it could vary between MR app submission instances. It would not be a requirement to put {{YARN_APPLICATION_CLASSPATH_PREPEND_DISTCACHE}} into the server configs and have the client and server share configs. Is there a detail I'm missing? > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch, > YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546359#comment-14546359 ] Vinod Kumar Vavilapalli commented on YARN-3626: --- bq. We may have to depend on some sort of a named environment variable or something, assuming adding a new field in CLC is not desirable. Can't we do the above? We definitely cannot insert mapreduce incantations like job.jar into YARN. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch, > YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546331#comment-14546331 ] Hadoop QA commented on YARN-3626: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 15m 24s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 47s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 10m 7s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 2m 16s | The applied patch generated 2 new checkstyle issues (total was 59, now 58). | | {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that end in whitespace. | | {color:green}+1{color} | install | 1m 33s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 3m 33s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. | | {color:green}+1{color} | mapreduce tests | 0m 45s | Tests passed in hadoop-mapreduce-client-common. | | {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in hadoop-yarn-api. | | {color:green}+1{color} | yarn tests | 6m 1s | Tests passed in hadoop-yarn-server-nodemanager. | | | | 48m 49s | | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12733257/YARN-3626.9.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / f7e051c | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/7954/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt | | hadoop-mapreduce-client-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/7954/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt | | hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/7954/artifact/patchprocess/testrun_hadoop-yarn-api.txt | | hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7954/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7954/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7954/console | This message was automatically generated. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch, > YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546288#comment-14546288 ] Craig Welch commented on YARN-3626: --- [~cnauroth], [~vvasudev] - This patch goes back to the original approach I passed by you offline - the fix itself is the same, but it uses the classpath instead of configuration to determine when the behavior should change. Your thoughts? > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch, > YARN-3626.9.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543904#comment-14543904 ] Hudson commented on YARN-3626: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2143 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2143/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/CHANGES.txt > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543856#comment-14543856 ] Hudson commented on YARN-3626: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #185 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/185/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543818#comment-14543818 ] Hudson commented on YARN-3626: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2125 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2125/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/CHANGES.txt > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543799#comment-14543799 ] Hudson commented on YARN-3626: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #195 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/195/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543581#comment-14543581 ] Hudson commented on YARN-3626: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #196 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/196/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543525#comment-14543525 ] Hudson commented on YARN-3626: -- FAILURE: Integrated in Hadoop-Yarn-trunk #927 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/927/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * hadoop-yarn-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14542629#comment-14542629 ] Hudson commented on YARN-3626: -- FAILURE: Integrated in Hadoop-trunk-Commit #7819 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7819/]) YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14542609#comment-14542609 ] Xuan Gong commented on YARN-3626: - Committed into trunk/branch-2/branch-2.7. Thanks, Craig. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.7.1 > > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14542598#comment-14542598 ] Xuan Gong commented on YARN-3626: - +1 LGTM. Will commit > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14542553#comment-14542553 ] Hadoop QA commented on YARN-3626: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 14m 44s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 32s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 1m 46s | The applied patch generated 3 new checkstyle issues (total was 211, now 214). | | {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that end in whitespace. | | {color:green}+1{color} | install | 1m 34s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 3m 23s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. | | {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in hadoop-mapreduce-client-common. | | {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in hadoop-yarn-api. | | {color:green}+1{color} | yarn tests | 5m 59s | Tests passed in hadoop-yarn-server-nodemanager. | | | | 46m 57s | | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12732645/YARN-3626.6.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / cdec12d | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/7924/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt | | hadoop-mapreduce-client-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/7924/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt | | hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/7924/artifact/patchprocess/testrun_hadoop-yarn-api.txt | | hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7924/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7924/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7924/console | This message was automatically generated. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14542559#comment-14542559 ] Craig Welch commented on YARN-3626: --- Checkstyle looks unimportant, everything else OK > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541530#comment-14541530 ] Hadoop QA commented on YARN-3626: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 15m 8s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 2 new or modified test files. | | {color:green}+1{color} | javac | 7m 42s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 44s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 1m 47s | The applied patch generated 3 new checkstyle issues (total was 211, now 214). | | {color:red}-1{color} | whitespace | 0m 1s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 36s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 3m 27s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. | | {color:green}+1{color} | mapreduce tests | 0m 47s | Tests passed in hadoop-mapreduce-client-common. | | {color:green}+1{color} | yarn tests | 0m 23s | Tests passed in hadoop-yarn-api. | | {color:red}-1{color} | yarn tests | 5m 11s | Tests failed in hadoop-yarn-server-nodemanager. | | | | 46m 54s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.yarn.server.nodemanager.containermanager.TestContainerManager | | | hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService | | | hadoop.yarn.server.nodemanager.containermanager.TestNMProxy | | | hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch | | | hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12732495/YARN-3626.4.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / fcd0702 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/whitespace.txt | | hadoop-mapreduce-client-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt | | hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/testrun_hadoop-yarn-api.txt | | hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7912/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7912/console | This message was automatically generated. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3626.0.patch, YARN-3626.4.patch > > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14538849#comment-14538849 ] Craig Welch commented on YARN-3626: --- To resolve this, the situation should be detected and, when applicable, localized resources should be put at the beginning of the classpath rather than the end. > On Windows localized resources are not moved to the front of the classpath > when they should be > -- > > Key: YARN-3626 > URL: https://issues.apache.org/jira/browse/YARN-3626 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Environment: Windows >Reporter: Craig Welch >Assignee: Craig Welch > > In response to the mapreduce.job.user.classpath.first setting the classpath > is ordered differently so that localized resources will appear before system > classpath resources when tasks execute. On Windows this does not work > because the localized resources are not linked into their final location when > the classpath jar is created. To compensate for that localized jar resources > are added directly to the classpath generated for the jar rather than being > discovered from the localized directories. Unfortunately, they are always > appended to the classpath, and so are never preferred over system resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)