[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129695#comment-13129695 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Hdfs-0.23-Build #43 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/43/]) Merge -c 1185447 from trunk to branch-0.23 to complete fix for MAPREDUCE-3068. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185448 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129713#comment-13129713 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Mapreduce-trunk #864 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/864/]) MAPREDUCE-3068. Added a whitelist of environment variables for containers from the NodeManager and set MALLOC_ARENA_MAX for all daemons and containers. Contributed by Chris Riccomini. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185447 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129722#comment-13129722 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Mapreduce-0.23-Build #55 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/55/]) Merge -c 1185447 from trunk to branch-0.23 to complete fix for MAPREDUCE-3068. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185448 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129731#comment-13129731 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Hdfs-trunk #834 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/834/]) MAPREDUCE-3068. Added a whitelist of environment variables for containers from the NodeManager and set MALLOC_ARENA_MAX for all daemons and containers. Contributed by Chris Riccomini. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185447 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128988#comment-13128988 ] Hadoop QA commented on MAPREDUCE-3068: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499394/MAPREDUCE-3068.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1041//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1041//console This message is automatically generated. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129058#comment-13129058 ] Chris Riccomini commented on MAPREDUCE-3068: Hey Arun, Do you want just the putIfAbsent vars specified in the latest patch, or do you want to modify setEnvFromInputString to putIfAbsent for ALL env variables specified in admin-env? I have some concern about doing the latter, as it changes the existing behavior of the setEnv.. method, which is used in MR, not just YARN. Thanks! Chris Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129134#comment-13129134 ] Hadoop QA commented on MAPREDUCE-3068: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499422/MAPREDUCE-3068.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.TestJobMonitorAndPrint +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1042//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1042//console This message is automatically generated. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129139#comment-13129139 ] Arun C Murthy commented on MAPREDUCE-3068: -- Chris - we should allow applications to override JAVA_HOME, HADOOP_COMMON_HOME, HADOOP_HDFS_HOME and YARN_HOME. Thus, we want the putIfAbsent semantics when we are setting them from either the NodeManager's environment or from the admin.env. Does that make sense? Thanks. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129142#comment-13129142 ] Chris Riccomini commented on MAPREDUCE-3068: Hey Arun, Could you have a look at the patch I just put up? It uses putIfAbsent from sanitizeEnv - setEnvFromInputStringIfAbsent - addToEnvironment. That way, any env variables set via admin-env will only be applied if they aren't already set in the environment. This should allow apps to override the variables, I believe. I thought it was a little goofy to have putIfAbsent specifically called out for certain env variables and not for others, so I put them all in admin-env, and made all admin-env variables putIfAbsent. That way it covers both the env variables that you list out, as well as any other variables that someone may add. Is this OK? Cheers, Chris Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129149#comment-13129149 ] Arun C Murthy commented on MAPREDUCE-3068: -- I think that makes sense. Thanks. I'll review your patch once more and commit. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129157#comment-13129157 ] Arun C Murthy commented on MAPREDUCE-3068: -- Oh wait, on second thoughts... (sorry, you aren't going to like it) I think we want a whitelist of stuff apps can override (MAPREDUCE-2784). Thus we allow only those to be overridable and everything else on admin.env shouldn't be. Hope that makes sense, sorry again! Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129159#comment-13129159 ] Arun C Murthy commented on MAPREDUCE-3068: -- Sorry, that ref. should have been MAPREDUCE-2748 which tracks a 'whitelist'. As Vinod mentioned earlier, we can close that as a dup of this one once we add the whitelist too. Makes sense? Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129165#comment-13129165 ] Chris Riccomini commented on MAPREDUCE-3068: Hey Arun, Just to clarify, basically, you want what I had before, right? Essentially, admin-env never over-writes, and we have hard coded putIfAbsent calls in sanitizeEnv for specific variables (JAVA_HOME, HADOOP_COMMON_HOME, HADOOP_HDFS_HOME and YARN_HOME) Cheers, Chris Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129172#comment-13129172 ] Arun C Murthy commented on MAPREDUCE-3068: -- Yep, that will work. Even better - add a whitelist config key instead of hardcoding the 4 env vars... I'd prefer that if you don't mind. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129171#comment-13129171 ] Hadoop QA commented on MAPREDUCE-3068: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499426/MAPREDUCE-3068.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1711 javac compiler warnings (more than the trunk's current 1710 warnings). -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.TestJobMonitorAndPrint +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1043//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1043//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1043//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1043//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1043//console This message is automatically generated. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129176#comment-13129176 ] Chris Riccomini commented on MAPREDUCE-3068: I was worried you would say that. ;) Give me 10 mins. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129187#comment-13129187 ] Arun C Murthy commented on MAPREDUCE-3068: -- Thanks Chris! Really appreciate your patience... :) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129206#comment-13129206 ] Arun C Murthy commented on MAPREDUCE-3068: -- +1, looks good. Thanks Chris! I'll commit after I get the go ahead from Mr. Hudson. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129212#comment-13129212 ] Hadoop QA commented on MAPREDUCE-3068: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499437/MAPREDUCE-3068.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1711 javac compiler warnings (more than the trunk's current 1710 warnings). -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.TestJobMonitorAndPrint +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1044//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1044//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1044//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1044//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1044//console This message is automatically generated. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129211#comment-13129211 ] Mahadev konar commented on MAPREDUCE-3068: -- Chris, One more Minor nit :) Your latest patch does not have the Apache licence grant check. You'd have to re upload the patch with the license granted. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129219#comment-13129219 ] Mahadev konar commented on MAPREDUCE-3068: -- thanks Chris, Ill let Arun do the honors :) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129234#comment-13129234 ] Chris Riccomini commented on MAPREDUCE-3068: OK, so now there's some test job monitor failure going off. Trunk doesn't seem very stable at this point. Especially given https://issues.apache.org/jira/browse/MAPREDUCE-3195 Is there an easy way to run trunk without my patch so that we can properly diff it? Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129239#comment-13129239 ] Chris Riccomini commented on MAPREDUCE-3068: Also, the test failure is coming from: TestJobMonitorAndPrint.java Layout layout = Logger.getRootLogger().getAppender(stdout).getLayout(); This doesn't have much to do with me. Same thing with the FindBugs stuff. I poked around the javac warnings and I think it's getting confused because of a cast in TestContainerLaunch that's not part of my code. Cheers, Chris Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129242#comment-13129242 ] Chris Riccomini commented on MAPREDUCE-3068: The last run of my job that didn't fail with this testjobmonitorandprint issue was four runs ago. Immediately after: https://issues.apache.org/jira/browse/MAPREDUCE-3003 I started seeing the testjobmonitorandprint failures. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129371#comment-13129371 ] Chris Riccomini commented on MAPREDUCE-3068: Hey Arun, Yea, my do-nothing patch with the same problems. Could we add the latest non-blank patch to commit? Thanks! Chris Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129402#comment-13129402 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Common-0.23-Commit #13 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/13/]) Merge -c 1185447 from trunk to branch-0.23 to complete fix for MAPREDUCE-3068. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185448 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129403#comment-13129403 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Hdfs-trunk-Commit #1177 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1177/]) MAPREDUCE-3068. Added a whitelist of environment variables for containers from the NodeManager and set MALLOC_ARENA_MAX for all daemons and containers. Contributed by Chris Riccomini. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185447 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129405#comment-13129405 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Common-trunk-Commit #1098 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1098/]) MAPREDUCE-3068. Added a whitelist of environment variables for containers from the NodeManager and set MALLOC_ARENA_MAX for all daemons and containers. Contributed by Chris Riccomini. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185447 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129406#comment-13129406 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Hdfs-0.23-Commit #14 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/14/]) Merge -c 1185447 from trunk to branch-0.23 to complete fix for MAPREDUCE-3068. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185448 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129411#comment-13129411 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Mapreduce-0.23-Commit #15 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/15/]) Merge -c 1185447 from trunk to branch-0.23 to complete fix for MAPREDUCE-3068. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185448 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129413#comment-13129413 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1117 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1117/]) MAPREDUCE-3068. Added a whitelist of environment variables for containers from the NodeManager and set MALLOC_ARENA_MAX for all daemons and containers. Contributed by Chris Riccomini. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185447 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127678#comment-13127678 ] Chris Riccomini commented on MAPREDUCE-3068: Hey Arun, Are you talking about the putIfAbsent stuff that I removed in the sanitizeEnv function? I'll have a look at the javadoc stuff. Thanks! Chris Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127858#comment-13127858 ] Arun C Murthy commented on MAPREDUCE-3068: -- Yep, the putIfAbsent semantics are important since we want to allow containers to set an alternate HADOOP_*_HOME. Does that make sense? Thanks. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127255#comment-13127255 ] Hadoop QA commented on MAPREDUCE-3068: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498958/MAPREDUCE-3068.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 20 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1016//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1016//console This message is automatically generated. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118690#comment-13118690 ] Chris Riccomini commented on MAPREDUCE-3068: Any feedback on this? Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116896#comment-13116896 ] Hadoop QA commented on MAPREDUCE-3068: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496943/MAPREDUCE-3068.diff against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/886//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/886//console This message is automatically generated. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13112699#comment-13112699 ] Chris Riccomini commented on MAPREDUCE-3068: Arun says in https://issues.apache.org/jira/browse/MAPREDUCE-3065 that the NM should be modified to do this. Based on poking around, it appears that bin/yarn-config.sh is a better place to do this, as it then makes it into the RM, NM, and the Container from there (given that the container launch context gets the env variables from its parent process, the NM). Does anyone have an argument for not doing it in yarn-config.sh? Already tested locally, and it appears to work. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Fix For: 0.23.0 This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13112727#comment-13112727 ] Arun C Murthy commented on MAPREDUCE-3068: -- Chris, setting this in yarn-default.sh will work for RM/NM itself. However, the containers don't inherit NM's environment - thus we'll need to explicitly do it. Take a look at ContainerLaunch.sanitizeEnv. To be more general we need to add a notion of 'admin env' so admins can set MALLOC_ARENA_MAX and others automatically for all containers. Makes sense? Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13112744#comment-13112744 ] Chris Riccomini commented on MAPREDUCE-3068: Think so. Regarding admin env, are you saying an admin-env.sh file (similar to hadoop-env.sh)? Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
