[jira] [Commented] (YARN-80) Support delay scheduling for node locality in MR2's capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450581#comment-13450581 ] Hudson commented on YARN-80: Integrated in Hadoop-Hdfs-trunk #1158 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1158/]) YARN-80. Add support for delaying rack-local containers in CapacityScheduler. Contributed by Arun C. Murthy. (Revision 1381872) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381872 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java > Support delay scheduling for node locality in MR2's capacity scheduler > -- > > Key: YARN-80 > URL: https://issues.apache.org/jira/browse/YARN-80 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Reporter: Todd Lipcon >Assignee: Arun C Murthy > Fix For: 2.1.0-alpha > > Attachments: YARN-80.patch, YARN-80.patch > > > The capacity scheduler in MR2 doesn't support delay scheduling for achieving > node-level locality. So, jobs exhibit poor data locality even if they have > good rack locality. Especially on clusters where disk throughput is much > better than network capacity, this hurts overall job performance. We should > optionally support node-level delay scheduling heuristics similar to what the > fair scheduler implements in MR1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-75) RMContainer should handle a RELEASE event while RUNNING
[ https://issues.apache.org/jira/browse/YARN-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450586#comment-13450586 ] Hudson commented on YARN-75: Integrated in Hadoop-Hdfs-trunk #1158 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1158/]) YARN-75. Missing test. Ugh. (Revision 1381811) YARN-75. Added file missing from previous commit. Bad day with svn. (Revision 1381809) YARN-75. Removing spurious file from the previous commit. (Revision 1381807) YARN-75. Modified ResourceManager's RMContainer to handle a valid RELEASE event at RUNNING state. Contributed by Siddharth Seth. (Revision 1381806) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381811 Files : * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381809 Files : * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381807 Files : * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java.orig vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381806 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java.orig > RMContainer should handle a RELEASE event while RUNNING > --- > > Key: YARN-75 > URL: https://issues.apache.org/jira/browse/YARN-75 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.0-alpha >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0-alpha > > Attachments: YARN-75_2.txt, YARN-75.txt > > > An AppMaster can send a container release at any point. Currently this > results in an exception, if this is done while the RM considers the container > to be RUNNING. > The event not being processed correctly also implies that these containers do > not show up in the Completed Container List seen by the AM (AMRMProtocol). > MR-3902 depends on this set being complete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-15) YarnConfiguration DEFAULT_YARN_APPLICATION_CLASSPATH should be updated
[
https://issues.apache.org/jira/browse/YARN-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450593#comment-13450593
]
Hudson commented on YARN-15:
Integrated in Hadoop-Hdfs-trunk #1158 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1158/])
YARN-15. Fixing CHANGES.txt (Revision 1381801)
YARN-15. Updated default classpath for YARN applications to reflect split of
YARN into a sub-project. Contributed by Arun C Murthy. (Revision 1381794)
Result = FAILURE
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381801
Files :
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381794
Files :
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
> YarnConfiguration DEFAULT_YARN_APPLICATION_CLASSPATH should be updated
> --
>
> Key: YARN-15
> URL: https://issues.apache.org/jira/browse/YARN-15
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.2.0-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Arun C Murthy
>Priority: Critical
> Fix For: 2.2.0-alpha
>
> Attachments: YARN-15-branch-0.23.txt, YARN-86.patch
>
>
> {code}
> /**
>* Default CLASSPATH for YARN applications. A comma-separated list of
>* CLASSPATH entries
>*/
> public static final String[] DEFAULT_YARN_APPLICATION_CLASSPATH = {
> "$HADOOP_CONF_DIR", "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/share/hadoop/mapreduce/*",
> "$YARN_HOME/share/hadoop/mapreduce/lib/*"};
> {code}
> It should have {{share/yarn/}} and MR should add the {{share/mapreduce/}}
> (another JIRA?)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-80) Support delay scheduling for node locality in MR2's capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450627#comment-13450627 ] Hudson commented on YARN-80: Integrated in Hadoop-Mapreduce-trunk #1189 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1189/]) YARN-80. Add support for delaying rack-local containers in CapacityScheduler. Contributed by Arun C. Murthy. (Revision 1381872) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381872 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java > Support delay scheduling for node locality in MR2's capacity scheduler > -- > > Key: YARN-80 > URL: https://issues.apache.org/jira/browse/YARN-80 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Reporter: Todd Lipcon >Assignee: Arun C Murthy > Fix For: 2.1.0-alpha > > Attachments: YARN-80.patch, YARN-80.patch > > > The capacity scheduler in MR2 doesn't support delay scheduling for achieving > node-level locality. So, jobs exhibit poor data locality even if they have > good rack locality. Especially on clusters where disk throughput is much > better than network capacity, this hurts overall job performance. We should > optionally support node-level delay scheduling heuristics similar to what the > fair scheduler implements in MR1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-75) RMContainer should handle a RELEASE event while RUNNING
[ https://issues.apache.org/jira/browse/YARN-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450632#comment-13450632 ] Hudson commented on YARN-75: Integrated in Hadoop-Mapreduce-trunk #1189 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1189/]) YARN-75. Missing test. Ugh. (Revision 1381811) YARN-75. Added file missing from previous commit. Bad day with svn. (Revision 1381809) YARN-75. Removing spurious file from the previous commit. (Revision 1381807) YARN-75. Modified ResourceManager's RMContainer to handle a valid RELEASE event at RUNNING state. Contributed by Siddharth Seth. (Revision 1381806) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381811 Files : * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381809 Files : * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381807 Files : * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java.orig vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381806 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java.orig > RMContainer should handle a RELEASE event while RUNNING > --- > > Key: YARN-75 > URL: https://issues.apache.org/jira/browse/YARN-75 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.0-alpha >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0-alpha > > Attachments: YARN-75_2.txt, YARN-75.txt > > > An AppMaster can send a container release at any point. Currently this > results in an exception, if this is done while the RM considers the container > to be RUNNING. > The event not being processed correctly also implies that these containers do > not show up in the Completed Container List seen by the AM (AMRMProtocol). > MR-3902 depends on this set being complete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-15) YarnConfiguration DEFAULT_YARN_APPLICATION_CLASSPATH should be updated
[
https://issues.apache.org/jira/browse/YARN-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450639#comment-13450639
]
Hudson commented on YARN-15:
Integrated in Hadoop-Mapreduce-trunk #1189 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1189/])
YARN-15. Fixing CHANGES.txt (Revision 1381801)
YARN-15. Updated default classpath for YARN applications to reflect split of
YARN into a sub-project. Contributed by Arun C Murthy. (Revision 1381794)
Result = SUCCESS
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381801
Files :
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381794
Files :
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
> YarnConfiguration DEFAULT_YARN_APPLICATION_CLASSPATH should be updated
> --
>
> Key: YARN-15
> URL: https://issues.apache.org/jira/browse/YARN-15
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.2.0-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Arun C Murthy
>Priority: Critical
> Fix For: 2.2.0-alpha
>
> Attachments: YARN-15-branch-0.23.txt, YARN-86.patch
>
>
> {code}
> /**
>* Default CLASSPATH for YARN applications. A comma-separated list of
>* CLASSPATH entries
>*/
> public static final String[] DEFAULT_YARN_APPLICATION_CLASSPATH = {
> "$HADOOP_CONF_DIR", "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/share/hadoop/mapreduce/*",
> "$YARN_HOME/share/hadoop/mapreduce/lib/*"};
> {code}
> It should have {{share/yarn/}} and MR should add the {{share/mapreduce/}}
> (another JIRA?)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-57) Plugable process tree
[ https://issues.apache.org/jira/browse/YARN-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-57: -- Fix Version/s: 3.0.0 > Plugable process tree > - > > Key: YARN-57 > URL: https://issues.apache.org/jira/browse/YARN-57 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager > Environment: FreeBSD 64 bit >Reporter: Radim Kolar >Assignee: Radim Kolar > Fix For: 3.0.0 > > Attachments: plugable-pstree-1.txt, plugable-pstree-2.txt, > plugable-pstree-3.txt, plugable-pstree-4.txt, > plugable-pstree-4-with-whitespace.txt, plugable-pstree-5-with-whitespace.txt, > plugable-pstree-6-typofix.txt, plugable-pstree-trunk-merge-7.txt, > plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-8.txt, > plugable-pstree.txt > > > Trunk version of Pluggable process tree. Work based on MAPREDUCE-4204 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-57) Plugable process tree
[ https://issues.apache.org/jira/browse/YARN-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450738#comment-13450738 ] Hudson commented on YARN-57: Integrated in Hadoop-Common-trunk-Commit #2705 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2705/]) YARN-57. Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java > Plugable process tree > - > > Key: YARN-57 > URL: https://issues.apache.org/jira/browse/YARN-57 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager > Environment: FreeBSD 64 bit >Reporter: Radim Kolar >Assignee: Radim Kolar > Fix For: 3.0.0 > > Attachments: plugable-pstree-1.txt, plugable-pstree-2.txt, > plugable-pstree-3.txt, plugable-pstree-4.txt, > plugable-pstree-4-with-whitespace.txt, plugable-pstree-5-with-whitespace.txt, > plugable-pstree-6-typofix.txt, plugable-pstree-trunk-merge-7.txt, > plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-8.txt, > plugable-pstree.txt > > > Trunk version of Pluggable process tree. Work based on MAPREDUCE-4204 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-57) Plugable process tree
[ https://issues.apache.org/jira/browse/YARN-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450739#comment-13450739 ] Hudson commented on YARN-57: Integrated in Hadoop-Hdfs-trunk-Commit #2768 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2768/]) YARN-57. Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java > Plugable process tree > - > > Key: YARN-57 > URL: https://issues.apache.org/jira/browse/YARN-57 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager > Environment: FreeBSD 64 bit >Reporter: Radim Kolar >Assignee: Radim Kolar > Fix For: 3.0.0 > > Attachments: plugable-pstree-1.txt, plugable-pstree-2.txt, > plugable-pstree-3.txt, plugable-pstree-4.txt, > plugable-pstree-4-with-whitespace.txt, plugable-pstree-5-with-whitespace.txt, > plugable-pstree-6-typofix.txt, plugable-pstree-trunk-merge-7.txt, > plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-8.txt, > plugable-pstree.txt > > > Trunk version of Pluggable process tree. Work based on MAPREDUCE-4204 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-57) Plugable process tree
[ https://issues.apache.org/jira/browse/YARN-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450773#comment-13450773 ] Hudson commented on YARN-57: Integrated in Hadoop-Mapreduce-trunk-Commit #2729 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2729/]) YARN-57. Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java > Plugable process tree > - > > Key: YARN-57 > URL: https://issues.apache.org/jira/browse/YARN-57 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager > Environment: FreeBSD 64 bit >Reporter: Radim Kolar >Assignee: Radim Kolar > Fix For: 3.0.0 > > Attachments: plugable-pstree-1.txt, plugable-pstree-2.txt, > plugable-pstree-3.txt, plugable-pstree-4.txt, > plugable-pstree-4-with-whitespace.txt, plugable-pstree-5-with-whitespace.txt, > plugable-pstree-6-typofix.txt, plugable-pstree-trunk-merge-7.txt, > plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-8.txt, > plugable-pstree.txt > > > Trunk version of Pluggable process tree. Work based on MAPREDUCE-4204 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-2) Enhance CS to schedule accounting for both memory and cpu cores
[ https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451094#comment-13451094 ] Vinod Kumar Vavilapalli commented on YARN-2: bq. Can anyone pls take a look? Tx. I am giving it a final look for commit. > Enhance CS to schedule accounting for both memory and cpu cores > --- > > Key: YARN-2 > URL: https://issues.apache.org/jira/browse/YARN-2 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacityscheduler, scheduler >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 2.0.2-alpha > > Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, > MAPREDUCE-4327.patch, MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, > MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, YARN-2-help.patch, > YARN-2.patch, YARN-2.patch > > > With YARN being a general purpose system, it would be useful for several > applications (MPI et al) to specify not just memory but also CPU (cores) for > their resource requirements. Thus, it would be useful to the > CapacityScheduler to account for both. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (YARN-89) Add tests for testing other NM components with disk failures
[ https://issues.apache.org/jira/browse/YARN-89?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli moved MAPREDUCE-3440 to YARN-89: Affects Version/s: (was: 0.23.0) Key: YARN-89 (was: MAPREDUCE-3440) Project: Hadoop YARN (was: Hadoop Map/Reduce) > Add tests for testing other NM components with disk failures > > > Key: YARN-89 > URL: https://issues.apache.org/jira/browse/YARN-89 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Ravi Gummadi > > Add more tests to test other components when disks fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (YARN-90) NodeManager should identify failed disks becoming good back again
[ https://issues.apache.org/jira/browse/YARN-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli moved MAPREDUCE-3441 to YARN-90: Component/s: (was: nodemanager) (was: mrv2) nodemanager Affects Version/s: (was: 0.23.0) Key: YARN-90 (was: MAPREDUCE-3441) Project: Hadoop YARN (was: Hadoop Map/Reduce) > NodeManager should identify failed disks becoming good back again > - > > Key: YARN-90 > URL: https://issues.apache.org/jira/browse/YARN-90 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Ravi Gummadi > > MAPREDUCE-3121 makes NodeManager identify disk failures. But once a disk goes > down, it is marked as failed forever. To reuse that disk (after it becomes > good), NodeManager needs restart. This JIRA is to improve NodeManager to > reuse good disks(which could be bad some time back). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-91) DFIP aka 'NodeManager should handle Disk-Failures In Place'
Vinod Kumar Vavilapalli created YARN-91: --- Summary: DFIP aka 'NodeManager should handle Disk-Failures In Place' Key: YARN-91 URL: https://issues.apache.org/jira/browse/YARN-91 Project: Hadoop YARN Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Moving stuff over from the MAPREDUCE JIRA: MAPREDUCE-3121 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (YARN-92) NM disk failure detection only covers local dirs
[ https://issues.apache.org/jira/browse/YARN-92?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli moved MAPREDUCE-3474 to YARN-92: Component/s: (was: nodemanager) (was: tasktracker) nodemanager Affects Version/s: (was: 0.20.205.0) (was: 0.23.0) Key: YARN-92 (was: MAPREDUCE-3474) Project: Hadoop YARN (was: Hadoop Map/Reduce) > NM disk failure detection only covers local dirs > - > > Key: YARN-92 > URL: https://issues.apache.org/jira/browse/YARN-92 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Eli Collins > > This is the MR counterpart to HDFS-1848. Like HDFS volume failure detection, > NM disk failure detection checks a subset of the disks, and a subset of the > directories. Eg the TT and the NM do not check the root disk for errors > unless a local dir resides on them. Even if a local dir resides on the root > disk the disk checking code only checks the local dirs so a failure only seen > when accessing a part of the disk no hosting the local dirs will not be > noticed. The disk that hosts the logs, pid, tmp dirs etc is critical, so if > needs to be checked as well, and the NM should shutdown if a critical disk is > not available (to prevent MR issues similar to HDFS-1848 and HDFS-2095). > Typically people currently work around this limitation by (aside from > ignoring it) by using raid-1 for the root disk or a health script that checks > the root disk health. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-92) NM disk failure detection only covers local dirs
[ https://issues.apache.org/jira/browse/YARN-92?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-92: Issue Type: Sub-task (was: Bug) Parent: YARN-91 > NM disk failure detection only covers local dirs > - > > Key: YARN-92 > URL: https://issues.apache.org/jira/browse/YARN-92 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Eli Collins > > This is the MR counterpart to HDFS-1848. Like HDFS volume failure detection, > NM disk failure detection checks a subset of the disks, and a subset of the > directories. Eg the TT and the NM do not check the root disk for errors > unless a local dir resides on them. Even if a local dir resides on the root > disk the disk checking code only checks the local dirs so a failure only seen > when accessing a part of the disk no hosting the local dirs will not be > noticed. The disk that hosts the logs, pid, tmp dirs etc is critical, so if > needs to be checked as well, and the NM should shutdown if a critical disk is > not available (to prevent MR issues similar to HDFS-1848 and HDFS-2095). > Typically people currently work around this limitation by (aside from > ignoring it) by using raid-1 for the root disk or a health script that checks > the root disk health. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-89) Add tests for testing other NM components with disk failures
[ https://issues.apache.org/jira/browse/YARN-89?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-89: Issue Type: Sub-task (was: Improvement) Parent: YARN-91 > Add tests for testing other NM components with disk failures > > > Key: YARN-89 > URL: https://issues.apache.org/jira/browse/YARN-89 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Ravi Gummadi > > Add more tests to test other components when disks fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-90) NodeManager should identify failed disks becoming good back again
[ https://issues.apache.org/jira/browse/YARN-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-90: Issue Type: Sub-task (was: Bug) Parent: YARN-91 > NodeManager should identify failed disks becoming good back again > - > > Key: YARN-90 > URL: https://issues.apache.org/jira/browse/YARN-90 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Ravi Gummadi > > MAPREDUCE-3121 makes NodeManager identify disk failures. But once a disk goes > down, it is marked as failed forever. To reuse that disk (after it becomes > good), NodeManager needs restart. This JIRA is to improve NodeManager to > reuse good disks(which could be bad some time back). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-78) Change UnmanagedAMLauncher to use YarnClientImpl
[ https://issues.apache.org/jira/browse/YARN-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451198#comment-13451198 ] Vinod Kumar Vavilapalli commented on YARN-78: - Looks good now. +1. Kicking Jenkins. > Change UnmanagedAMLauncher to use YarnClientImpl > > > Key: YARN-78 > URL: https://issues.apache.org/jira/browse/YARN-78 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.0.2-alpha, 2.0.0-alpha >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: YARN-78.1.patch, YARN-78.2.patch, YARN-78.3.patch > > > YARN-29 added a common client impl to talk to the RM. Use that in the > UnmanagedAMLauncher. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
