[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14185578#comment-14185578 ] Thomas Graves commented on YARN-443: Can you be more specific, what is different about it and why it is a problem? The trunk patch shows that there was an existing getRunCommand() routine (before this change) where as the other didn't have one before (it looks like for windows support). > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.23.7, 2.0.4-alpha > > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14185542#comment-14185542 ] Xuan Gong commented on YARN-443: [~tgraves] I found that the function ContainerExecutor.getRunCommand() in the trunk patch is different from that in branch-2/branch-0.23 patch. Is there any reason why we are doing that ? > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.23.7, 2.0.4-alpha > > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155840#comment-14155840 ] Vinod Kumar Vavilapalli commented on YARN-443: -- The new test TestContainerExecutor was missed in branch-2, I committed it as part of YARN-1972. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.23.7, 2.0.4-alpha > > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597950#comment-13597950 ] Hudson commented on YARN-443: - Integrated in Hadoop-Mapreduce-trunk #1367 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1367/]) YARN-443. allow OS scheduling priority of NM to be different than the containers it launches (tgraves) (Revision 1454411) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454411 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 3.0.0, 0.23.7, 2.0.4-alpha > > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597928#comment-13597928 ] Hudson commented on YARN-443: - Integrated in Hadoop-Hdfs-trunk #1339 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1339/]) YARN-443. allow OS scheduling priority of NM to be different than the containers it launches (tgraves) (Revision 1454411) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454411 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 3.0.0, 0.23.7, 2.0.4-alpha > > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597919#comment-13597919 ] Hudson commented on YARN-443: - Integrated in Hadoop-Hdfs-0.23-Build #548 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/548/]) YARN-443. allow OS scheduling priority of NM to be different than the containers it launches (tgraves) (Revision 1454412) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454412 Files : * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 3.0.0, 0.23.7, 2.0.4-alpha > > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597900#comment-13597900 ] Hudson commented on YARN-443: - Integrated in Hadoop-Yarn-trunk #150 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/150/]) YARN-443. allow OS scheduling priority of NM to be different than the containers it launches (tgraves) (Revision 1454411) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454411 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 3.0.0, 0.23.7, 2.0.4-alpha > > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597191#comment-13597191 ] Hudson commented on YARN-443: - Integrated in Hadoop-trunk-Commit #3440 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3440/]) YARN-443. allow OS scheduling priority of NM to be different than the containers it launches (tgraves) (Revision 1454411) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454411 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597154#comment-13597154 ] Thomas Graves commented on YARN-443: I intended to leave that log message in there because if anything comes back on stderr from try running the container, it goes into the exception but then its never logged. I'll actually go ahead and remove that and file a follow up jira as ideally any message would go into the diagnostics that get returned to the user also. I'll fix the typo for than and commit. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596516#comment-13596516 ] Bikas Saha commented on YARN-443: - The LCE changes look good too. Dont think theres anything wrong there. Mostly minor changes. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[
https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596514#comment-13596514
]
Bikas Saha commented on YARN-443:
-
+1 looks good to me subject to above LCE caveat :P
Minor nits
Did you intend to keep this log message?
{code}
+ logOutput(e.getMessage());
{code}
You mean "than"
{code}
+ * On Linux, higher values mean run the containers at a less
+ * favorable priority then the NM.
{code}
> allow OS scheduling priority of NM to be different than the containers it
> launches
> --
>
> Key: YARN-443
> URL: https://issues.apache.org/jira/browse/YARN-443
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.0.3-alpha, 0.23.6
>Reporter: Thomas Graves
>Assignee: Thomas Graves
> Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
> YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
> YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch,
> YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch,
> YARN-443.patch, YARN-443.patch, YARN-443.patch
>
>
> It would be nice if we could have the nodemanager run at a different OS
> scheduling priority than the containers so that you can still communicate
> with the nodemanager if the containers out of control.
> On linux we could launch the nodemanager at a higher priority, but then all
> the containers it launches would also be at that higher priority, so we need
> a way for the container executor to launch them at a lower priority.
> I'm not sure how this applies to windows if at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[
https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596471#comment-13596471
]
Hadoop QA commented on YARN-443:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572619/YARN-443.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/479//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/479//console
This message is automatically generated.
> allow OS scheduling priority of NM to be different than the containers it
> launches
> --
>
> Key: YARN-443
> URL: https://issues.apache.org/jira/browse/YARN-443
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.0.3-alpha, 0.23.6
>Reporter: Thomas Graves
>Assignee: Thomas Graves
> Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
> YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
> YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch,
> YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch,
> YARN-443.patch, YARN-443.patch, YARN-443.patch
>
>
> It would be nice if we could have the nodemanager run at a different OS
> scheduling priority than the containers so that you can still communicate
> with the nodemanager if the containers out of control.
> On linux we could launch the nodemanager at a higher priority, but then all
> the containers it launches would also be at that higher priority, so we need
> a way for the container executor to launch them at a lower priority.
> I'm not sure how this applies to windows if at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596430#comment-13596430 ] Thomas Graves commented on YARN-443: thanks Bikas, I'll remove that if and upload a new trunk patch shortly. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596399#comment-13596399 ] Bikas Saha commented on YARN-443: - I am not that familiar with the LCE code but it looks good overall. There seems to be some command line generation code duplication in the LCE code but I dont think its for this jira. I dont think we need an if(Shell.Windows) in the test for LCE.java, right? > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, > YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[
https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596345#comment-13596345
]
Hadoop QA commented on YARN-443:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572602/YARN-443.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/478//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/478//console
This message is automatically generated.
> allow OS scheduling priority of NM to be different than the containers it
> launches
> --
>
> Key: YARN-443
> URL: https://issues.apache.org/jira/browse/YARN-443
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.0.3-alpha, 0.23.6
>Reporter: Thomas Graves
>Assignee: Thomas Graves
> Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
> YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
> YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch,
> YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch,
> YARN-443.patch, YARN-443.patch
>
>
> It would be nice if we could have the nodemanager run at a different OS
> scheduling priority than the containers so that you can still communicate
> with the nodemanager if the containers out of control.
> On linux we could launch the nodemanager at a higher priority, but then all
> the containers it launches would also be at that higher priority, so we need
> a way for the container executor to launch them at a lower priority.
> I'm not sure how this applies to windows if at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[
https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596325#comment-13596325
]
Hadoop QA commented on YARN-443:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572597/YARN-443.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/477//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/477//console
This message is automatically generated.
> allow OS scheduling priority of NM to be different than the containers it
> launches
> --
>
> Key: YARN-443
> URL: https://issues.apache.org/jira/browse/YARN-443
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.0.3-alpha, 0.23.6
>Reporter: Thomas Graves
>Assignee: Thomas Graves
> Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
> YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
> YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch,
> YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch,
> YARN-443.patch, YARN-443.patch
>
>
> It would be nice if we could have the nodemanager run at a different OS
> scheduling priority than the containers so that you can still communicate
> with the nodemanager if the containers out of control.
> On linux we could launch the nodemanager at a higher priority, but then all
> the containers it launches would also be at that higher priority, so we need
> a way for the container executor to launch them at a lower priority.
> I'm not sure how this applies to windows if at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596169#comment-13596169 ] Thomas Graves commented on YARN-443: Thanks for the review Bikas, I'll remove the bits for modifying the windows stuff and update based on your comments. I agree, I wish it was more common but didn't want to take that on here. I can update the comment to say higher values mean less favorable priority on Linux. We could make that the standard if windows wants to handle converting that. I changed the name to be the adjustment, one because it fit as parameter to nice but also in an attempt to make it less platform specific. I was hoping it would allows us to say things like positive numbers in adjustment on any platform make the container run less favorable to NM and then each platform could interpret that. We don't have to state that now though. It doesn't necessarily have to be +ve either. If you nodemanager has root priveleges it can be negative on linux although I don't know of anyone who would do that. I also didn't want to restrict that for other platforms. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596141#comment-13596141 ] Bikas Saha commented on YARN-443: - winutils will not support nice. and windows support does not use cygwin. infact, it removes cygwin. so the above patch will likely cause all tests to fail on Windows :) Why dont we set a local variable with the result of the scheduling configuration? And use that in the non-Windows part of the command line generation? Later we can use that value for the windows part when it gets supported. YARN-456. Minor nit, how about a default value in config instead of the hard coded 0 for priority? Also if the comments describing the config could be made easier for someone implementing YARN-456 or using the feature. something like. The value must be a +ve integer and higher values mean lower priority? Then the current comments could explain how the value is used in linux. I wish linuxcontainerexecutor.java did not need to replicate almost all the changes made in defaultcontainerexecutor :( > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, > YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, > YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[
https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596107#comment-13596107
]
Hadoop QA commented on YARN-443:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572561/YARN-443.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/476//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/476//console
This message is automatically generated.
> allow OS scheduling priority of NM to be different than the containers it
> launches
> --
>
> Key: YARN-443
> URL: https://issues.apache.org/jira/browse/YARN-443
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.0.3-alpha, 0.23.6
>Reporter: Thomas Graves
>Assignee: Thomas Graves
> Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
> YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch,
> YARN-443.patch
>
>
> It would be nice if we could have the nodemanager run at a different OS
> scheduling priority than the containers so that you can still communicate
> with the nodemanager if the containers out of control.
> On linux we could launch the nodemanager at a higher priority, but then all
> the containers it launches would also be at that higher priority, so we need
> a way for the container executor to launch them at a lower priority.
> I'm not sure how this applies to windows if at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[
https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595980#comment-13595980
]
Hadoop QA commented on YARN-443:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572548/YARN-443.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/475//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/475//console
This message is automatically generated.
> allow OS scheduling priority of NM to be different than the containers it
> launches
> --
>
> Key: YARN-443
> URL: https://issues.apache.org/jira/browse/YARN-443
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.0.3-alpha, 0.23.6
>Reporter: Thomas Graves
>Assignee: Thomas Graves
> Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
> YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch
>
>
> It would be nice if we could have the nodemanager run at a different OS
> scheduling priority than the containers so that you can still communicate
> with the nodemanager if the containers out of control.
> On linux we could launch the nodemanager at a higher priority, but then all
> the containers it launches would also be at that higher priority, so we need
> a way for the container executor to launch them at a lower priority.
> I'm not sure how this applies to windows if at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595129#comment-13595129 ] Thomas Graves commented on YARN-443: after thinking about it some more the issue with using nice means the NM needs to have sudo access on linux to do negatives values (more favorable priorities) and I was hoping to leave that open as an option. I'll go investigate other possibilities. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595111#comment-13595111 ] Thomas Graves commented on YARN-443: The container-executor generally takes in params on command line. One option though initialize_app (which is basically localize) already takes a variable number of arguments though. So that would require me to make the priority a mandatory parameter and then I would have to come up with a special value for if its not set. Yeah I'm not a fan of env variables either so would rather not do that. It looks like nice should work on cygwin also so I think I can just use that and append it to the command before calling, any objections? For example DefaultContainerExecutor command could become " nice -n bash WRAPPER_LAUNCH_SCRIPT. The sched.priority was meant to stand for os scheduling priority. I can change it to whatever people agree on would make the most sense. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595080#comment-13595080 ] Bikas Saha commented on YARN-443: - How about not having "sched" in the param name? We are launching containers at different priorities as opposed to scheduling. Or did you mean OS scheduling priority? > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595071#comment-13595071 ] Bikas Saha commented on YARN-443: - The native binary on windows is called winutils,exe. It does a lot more than container launch (eg file systems operations). Though it does not yet do everything that linux-executor does (cgroups/security). How are other configurable parameters passed onto the linux-executor? Though I am not a fan of env vars, we could use env vars as a method to pass config parameters to different executors? > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595063#comment-13595063 ] Thomas Graves commented on YARN-443: Bikas, where is the windows container executor? I looked in the branch-trunk-win but didn't see it. I'm ok with moving it up to yarn-site. I see some benefit for cross platforms, but I think it also makes it more complex. I'll make a config something like yarn.nodemanager.container-executor.sched.priority of type int. I'm going to change it to see if I can call nice from the java containerexecutor code as it builds up the command rather then inside the container-executor.c code. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443.patch, YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594905#comment-13594905 ] Bikas Saha commented on YARN-443: - I like Arun's suggestion of making this process priority choice explicit in yarn-site. Unless this gets exposed in the configuration/api/java-code it will be non-transparent to cross platform compatibility. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594800#comment-13594800 ] Thomas Graves commented on YARN-443: Note the patch attached isn't quite complete either. I want to do it for both initializing apps and launching containers. I think the delete files and signal container can stay at the same priority as NM. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594794#comment-13594794 ] Thomas Graves commented on YARN-443: Can you elaborate on your proposal. Are you saying use a yarn config and then pass an argument into the container executor or do something actually higher up like call nice when calling the container-executor? What do you propose for the DefaultContainerExecutor since it could run across multiple platforms and all it does is call a bash script? I figured with it in the container-executor itself each one could more easily decide if it applies and how exactly to implement it. I also like the fact that I can change it without rebooting the NM. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594767#comment-13594767 ] Arun C Murthy commented on YARN-443: Thomas, wouldn't it be better to come from yarn-site.xml so that it's more visible to admins/users who are configuring clusters? This way we wudn't need to set this in container-executor.cfg separately for the OS... > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594319#comment-13594319 ] Thomas Graves commented on YARN-443: I chose to do this in the container-executor itself and the config goes into the container-executor.cfg. This made the change more straight forward in that we didn't have to change the args passed to container-executor. Having the config in the container-executor.cfg also allows you to change it without restarting the NM. new config is: process.sched.priority. It can be set to anything that setpriority takes, including negative values since container-executor has root permissions. If the config is left out it retains the same behavior of having priority 0. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-443.patch > > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13590963#comment-13590963 ] Bikas Saha commented on YARN-443: - Yes, this can be done via the methods used in the current windows container launch code. > allow OS scheduling priority of NM to be different than the containers it > launches > -- > > Key: YARN-443 > URL: https://issues.apache.org/jira/browse/YARN-443 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha, 0.23.6 >Reporter: Thomas Graves >Assignee: Thomas Graves > > It would be nice if we could have the nodemanager run at a different OS > scheduling priority than the containers so that you can still communicate > with the nodemanager if the containers out of control. > On linux we could launch the nodemanager at a higher priority, but then all > the containers it launches would also be at that higher priority, so we need > a way for the container executor to launch them at a lower priority. > I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
