[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682552#comment-13682552 ] Hadoop QA commented on YARN-736: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587678/YARN-736.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1220//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1220//console This message is automatically generated. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682689#comment-13682689 ] Sandy Ryza commented on YARN-736: - Attached an updated patch that takes maximum shares into account. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682740#comment-13682740 ] Hadoop QA commented on YARN-736: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587698/YARN-736-1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1224//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1224//console This message is automatically generated. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13685854#comment-13685854 ] Karthik Kambatla commented on YARN-736: --- Thanks Sandy. Few comments: # The name ShareComputer doesn't capture the purpose of the class. What do you think of ComputeResourceShares or something along those lines? # In FairScheduler#isStarvedForDesiredFairShare(), the .5 in Resources.min() should be made configurable. # Add comments to capture the logic in ShareComputer/ComputeResourceShares and FS#isStarvedForDesiredFairShare > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13686301#comment-13686301 ] Sandy Ryza commented on YARN-736: - Uploaded a patch that does 1 and 3. For 2, I realized that there are a few more changes we will need to make to do preemption based on multiple resources, so I would prefer to make the change in a separate patch. What do you think? > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13686312#comment-13686312 ] Hadoop QA commented on YARN-736: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588278/YARN-736-2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1319//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1319//console This message is automatically generated. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688189#comment-13688189 ] Karthik Kambatla commented on YARN-736: --- Thanks Sandy. Yes, it should be okay to handle #2 in a separate JIRA. Nits: # AppSchedulable#getMaxShare(): instead of creating a resource with max values every time, we should create a Resource object on the first call and return it from then on. # There seem to be several constructors for FakeSchedulable. Can we git rid of any that are not used anywhere. Ignore if all of them are used. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688204#comment-13688204 ] Sandy Ryza commented on YARN-736: - All the FakeSchedulable constructors appear to be used. Uploaded a patch in which AppSchedulable#getMaxShare does not create a new Resource each time. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688244#comment-13688244 ] Hadoop QA commented on YARN-736: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588643/YARN-736-3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1351//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1351//console This message is automatically generated. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688254#comment-13688254 ] Karthik Kambatla commented on YARN-736: --- Having an UNBOUNDED resource in Resources can be handy, even though having a local copy in AppSchedulable should be enough for this JIRA. +1 to the yarn-736-3.patch. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13690697#comment-13690697 ] Alejandro Abdelnur commented on YARN-736: - looks good, a couple of things though: * DominantResourceFairnessPolicy#computeShares(), if Schedulable sets an empty fairshare Resource on initialization, the loop setting an empty resource before the loop computing the fair share is not needed. * the documentation changes should have ENTERs at 80 chars max > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13690859#comment-13690859 ] Sandy Ryza commented on YARN-736: - Uploaded a patch that addresses Alejandro's comments > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13690887#comment-13690887 ] Hadoop QA commented on YARN-736: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12589181/YARN-736-4.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site: org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1379//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1379//console This message is automatically generated. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13690898#comment-13690898 ] Sandy Ryza commented on YARN-736: - The test failure appears unrelated. I get also get it without the patch. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13692212#comment-13692212 ] Alejandro Abdelnur commented on YARN-736: - +1 > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13692241#comment-13692241 ] Hudson commented on YARN-736: - Integrated in Hadoop-trunk-Commit #4005 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4005/]) YARN-736. Add a multi-resource fair sharing metric. (sandyr via tucu) (Revision 1496153) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1496153 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/ComputeFairShares.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FakeSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestComputeFairShares.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/TestDominantResourceFairnessPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.2.0 > > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13692919#comment-13692919 ] Hudson commented on YARN-736: - Integrated in Hadoop-Yarn-trunk #251 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/251/]) YARN-736. Add a multi-resource fair sharing metric. (sandyr via tucu) (Revision 1496153) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1496153 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/ComputeFairShares.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FakeSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestComputeFairShares.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/TestDominantResourceFairnessPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.2.0 > > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13693009#comment-13693009 ] Hudson commented on YARN-736: - Integrated in Hadoop-Hdfs-trunk #1441 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1441/]) YARN-736. Add a multi-resource fair sharing metric. (sandyr via tucu) (Revision 1496153) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1496153 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/ComputeFairShares.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FakeSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestComputeFairShares.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/TestDominantResourceFairnessPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.2.0 > > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13693040#comment-13693040 ] Hudson commented on YARN-736: - Integrated in Hadoop-Mapreduce-trunk #1468 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1468/]) YARN-736. Add a multi-resource fair sharing metric. (sandyr via tucu) (Revision 1496153) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1496153 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/ComputeFairShares.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FakeSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestComputeFairShares.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/TestDominantResourceFairnessPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.2.0 > > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705324#comment-13705324 ] Hudson commented on YARN-736: - Integrated in Hadoop-trunk-Commit #4064 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4064/]) updating CHANGES.txt after committing MAPREDUCE-5333,HADOOP-9661,HADOOP-9355,HADOOP-9673,HADOOP-9414,HADOOP-9416,HDFS-4797,YARN-866,YARN-736,YARN-883 to 2.1-beta branch (Revision 1502075) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1502075 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705694#comment-13705694 ] Hudson commented on YARN-736: - Integrated in Hadoop-Yarn-trunk #267 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/267/]) updating CHANGES.txt after committing MAPREDUCE-5333,HADOOP-9661,HADOOP-9355,HADOOP-9673,HADOOP-9414,HADOOP-9416,HDFS-4797,YARN-866,YARN-736,YARN-883 to 2.1-beta branch (Revision 1502075) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1502075 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705794#comment-13705794 ] Hudson commented on YARN-736: - Integrated in Hadoop-Hdfs-trunk #1457 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1457/]) updating CHANGES.txt after committing MAPREDUCE-5333,HADOOP-9661,HADOOP-9355,HADOOP-9673,HADOOP-9414,HADOOP-9416,HDFS-4797,YARN-866,YARN-736,YARN-883 to 2.1-beta branch (Revision 1502075) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1502075 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705855#comment-13705855 ] Hudson commented on YARN-736: - Integrated in Hadoop-Mapreduce-trunk #1484 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1484/]) updating CHANGES.txt after committing MAPREDUCE-5333,HADOOP-9661,HADOOP-9355,HADOOP-9673,HADOOP-9414,HADOOP-9416,HDFS-4797,YARN-866,YARN-736,YARN-883 to 2.1-beta branch (Revision 1502075) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1502075 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira