subject:"\[jira\] \[Commented\] \(YARN\-311\) Dynamic node resource configuration\: core scheduler changes"

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-12-16 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849230#comment-13849230
]

Junping Du commented on YARN-311:
-

Thanks Bikas for comments! Please see my reply:
bq. Sorry for coming in so late on this.
No problem. Given this patch is already commited, I will address them in
separated JIRAs. Make sense?
bq. Some javadoc on ResourceOption would be helpful. Explanation of the context
of dynamic resource changes on a node and resulting over-commitment of
resources would be good. ResourceOption as a name did not make it clear to me
the nature/use of the object.
Yes. Vinod had similar comments in YARN-312. Will file a JIRA [let's call it
JIRA-1] to handle naming issue as well as more document on over-commitment
cases.
bq. Would it be less error prone if we compared the total size of
schedulernode and rmnode instead of the difference in their current available
capacity? Also, in the update to the node. Why are we updating only the
availableResource and skipping totalresource? Total resource is used during
scheduling decisions.
That's nice catch! When this patch was developed, there is actually no total
capacity in scheduler node (it is get involved in YARN-957, checked in several
months ago) so comparing RMNode's total resource with schedulerNode'S (used
resource + available resource) is the only choice at that time. I will file
another JIRA [let's call it JIRA-2] to address this.
bq. The current impl of addTo will work for both +ve and -ve deltas but given
that there are addTo and subtractFrom methods, its not clear to me if that is a
coincidence or not. Ideally there should have been one update method that by
definition should handle +ve and -ve updates.
This is to handle some complicated cases. i.e. increase CPU resource while
decrease Memory resource. Shall we use addTo or subtractFrom? Keep it simple
and stupid like here may make more sense?
bq. Since changing the resource on a node would be an admin/service operation,
why are we adding resourceOption to the rmnode and setting it in
registernodemanager? Similarly, why are we trying to update the node on every
heartbeat. I was expecting that whenever the node resource would be updated
then an event would be sent to the scheduler. Upon receiving the event, the
scheduler would make a one time update of the internal book-keeping objects.
I think RMNode should be updated as resource view should be consistent across
the system or it will cause system/user get confused. That's a nice suggestion
to have resource update event on schedulerNode. Vinod has similar comments on
RMNode update event, and I will address these comments with filing [JIRA-3].
bq. Again, I apologize for coming in so late on this jira.
Never mind. Good comments are never too late. Thanks for this, Bikas!

Dynamic node resource configuration: core scheduler changes
---

Attachments: YARN-311-v1.patch, YARN-311-v10.patch,
YARN-311-v11.patch, YARN-311-v12.patch, YARN-311-v12b.patch,
YARN-311-v13.patch, YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch,
YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch,
YARN-311-v9.patch

As the first step, we go for resource change on RM side and expose admin APIs
(admin protocol, CLI, REST and JMX API) later. In this jira, we will only
contain changes in scheduler.
The flow to update node's resource and awareness in resource scheduling is:
1. Resource update is through admin API to RM and take effect on RMNodeImpl.
2. When next NM heartbeat for updating status comes, the RMNode's resource
change will be aware and the delta resource is added to schedulerNode's
availableResource before actual scheduling happens.
3. Scheduler do resource allocation according to new availableResource in
SchedulerNode.
For more design details, please refer proposal and discussions in parent
JIRA: YARN-291.

--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-12-16 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849263#comment-13849263
]

Junping Du commented on YARN-311:
-

Filed YARN-1508 for addressing rename and documentation issue. Combine rest of
comments for addressing together in YARN-1506.

Dynamic node resource configuration: core scheduler changes
---

--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-12-16 Thread Bikas Saha (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849278#comment-13849278
]

Bikas Saha commented on YARN-311:
-

Sounds good!

Dynamic node resource configuration: core scheduler changes
---

--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-12-15 Thread Bikas Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848615#comment-13848615
 ] 

Bikas Saha commented on YARN-311:
-

Sorry for coming in so late on this.

Some javadoc on ResourceOption would be helpful. Explanation of the context of 
dynamic resource changes on a node and resulting over-commitment of resources 
would be good.

ResourceOption as a name did not make it clear to me the nature/use of the 
object.

Would it be less error prone if we compared the total size of schedulernode and 
rmnode instead of the difference in their current available capacity? 
{code}
+Resource oldAvailableResource = node.getAvailableResource();
+Resource newAvailableResource = Resources.subtract(
+rmNode.getTotalCapability(), node.getUsedResource());
{code}

Also, in the update to the node. Why are we updating only the availableResource 
and skipping totalresource? Total resource is used during scheduling decisions.
{code}
+  @Override
+  public synchronized void applyDeltaOnAvailableResource(Resource 
deltaResource) {
+// we can only adjust available resource if total resource is changed.
+Resources.addTo(this.availableResource, deltaResource);
+  }
{code}
The current impl of addTo will work for both +ve and -ve deltas but given that 
there are addTo and subtractFrom methods, its not clear to me if that is a 
coincidence or not. Ideally there should have been one update method that by 
definition should handle +ve and -ve updates.

Since changing the resource on a node would be an admin/service operation, why 
are we adding resourceOption to the rmnode and setting it in 
registernodemanager?
{code}
 RMNode rmNode = new RMNodeImpl(nodeId, rmContext, host, cmPort, httpPort,
-resolve(host), capability, nodeManagerVersion);
+resolve(host), ResourceOption.newInstance(capability, 
RMNode.OVER_COMMIT_TIMEOUT_MILLIS_DEFAULT),
+nodeManagerVersion);
{code}

Similarly, why are we trying to update the node on every heartbeat. I was 
expecting that whenever the node resource would be updated then an event would 
be sent to the scheduler. Upon receiving the event, the scheduler would make a 
one time update of the internal book-keeping objects.
{code}
+// Update resource if any change
+SchedulerUtils.updateResourceIfChanged(node, nm, clusterResource, LOG);
{code}

Again, I apologize for coming in so late on this jira.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Fix For: 2.4.0

 Attachments: YARN-311-v1.patch, YARN-311-v10.patch, 
 YARN-311-v11.patch, YARN-311-v12.patch, YARN-311-v12b.patch, 
 YARN-311-v13.patch, YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, 
 YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, 
 YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, 
 YARN-311-v9.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API) later. In this jira, we will only 
 contain changes in scheduler. 
 The flow to update node's resource and awareness in resource scheduling is: 
 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
 2. When next NM heartbeat for updating status comes, the RMNode's resource 
 change will be aware and the delta resource is added to schedulerNode's 
 availableResource before actual scheduling happens.
 3. Scheduler do resource allocation according to new availableResource in 
 SchedulerNode.
 For more design details, please refer proposal and discussions in parent 
 JIRA: YARN-291.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-11-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814787#comment-13814787
 ] 

Hudson commented on YARN-311:
-

SUCCESS: Integrated in Hadoop-Yarn-trunk #384 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/384/])
YARN-311. RM/scheduler support for dynamic resource configuration. (Junping Du 
via llu) (llu: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1539134)
* 
/hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java
* 
/hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceOption.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceOptionPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerNode.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java


 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Fix For: 2.3.0

 Attachments: YARN-311-v1.patch, YARN-311-v10.patch, 
 YARN-311-v11.patch, YARN-311-v12.patch, YARN-311-v12b.patch, 
 YARN-311-v13.patch, YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, 
 YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, 
 YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, 
 YARN-311-v9.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API) later. In this jira, we will only 
 contain changes in scheduler. 
 The flow to update node's resource and awareness in resource scheduling is: 
 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
 2. When next NM heartbeat for updating

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-11-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814859#comment-13814859
 ] 

Hudson commented on YARN-311:
-

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1601 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1601/])
YARN-311. RM/scheduler support for dynamic resource configuration. (Junping Du 
via llu) (llu: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1539134)
* 
/hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java
* 
/hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceOption.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceOptionPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerNode.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java


 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Fix For: 2.3.0

 Attachments: YARN-311-v1.patch, YARN-311-v10.patch, 
 YARN-311-v11.patch, YARN-311-v12.patch, YARN-311-v12b.patch, 
 YARN-311-v13.patch, YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, 
 YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, 
 YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, 
 YARN-311-v9.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API) later. In this jira, we will only 
 contain changes in scheduler. 
 The flow to update node's resource and awareness in resource scheduling is: 
 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
 2. When next NM heartbeat for

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-11-05 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814478#comment-13814478
]

Junping Du commented on YARN-311:
-

Thanks Luke for review!

Dynamic node resource configuration: core scheduler changes
---

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-11-04 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813139#comment-13813139
]

Luke Lu commented on YARN-311:
--

[~djp]: Unfortunately YARN-1343 got in before I tried to merge the patch. Now
the patch won't compile due to the old RMNodeImpl ctor usage in
TestRMNodeTransition. Can you rebase the patch?

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v1.patch, YARN-311-v10.patch,
YARN-311-v11.patch, YARN-311-v12.patch, YARN-311-v12b.patch,
YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-11-04 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813547#comment-13813547
]

Junping Du commented on YARN-311:
-

Sure. Will update patch soon. Thx!

Dynamic node resource configuration: core scheduler changes
---

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-11-04 Thread Junping Du (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813574#comment-13813574
 ] 

Junping Du commented on YARN-311:
-

Updated in v13 patch.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v1.patch, YARN-311-v10.patch, 
 YARN-311-v11.patch, YARN-311-v12.patch, YARN-311-v12b.patch, 
 YARN-311-v13.patch, YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, 
 YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, 
 YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, 
 YARN-311-v9.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API) later. In this jira, we will only 
 contain changes in scheduler. 
 The flow to update node's resource and awareness in resource scheduling is: 
 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
 2. When next NM heartbeat for updating status comes, the RMNode's resource 
 change will be aware and the delta resource is added to schedulerNode's 
 availableResource before actual scheduling happens.
 3. Scheduler do resource allocation according to new availableResource in 
 SchedulerNode.
 For more design details, please refer proposal and discussions in parent 
 JIRA: YARN-291.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-11-04 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813661#comment-13813661
 ] 

Hadoop QA commented on YARN-311:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12612092/YARN-311-v13.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-sls hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/2367//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2367//console

This message is automatically generated.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v1.patch, YARN-311-v10.patch, 
 YARN-311-v11.patch, YARN-311-v12.patch, YARN-311-v12b.patch, 
 YARN-311-v13.patch, YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, 
 YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, 
 YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, 
 YARN-311-v9.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API) later. In this jira, we will only 
 contain changes in scheduler. 
 The flow to update node's resource and awareness in resource scheduling is: 
 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
 2. When next NM heartbeat for updating status comes, the RMNode's resource 
 change will be aware and the delta resource is added to schedulerNode's 
 availableResource before actual scheduling happens.
 3. Scheduler do resource allocation according to new availableResource in 
 SchedulerNode.
 For more design details, please refer proposal and discussions in parent 
 JIRA: YARN-291.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-11-01 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811714#comment-13811714
]

Luke Lu commented on YARN-311:
--

v12 patch lgtm. +1. Will commit soon.

Dynamic node resource configuration: core scheduler changes
---

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-30 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809781#comment-13809781
]

Hadoop QA commented on YARN-311:

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611210/YARN-311-v12.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.

{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2324//console

This message is automatically generated.

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v10.patch, YARN-311-v11.patch,
YARN-311-v12.patch, YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch,
YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-30 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809797#comment-13809797
]

Junping Du commented on YARN-311:
-

The log didn't show it is an build failure (it works well locally), so the
jenkins failure above is not related with patch but an accident. Rename it to
v12b (exactly the same) patch and submit it again.

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v10.patch, YARN-311-v11.patch,
YARN-311-v12b.patch, YARN-311-v12.patch, YARN-311-v1.patch,
YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-30 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809820#comment-13809820
 ] 

Hadoop QA commented on YARN-311:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12611218/YARN-311-v12b.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-sls hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/2325//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2325//console

This message is automatically generated.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v10.patch, YARN-311-v11.patch, 
 YARN-311-v12b.patch, YARN-311-v12.patch, YARN-311-v1.patch, 
 YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch, 
 YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch, 
 YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API) later. In this jira, we will only 
 contain changes in scheduler. 
 The flow to update node's resource and awareness in resource scheduling is: 
 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
 2. When next NM heartbeat for updating status comes, the RMNode's resource 
 change will be aware and the delta resource is added to schedulerNode's 
 availableResource before actual scheduling happens.
 3. Scheduler do resource allocation according to new availableResource in 
 SchedulerNode.
 For more design details, please refer proposal and discussions in parent 
 JIRA: YARN-291.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-29 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13807773#comment-13807773
]

Luke Lu commented on YARN-311:
--

Junping: your [previous
comment|https://issues.apache.org/jira/browse/YARN-311?focusedCommentId=13804690page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13804690]
actually convinced me that being able independently set overCommitTimeout is a
reasonable use case and that the condition (resource1, overCommitTimeout2) is a
reasonable behavior (the later timeout takes effect), given the benefits.

I was concerned that
# Requiring RMNode lock at usage site is brittle and error-prone.
# Extra RMNode lock per heartbeat could be expensive.

We can even preserve the consistency without locks by using a ResourceConfig
object which holds the resource and timeout together.

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v10.patch, YARN-311-v1.patch,
YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-29 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808027#comment-13808027
]

Junping Du commented on YARN-311:
-

Hi Luke, Thanks for comments.
bq. 1.Requiring RMNode lock at usage site is brittle and error-prone.
Agree. How about we put write/read lock inside method like other methods
currently in RMNodeImpl?
bq. 2.Extra RMNode lock per heartbeat could be expensive.
With my answer to 1, it is just read lock and it is already to be granted in
heartbeat behavior, i.e., checking if it is fresh heartbeat.
bq. We can even preserve the consistency without locks by using a
ResourceConfig object which holds the resource and timeout together.
IMO, It involve more complexity for creating a new object as we should put it
over wire so new protocol buf object is created as well...
How about we just keep it simple with read/write lock just like other methods
in RMNodeImpl today?

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v10.patch, YARN-311-v1.patch,
YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-29 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808130#comment-13808130
]

Luke Lu commented on YARN-311:
--

IMO, using an immutable ResourceOption object is cleaner than mess with
read/write locks, which has high constant overhead even when the lock is not
contended.

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v10.patch, YARN-311-v1.patch,
YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-29 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808353#comment-13808353
]

Junping Du commented on YARN-311:
-

The problem is even we involve ResourceOption, it is still not immutable (in
thread-safe prospective) and non-atomic. Look at the previous case we set
overCommitTimeout only, the implementation for setOverCommitTimeout() API is
still read-and-set (read resource from current ResourceOption and set it back
to new ResourceOption) which is not thread-safe without synchronization.
If we look at existing implementation for APIs in RMNodeImpl now, most of them
are protected by read/write lock. I think case here should be keep consistent
with them or fix them all in separated JIRA. What do you think?
BTW, the major lock overhead in YARN scheduler to me is locking entire
scheduler when doing heartbeat for each nodes. Some heavy work like launch
containers are even unnecessary included in synchronized block. I think we
should fix that to increase scheduling throughput (some other effort address it
in different ways, i.e. AttemptScheduling in FSScheduler). Thoughts?

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v10.patch, YARN-311-v1.patch,
YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-29 Thread Luke Lu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808557#comment-13808557
 ] 

Luke Lu commented on YARN-311:
--

With ResourceOption, you don't need a separate get/setOverCommitTimeout method. 
You only need one pair of methods for all future resource options (prioritize 
certain containers (last to kill) in over commit situations etc.)
{code}
private volatile ResourceOption totalCapacity;
...
void setTotalCapacity(ResourceOption ro);
void ResourceOption getTotalCapacity();
{code}

{{setTotalCapacity}} is now idempotent. Seems simpler to me.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v10.patch, YARN-311-v1.patch, 
 YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch, 
 YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch, 
 YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API) later. In this jira, we will only 
 contain changes in scheduler. 
 The flow to update node's resource and awareness in resource scheduling is: 
 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
 2. When next NM heartbeat for updating status comes, the RMNode's resource 
 change will be aware and the delta resource is added to schedulerNode's 
 availableResource before actual scheduling happens.
 3. Scheduler do resource allocation according to new availableResource in 
 SchedulerNode.
 For more design details, please refer proposal and discussions in parent 
 JIRA: YARN-291.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-29 Thread Junping Du (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808577#comment-13808577
 ] 

Junping Du commented on YARN-311:
-

Luke, thanks for comments! That's also doable, but I have two concerns below:
1. API of getTotalCapacity() are consumed a lot of places. Do you think it is 
too overkill to make this change?
2. The new API didn't explicitly express it set OverCommitTimeout as long as 
capacity.
If we don't care idempotent for API in this case, we might have something like 
below:
{code}
private volatile ResourceOption resourceOption;
void setResourceOption(ResourceOption ro);
void Resource getTotalCapacity() {
return resourceOption.getResource();
};
{code}
How it sounds to you?

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v10.patch, YARN-311-v1.patch, 
 YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch, 
 YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch, 
 YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API) later. In this jira, we will only 
 contain changes in scheduler. 
 The flow to update node's resource and awareness in resource scheduling is: 
 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
 2. When next NM heartbeat for updating status comes, the RMNode's resource 
 change will be aware and the delta resource is added to schedulerNode's 
 availableResource before actual scheduling happens.
 3. Scheduler do resource allocation according to new availableResource in 
 SchedulerNode.
 For more design details, please refer proposal and discussions in parent 
 JIRA: YARN-291.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-29 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808600#comment-13808600
]

Luke Lu commented on YARN-311:
--

bq. How it sounds to you?

Sounds good to me :), assuming you'll add getResourceOption as well.

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v10.patch, YARN-311-v1.patch,
YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-29 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808757#comment-13808757
 ] 

Hadoop QA commented on YARN-311:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12610986/YARN-311-v11.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-sls hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/2314//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2314//console

This message is automatically generated.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v10.patch, YARN-311-v11.patch, 
 YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, 
 YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, 
 YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, 
 YARN-311-v9.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API) later. In this jira, we will only 
 contain changes in scheduler. 
 The flow to update node's resource and awareness in resource scheduling is: 
 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
 2. When next NM heartbeat for updating status comes, the RMNode's resource 
 change will be aware and the delta resource is added to schedulerNode's 
 availableResource before actual scheduling happens.
 3. Scheduler do resource allocation according to new availableResource in 
 SchedulerNode.
 For more design details, please refer proposal and discussions in parent 
 JIRA: YARN-291.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-28 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13807484#comment-13807484
]

Luke Lu commented on YARN-311:
--

I share the same concern with Bikas with the RMNode lock inside the static util
method. It doesn't appear to be necessary, as long as both
RMNodeImpl#totalCapacity and overCommitTimeout is volatile (the latter is not
in the patch), which are simply accessed via get/set (no dependent op like
increment). The lock could lead to deadlocks latter (after new code/refactor)
even if it appears to be fine now.

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v10.patch, YARN-311-v1.patch,
YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-28 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13807505#comment-13807505
]

Junping Du commented on YARN-311:
-

Luke, the volatile is not enough as you cannot set capacity and
overCommitTimeout in an atomic way (even in api with two parameters). Take
following example (API in you suggested way to have two parameters in set)
{code}
void setTotalCapacity(int resource, int overCommitTimeout) {
this.resource = resource;
this.overCommitTimeout = overCommitTimeout;
}
{code}
Thread 1 finished update it to (resource1 and overCommitTimeout1), thread 2
want to update it to (resource2, overCommitTimeout2) but when it just update
resource2, another thread is reading and may get a mixed result (resource1,
overCommitTimeout2) which should never appear in user's operation. I think
putting a synchronized tag is worth it and it equals exactly the same as put
synchronized on method (even in deadlock prospective). Thoughts?

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v10.patch, YARN-311-v1.patch,
YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-25 Thread Bikas Saha (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13805522#comment-13805522
]

Bikas Saha commented on YARN-311:
-

Can we please double check and assure ourselves that this is deadlock free.
{code}
+// Update resource if any change
+synchronized(nm) {
+ SchedulerUtils.updateResourceIfChanged(node, nm, clusterResource, LOG);
+}
{code}

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v10.patch, YARN-311-v1.patch,
YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-25 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1380#comment-1380
]

Junping Du commented on YARN-311:
-

Thanks for comments, Bikas! The synchronization here is to make sure the read
of nm (rmNode) resource is thread-safe while another thread do write
(nm.setTotalCapacity()) triggered in AdminService (an implementation of
RMAdminProtocol). Given SchedulerUtils.updateResourceIfChanged() itself is
lock-free and nm.setTotalCapacity() is also lock-free, it is easily to execute
through when getting nm synchronization lock, so it is deadlock free. Does it
make sense?

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v10.patch, YARN-311-v1.patch,
YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-24 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804038#comment-13804038
]

Luke Lu commented on YARN-311:
--

v9 patch lgtm. +1. For the record, we've discussed the changes with Vinod et
al offline. The consensus is that change is reasonable, low risk and fulfill a
useful use case (resource arbitration between different resource pools managed
by YARN or other (versions of) systems including but not limited to Hadoop 1.x
on the same physical hardware).

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch,
YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-24 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804320#comment-13804320
]

Luke Lu commented on YARN-311:
--

[~djp]: Upon second thought, I think we should improve the setCapacity API to
include scheduler hints/options (to deal with running containers when node
capacity is set lower than the used capacity on the node) in this JIRA, in case
it gets lost through the crack later.

I recall the options as FINISH_RUNNING, TIMEOUT_RUNNING, TERMINATE_RUNNING.

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch,
YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-24 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804335#comment-13804335
]

Luke Lu commented on YARN-311:
--

We can probably infer the option by an overcommit timeout argument 0: keep
running; == 0: terminate; 0 timeout:
{code}
void setTotalCapacity(Resource resource, int overCommitTimeoutMillis);
{code}

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch,
YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-24 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804465#comment-13804465
]

Junping Du commented on YARN-311:
-

Thanks Luke for review and comments! Yes. We discussed different
options/policies to handle resource overcommitment case. It is reasonable to
update API of setCapacity() to above for flushing resource change to allocated
containers immediately or delay. Will make API change here, but only implement
a default policy (keep container running until end). Other policies/options
will be addressed in YARN-999. Does it make sense?

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch,
YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-24 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804532#comment-13804532
]

Luke Lu commented on YARN-311:
--

bq. Other policies/options will be addressed in YARN-999

Agreed. That was the consensus (default policy is to keep containers running
until they finish) as well.

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch,
YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-24 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804690#comment-13804690
]

Junping Du commented on YARN-311:
-

Luke, instead of void setTotalCapacity(Resource resource, int
overCommitTimeoutMillis); I add separated API of setOverCommitTimeoutMillis()
to provide flexibility of update overCommitTimeoutMillis only(e.g. admin lose
patient to wait) and make API looks more symmetric. The synchronized could
happen in AdminService (RMAdminProtocol service provider). Let me know if you
have any concern on this way. Thx!

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v10.patch, YARN-311-v1.patch,
YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-24 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804719#comment-13804719
 ] 

Hadoop QA commented on YARN-311:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12610153/YARN-311-v10.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-sls 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/2278//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2278//console

This message is automatically generated.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v10.patch, YARN-311-v1.patch, 
 YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch, 
 YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch, 
 YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API) later. In this jira, we will only 
 contain changes in scheduler. 
 The flow to update node's resource and awareness in resource scheduling is: 
 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
 2. When next NM heartbeat for updating status comes, the RMNode's resource 
 change will be aware and the delta resource is added to schedulerNode's 
 availableResource before actual scheduling happens.
 3. Scheduler do resource allocation according to new availableResource in 
 SchedulerNode.
 For more design details, please refer proposal and discussions in parent 
 JIRA: YARN-291.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-23 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803534#comment-13803534
]

Junping Du commented on YARN-311:
-

Hi [~vinodkv] and [~vicaya], would you help to review it again? Thx!

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch,
YARN-311-v9.patch

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13797463#comment-13797463
 ] 

Hadoop QA commented on YARN-311:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12608826/YARN-311-v8.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-sls 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/2195//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2195//console

This message is automatically generated.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, 
 YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, 
 YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API) later. In this jira, we will only 
 contain changes in scheduler. 
 The flow to update node's resource and awareness in resource scheduling is: 
 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
 2. When next NM heartbeat for updating status comes, the RMNode's resource 
 change will be aware and the delta resource is added to schedulerNode's 
 availableResource before actual scheduling happens.
 3. Scheduler do resource allocation according to new availableResource in 
 SchedulerNode.
 For more design details, please refer proposal and discussions in parent 
 JIRA: YARN-291.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-24 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777171#comment-13777171
 ] 

Hadoop QA commented on YARN-311:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12604962/YARN-311-v7.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/2012//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2012//console

This message is automatically generated.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, 
 YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, 
 YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API) later. In this jira, we will only 
 contain changes in scheduler. 
 The flow to update node's resource and awareness in resource scheduling is: 
 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
 2. When next NM heartbeat for updating status comes, the RMNode's resource 
 change will be aware and the delta resource is added to schedulerNode's 
 availableResource before actual scheduling happens.
 3. Scheduler do resource allocation according to new availableResource in 
 SchedulerNode.
 For more design details, please refer proposal and discussions in parent 
 JIRA: YARN-291.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-13 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766141#comment-13766141
]

Junping Du commented on YARN-311:
-

Thanks Luke for review and comments!

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-13 Thread Arun C Murthy (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766787#comment-13766787
]

Arun C Murthy commented on YARN-311:

Can I get a few more days to review this? Thanks.

Also, let's put this in 2.3.0 (not 2.1.1). Thanks.

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-13 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767095#comment-13767095
]

Junping Du commented on YARN-311:
-

Sure. Thanks for review. Arun!

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-12 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13765760#comment-13765760
]

Luke Lu commented on YARN-311:
--

I think the v6.2 patch looks reasonable. If there is no further objections, I
plan to commit it over the weekend.

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-05 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13759722#comment-13759722
]

Junping Du commented on YARN-311:
-

[~tucu00], would you help to review it again? Thx!

Dynamic node resource configuration: core scheduler changes
---

Key: YARN-311
URL: https://issues.apache.org/jira/browse/YARN-311
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
YARN-311-v6.2.patch, YARN-311-v6.patch

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-04 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757710#comment-13757710
]

Junping Du commented on YARN-311:
-

Thanks for review! [~tucu00]
bq. If we make totalCapability volatile then we don't need to use a read/write
lock.
Yes. Make it as volatile sounds better as locking whole object is not
necessary. Will update patch soon.
bq. Does this mean that if the node is restarted we lose the capacity
correction done thru the RM admin API for that node?
Yes and No. It is correct that this patch will not guarantee capacity
correction persist through NM restart but the other jira (YARN-998) under the
same umbrella will address this persistent issue. My current thinking is we can
cache a mapping in RM as NodeID - updatedResource which is updated by RM admin
call and NM restart heartbeat will try to find if new resource there before
registering node's resource. Does that make sense to you? May be we can discuss
more options in YARN-998.

Dynamic node resource configuration: core scheduler changes
---

As the first step, we go for resource change on RM side and expose admin APIs
(admin protocol, CLI, REST and JMX API). In this jira, we will only contain
changes in scheduler.
For design details, please refer proposal and discussions in parent JIRA:
YARN-291.

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-04 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757791#comment-13757791
 ] 

Hadoop QA commented on YARN-311:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12601377/YARN-311-v6.2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1829//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1829//console

This message is automatically generated.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, 
 YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, 
 YARN-311-v6.2.patch, YARN-311-v6.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API). In this jira, we will only contain 
 changes in scheduler.
 For design details, please refer proposal and discussions in parent JIRA: 
 YARN-291.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-03 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13756599#comment-13756599
 ] 

Hadoop QA commented on YARN-311:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12601148/YARN-311-v5.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1819//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1819//console

This message is automatically generated.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, 
 YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API). In this jira, we will only contain 
 changes in scheduler.
 For design details, please refer proposal and discussions in parent JIRA: 
 YARN-291.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-03 Thread Alejandro Abdelnur (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13756614#comment-13756614
]

Alejandro Abdelnur commented on YARN-311:
-

* the patch as a few false changes (formatting)
* RMNodeImpl#setTotalCapability() is modifying the existing resource instead
assigning the received one? is that intentional? If so, the locking is kind of
pointless as holders of a RMNodeImpl reference will see an
inconsistent/non-locked value.
* the SchedulerNode#updateAvailableResource() method name is not clear with
respect of a delta correction happening (which is clear with the formal
parameter name). Either we should make the method name more obvious of we
should set the new value. It is kind of confusing that for RMNodeImpl the patch
uses the full new capacity while for SchedulerNode the patch uses the delta
change; could we use full or delta values in both?

Also, in this patch it seems the change will be triggered from a NM heartbeat.
Under witch situation a NM would do such change without being restarted? If
seems to me the change should come from an admin API to the RM, this would be
set as a correction in the RMNodeImpl, and the RMNodeImpl would use the correct
valued as the total instead of the total reported by the NM. Am I missing
something?

Dynamic node resource configuration: core scheduler changes
---

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-03 Thread Junping Du (JIRA)

[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13756634#comment-13756634
]

Junping Du commented on YARN-311:
-

Hi, [~tucu00] Thanks for comments!
bq. RMNodeImpl#setTotalCapability() is modifying the existing resource instead
assigning the received one? is that intentional? If so, the locking is kind of
pointless as holders of a RMNodeImpl reference will see an
inconsistent/non-locked value.
You are right. We should set resource directly rather than current way (my
previous intention is to check null and get rid of NPE).
bq. the SchedulerNode#updateAvailableResource() method name is not clear with
respect of a delta correction happening (which is clear with the formal
parameter name). Either we should make the method name more obvious of we
should set the new value. It is kind of confusing that for RMNodeImpl the patch
uses the full new capacity while for SchedulerNode the patch uses the delta
change; could we use full or delta values in both?
It is because RMNodeImpl is tracking totalCapacity while SchedulerNode is
tracking availableResource and usedResource. So we will set new resource
directly in RMNodeImpl and update the delta resource to SchedulerNode's
availableResource to make sure availableResource + usedResource = newResource,
does that make sense?
bq. Also, in this patch it seems the change will be triggered from a NM
heartbeat. Under witch situation a NM would do such change without being
restarted? If seems to me the change should come from an admin API to the RM,
this would be set as a correction in the RMNodeImpl, and the RMNodeImpl would
use the correct valued as the total instead of the total reported by the NM. Am
I missing something?
You previous thinking is correct. the current flow is still: resource update is
through admin API to RM and take effect on RMNodeImpl, when NM heartbeat (just
status update, but not register), the change will be aware there and pass to
schedulerNode before scheduling.

Dynamic node resource configuration: core scheduler changes
---

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-03 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13756674#comment-13756674
 ] 

Hadoop QA commented on YARN-311:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12601167/YARN-311-v6.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1820//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1820//console

This message is automatically generated.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, 
 YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API). In this jira, we will only contain 
 changes in scheduler.
 For design details, please refer proposal and discussions in parent JIRA: 
 YARN-291.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-09-03 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757256#comment-13757256
 ] 

Hadoop QA commented on YARN-311:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12601265/YARN-311-v6.1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1825//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1825//console

This message is automatically generated.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, 
 YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, 
 YARN-311-v6.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API). In this jira, we will only contain 
 changes in scheduler.
 For design details, please refer proposal and discussions in parent JIRA: 
 YARN-291.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

2013-06-13 Thread Junping Du (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683044#comment-13683044
 ] 

Junping Du commented on YARN-311:
-

According to discussion above, let's split JMX api into a separated JIRA, so we 
only do slightly core scheduler changes here.

 Dynamic node resource configuration: core scheduler changes
 ---

 Key: YARN-311
 URL: https://issues.apache.org/jira/browse/YARN-311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
 Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, 
 YARN-311-v4.patch, YARN-311-v4.patch


 As the first step, we go for resource change on RM side and expose admin APIs 
 (admin protocol, CLI, REST and JMX API). In this jira, we will only contain 
 changes in scheduler.
 For design details, please refer proposal and discussions in parent JIRA: 
 YARN-291.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

50 matches

Mail list logo