[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[ https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167283#comment-14167283 ] Vinod Kumar Vavilapalli commented on YARN-2494: --- Looks good now, +1. Checking this in. [YARN-796] Node label manager API and storage implementations - Key: YARN-2494 URL: https://issues.apache.org/jira/browse/YARN-2494 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2494.20141009-1.patch, YARN-2494.20141009-2.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch This JIRA includes APIs and storage implementations of node label manager, NodeLabelManager is an abstract class used to manage labels of nodes in the cluster, it has APIs to query/modify - Nodes according to given label - Labels according to given hostname - Add/remove labels - Set labels of nodes in the cluster - Persist/recover changes of labels/labels-on-nodes to/from storage And it has two implementations to store modifications - Memory based storage: It will not persist changes, so all labels will be lost when RM restart - FileSystem based storage: It will persist/recover to/from FileSystem (like HDFS), and all labels and labels-on-nodes will be recovered upon RM restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[ https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167296#comment-14167296 ] Hudson commented on YARN-2494: -- FAILURE: Integrated in Hadoop-trunk-Commit #6237 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6237/]) YARN-2494. Added NodeLabels Manager internal API and implementation. Contributed by Wangda Tan. (vinodkv: rev db7f1653198b950e89567c06898d64f6b930a0ee) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/DummyCommonNodeLabelsManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/DummyRMNodeLabelsManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/event/NodeLabelsStoreEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestCommonNodeLabelsManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabelsStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/event/UpdateNodeToLabelsMappingsEvent.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/NodeLabelTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/event/StoreNewClusterNodeLabels.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/event/RemoveClusterNodeLabels.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/event/NodeLabelsStoreEventType.java [YARN-796] Node label manager API and storage implementations - Key: YARN-2494 URL: https://issues.apache.org/jira/browse/YARN-2494 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.6.0 Attachments: YARN-2494.20141009-1.patch, YARN-2494.20141009-2.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch This JIRA includes APIs and storage implementations of node label manager, NodeLabelManager is an abstract class used to manage labels of nodes in the cluster, it has APIs to query/modify - Nodes according to given label - Labels according to given hostname - Add/remove labels - Set labels of nodes in the cluster - Persist/recover changes of labels/labels-on-nodes to/from storage And it has two implementations to store modifications - Memory based storage: It will not persist changes, so all labels will be lost when RM restart - FileSystem based storage: It will persist/recover to/from FileSystem (like HDFS), and all labels and labels-on-nodes will be recovered upon RM restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14165887#comment-14165887
]
Vinod Kumar Vavilapalli commented on YARN-2494:
---
This is very close. Final set of comments:
- The pattern {quote}^[0-9a-zA-Z][0-9a-zA-z-_]* :
^[0-9a-zA-Z][0-9a-zA-z-_]*{quote} has a bug: small z instead of big Z
- LabelType, HostType, QueueType, NodeType - Drop the Type suffix
- Move QueueType into DynamicNodeLabelsManager.
- DynamicNodeLabelManager
-- updateRunningNodes: No need to call getNMInNodeSet twice
-- updateRunningNodes - updatingResourceMappings
-- QueueType.labels - accessibleLabels
-- Move acls also into the RMNodeLabelsManager
- NodeLabelsManager - CommonNodeLabelsManager
- DynamicNodeLabelManager - RMNodeLabelManager. Similarly
TestDynamicNodeLabelsManager.
- Let’s rename events like AddToClusterNodeLabelsEvent and store operations
similar to RMStateStore. For e.g. storeNewClusterNodeLables
{quote}
// if here, nm is still null, the only reason is, registered nodeId has
// port = 0. This will only happen in unit test. Some tests registered NM
// with port = 0. Just print a log and skip following step
if (null == nm) {
LOG.warn(Register nodeId is illegal, nodeId= + nodeId.toString());
return;
}
{quote}
That doesn't make sense. Any activated node will have a non-zero port.
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch,
YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch,
YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14166135#comment-14166135
]
Hadoop QA commented on YARN-2494:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674032/YARN-2494.20141009-1.patch
against trunk revision d8d628d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5350//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/5350//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5350//console
This message is automatically generated.
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.20141009-1.patch, YARN-2494.patch,
YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch,
YARN-2494.patch, YARN-2494.patch, YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14166358#comment-14166358
]
Hadoop QA commented on YARN-2494:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674087/YARN-2494.20141009-2.patch
against trunk revision cbd21fd.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5355//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/5355//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5355//console
This message is automatically generated.
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.20141009-1.patch, YARN-2494.20141009-2.patch,
YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch,
YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14153526#comment-14153526
]
Wangda Tan commented on YARN-2494:
--
I feel the Cluster is almost a super set of Collection. I prefer what
[~cwelch] suggested set of method name.
Maybe {{ClusterNodeLabelsCollection}} is slightly clear than
{{ClusterNodeLabels}}, but its name is too long I think :)
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch,
YARN-2494.patch, YARN-2494.patch, YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[ https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151658#comment-14151658 ] Wangda Tan commented on YARN-2494: -- Hi [~vinodkv] and [~cwelch], Thanks for reply! Still working on handling your last comments, will upload patch soon. Regarding method name of NodeLabelManager, I think following suggestion make sense to me: bq. What I really want is to convey is that these are just system recognized nodelabels as opposed to node-lables that are actually mapped against a node. How about addToNodeLabelsCollection(), removeFromNodeLabelsCollection(), addLabelsToNode() and removeLabelsFromNode(). The point about addToNodeLabelsCollection() is that it clearly conveys that there is a NodeLabelsCollection - a set of node-labels known by the system. And regarding bq. Once you have the store abstraction, this will be less of a problem? Clearly NodeLabelsManager is not something that the client needs access to? I think it still has problem: Even if we have store abstraction, we still need some logic to guarantee labels being added are valid (e.g. we need check if a label existed in collection, and label existed in node when we trying to remove some labels from a node). That makes we need put a greater chunk of logic to the store abstraction -- it isn't a simple store abstraction if we do this. I suggest to keep it in common to make node label major logic are live together. Thanks, Wangda [YARN-796] Node label manager API and storage implementations - Key: YARN-2494 URL: https://issues.apache.org/jira/browse/YARN-2494 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch This JIRA includes APIs and storage implementations of node label manager, NodeLabelManager is an abstract class used to manage labels of nodes in the cluster, it has APIs to query/modify - Nodes according to given label - Labels according to given hostname - Add/remove labels - Set labels of nodes in the cluster - Persist/recover changes of labels/labels-on-nodes to/from storage And it has two implementations to store modifications - Memory based storage: It will not persist changes, so all labels will be lost when RM restart - FileSystem based storage: It will persist/recover to/from FileSystem (like HDFS), and all labels and labels-on-nodes will be recovered upon RM restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[ https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152078#comment-14152078 ] Craig Welch commented on YARN-2494: --- Not to dither about names - but Collection is still not terribly clear to me (overly generic), I was thinking previously about Cluster as the differentiator, so: addToClusterNodeLabels(), removeFromClusterNodeLabels(), addLabelsToNode() and removeLabelsFromNode(). I think this conveys the different notions of what the operations are applying to in a pretty clear way. Thoughts? [YARN-796] Node label manager API and storage implementations - Key: YARN-2494 URL: https://issues.apache.org/jira/browse/YARN-2494 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch This JIRA includes APIs and storage implementations of node label manager, NodeLabelManager is an abstract class used to manage labels of nodes in the cluster, it has APIs to query/modify - Nodes according to given label - Labels according to given hostname - Add/remove labels - Set labels of nodes in the cluster - Persist/recover changes of labels/labels-on-nodes to/from storage And it has two implementations to store modifications - Memory based storage: It will not persist changes, so all labels will be lost when RM restart - FileSystem based storage: It will persist/recover to/from FileSystem (like HDFS), and all labels and labels-on-nodes will be recovered upon RM restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152270#comment-14152270
]
Vinod Kumar Vavilapalli commented on YARN-2494:
---
bq. I think it still has problem: Even if we have store abstraction, we still
need some logic to guarantee labels being added are valid (e.g. we need check
if a label existed in collection, and label existed in node when we trying to
remove some labels from a node).
Then that validation code needs to get pulled out in a common layer. My goal it
not put the entire NodelabelsManager in yarn-common - it just doesn't belong
there.
bq. How about addToNodeLabelsCollection(), removeFromNodeLabelsCollection(),
addLabelsToNode() and removeLabelsFromNode()
bq. addToClusterNodeLabels(), removeFromClusterNodeLabels(), addLabelsToNode()
and removeLabelsFromNode().
[~leftnoteasy], [~cwelch], I'm okay with either of the above. Or should we call
it {{ClusterNodeLabelsCollection}}? :)
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch,
YARN-2494.patch, YARN-2494.patch, YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151286#comment-14151286
]
Vinod Kumar Vavilapalli commented on YARN-2494:
---
Still reading up the comments and the latest patch. But some quick replies
bq. Because we need yarn-client access this too (edit labels in storage
directly when RM is offline), so I put these code to common dependency between
yarn-client and yarn-server-resourcemanager.
Once you have the store abstraction, this will be less of a problem? Clearly
NodeLabelsManager is not something that the client needs access to?
bq. I'm not quite agree about this, we can say remove a known label, but we
cannoy say add a known label – a label being add is unknown actually. And
IMO, existing is very similar to known, if a label is existing in
NodeLabelManager, that is a known label.
What I really want is to convey is that these are just system recognized
nodelabels as opposed to node-lables that are actually mapped against a node.
How about {{addToNodeLabelsCollection()}},
{{removeFromNodeLabelsCollection()}}, {{addLabelsToNode()}} and
{{removeLabelsFromNode()}}. The point about {{addToNodeLabelsCollection()}} is
that it clearly conveys that there is a NodeLabelsCollection - a set of
node-labels known by the system.
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch,
YARN-2494.patch, YARN-2494.patch, YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[ https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14148094#comment-14148094 ] Craig Welch commented on YARN-2494: --- -re I suggest to change addLabels to addNodeLabels because we may support more different kind of labels in the future, change removeLabels to removeExistingLabels, and leave NodeLabelsManager.existingLabels unchanged. I thought we'd setteled on just adding Node to the names which did not have it, so addNodeLables, removeNodeLabels, etc. I don't think Existing and Known are particularly helpful, the concern was to distinguish these as NodeLabel operations, to leave room in the future for other kinds of nodes. Also, with the refactor to a store type and dropping the configuration option, do we still have a way to specify something other than the hdfs store? wrt leveldb - we ended up with hdfs for the ha case, I think anything we do should be distributed, not local - so zookeeper, hbase, etc. [YARN-796] Node label manager API and storage implementations - Key: YARN-2494 URL: https://issues.apache.org/jira/browse/YARN-2494 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch This JIRA includes APIs and storage implementations of node label manager, NodeLabelManager is an abstract class used to manage labels of nodes in the cluster, it has APIs to query/modify - Nodes according to given label - Labels according to given hostname - Add/remove labels - Set labels of nodes in the cluster - Persist/recover changes of labels/labels-on-nodes to/from storage And it has two implementations to store modifications - Memory based storage: It will not persist changes, so all labels will be lost when RM restart - FileSystem based storage: It will persist/recover to/from FileSystem (like HDFS), and all labels and labels-on-nodes will be recovered upon RM restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[ https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14148095#comment-14148095 ] Craig Welch commented on YARN-2494: --- ...other kinds of labels..., rather [YARN-796] Node label manager API and storage implementations - Key: YARN-2494 URL: https://issues.apache.org/jira/browse/YARN-2494 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch This JIRA includes APIs and storage implementations of node label manager, NodeLabelManager is an abstract class used to manage labels of nodes in the cluster, it has APIs to query/modify - Nodes according to given label - Labels according to given hostname - Add/remove labels - Set labels of nodes in the cluster - Persist/recover changes of labels/labels-on-nodes to/from storage And it has two implementations to store modifications - Memory based storage: It will not persist changes, so all labels will be lost when RM restart - FileSystem based storage: It will persist/recover to/from FileSystem (like HDFS), and all labels and labels-on-nodes will be recovered upon RM restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146333#comment-14146333
]
Wangda Tan commented on YARN-2494:
--
[~cwelch],
I replied your comment in YARN-796 here, I think it's an implementation detail
of FileSystemNodeLabelManager:
bq. It looks like the FileSystemNodeLabelManager will just append changes to
the edit log forever, until it is restarted, is that correct? If so, a
long-running cluster with lots of changes could result in a rather large edit
log. I think every so many writes (N writes) a recovery should be forced to
clean up the edit log and consolidate state (do a recover...)
It a good suggestion, but I think it's more like an enhancement to me. I
roughly estimate, if we have 10,000 node label changes in one hour, average
size of the label is 16 (8 for label and 8 for node), if we have the cluster
running for one year, size of the editlog will be: {{1 * 16 * 24 * 365 /
1024 / 1024}} MB = 1336 MB, according to existing HDFS read throughput (at
least we can get 50MB / sec), it should be acceptable to me if restart a RM ran
for a whole year and cost about 30s extra time.
I agree that periodically create a new mirror and cleanup editlog is better
than this, we can do it if we have other high priority problems addressed.
Thanks,
Wangda
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch,
YARN-2494.patch, YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[ https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14144501#comment-14144501 ] Wangda Tan commented on YARN-2494: -- This is make sense to me, we may have rack label, etc. support in the future. I think we need modify this in rmadmin CLI and REST API, agree? [YARN-796] Node label manager API and storage implementations - Key: YARN-2494 URL: https://issues.apache.org/jira/browse/YARN-2494 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch This JIRA includes APIs and storage implementations of node label manager, NodeLabelManager is an abstract class used to manage labels of nodes in the cluster, it has APIs to query/modify - Nodes according to given label - Labels according to given hostname - Add/remove labels - Set labels of nodes in the cluster - Persist/recover changes of labels/labels-on-nodes to/from storage And it has two implementations to store modifications - Memory based storage: It will not persist changes, so all labels will be lost when RM restart - FileSystem based storage: It will persist/recover to/from FileSystem (like HDFS), and all labels and labels-on-nodes will be recovered upon RM restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14144556#comment-14144556
]
Hadoop QA commented on YARN-2494:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670661/YARN-2494.patch
against trunk revision f557820.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5082//console
This message is automatically generated.
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch,
YARN-2494.patch, YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[ https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143665#comment-14143665 ] Craig Welch commented on YARN-2494: --- The other day [~vinodkv] suggested changing the addLabel removeLabel ... api's to addNodeLabel removeNodeLabel... to make it more clear (and presumably make adding other possible types of labels in the future more smooth). This would not effect the label apis, the node-to-label ones are ok already, I think. Thoughts? [YARN-796] Node label manager API and storage implementations - Key: YARN-2494 URL: https://issues.apache.org/jira/browse/YARN-2494 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch, YARN-2494.patch This JIRA includes APIs and storage implementations of node label manager, NodeLabelManager is an abstract class used to manage labels of nodes in the cluster, it has APIs to query/modify - Nodes according to given label - Labels according to given hostname - Add/remove labels - Set labels of nodes in the cluster - Persist/recover changes of labels/labels-on-nodes to/from storage And it has two implementations to store modifications - Memory based storage: It will not persist changes, so all labels will be lost when RM restart - FileSystem based storage: It will persist/recover to/from FileSystem (like HDFS), and all labels and labels-on-nodes will be recovered upon RM restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136872#comment-14136872
]
Wangda Tan commented on YARN-2494:
--
Hi [~cwelch],
Thanks for your comments,
bq. the only change is the (?automated) removal of an import, I think you
should just drop it from the change set.
Good catch, reverted this file.
bq. why force all to lower case? Discussion favored dropping that...
Updated according to our discussion
bq. checks for valid labels, there must be an easier way/stringlib/regex
Good suggestion, updated
bq. also in updateLableResource - it looks like if node1 has label a b and
queue q1 has label a b it’s resources will be added 2x and removed 2x, while
present it will have a 2x value (1x too many)
It should not, you can check the test:
{{TestNodeLabelManager#testGetQueueResource}}, it should cover the case you
described.
bq. line 603 exception message needs to include “or not present”
I found there's no exception msg around line 603, could you please update
according to latest patch?
bq. pls rename activeNode deactiveNode to activateNode and deactivateNode
Renamed,
Attached a new patch address your comments, please kindly review.
Thanks!
Wangda
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch,
YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136885#comment-14136885
]
Hadoop QA commented on YARN-2494:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12669353/YARN-2494.patch
against trunk revision c0c7e6f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4986//console
This message is automatically generated.
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch,
YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[ https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135774#comment-14135774 ] Craig Welch commented on YARN-2494: --- UpdateNodeResourceRequestPBImple.java: the only change is the (?automated) removal of an import, I think you should just drop it from the change set. NodeLabelManager.java why force all to lower case? Discussion favored dropping that... checks for valid labels, there must be an easier way/stringlib/regex addNMInNodeAlreadyHasNM - what happens when the same node has multiple labels which are all usable by a given queue? Looks like it would be added + times also in updateLableResource - it looks like if node1 has label a b and queue q1 has label a b it’s resources will be added 2x and removed 2x, while present it will have a 2x value (1x too many) (I’m going to write a test to see, but that’s what I think might happen based on what I see in the code) line 603 exception message needs to include “or not present” pls rename activeNode deactiveNode to activateNode and deactivateNode [YARN-796] Node label manager API and storage implementations - Key: YARN-2494 URL: https://issues.apache.org/jira/browse/YARN-2494 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch This JIRA includes APIs and storage implementations of node label manager, NodeLabelManager is an abstract class used to manage labels of nodes in the cluster, it has APIs to query/modify - Nodes according to given label - Labels according to given hostname - Add/remove labels - Set labels of nodes in the cluster - Persist/recover changes of labels/labels-on-nodes to/from storage And it has two implementations to store modifications - Memory based storage: It will not persist changes, so all labels will be lost when RM restart - FileSystem based storage: It will persist/recover to/from FileSystem (like HDFS), and all labels and labels-on-nodes will be recovered upon RM restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131615#comment-14131615
]
Hadoop QA commented on YARN-2494:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668344/YARN-2494.patch
against trunk revision 78b0483.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4916//console
This message is automatically generated.
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.patch, YARN-2494.patch, YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125673#comment-14125673
]
Hadoop QA commented on YARN-2494:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667170/YARN-2494.patch
against trunk revision 0974f43.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 6
warning messages.
See
https://builds.apache.org/job/PreCommit-YARN-Build/4843//artifact/trunk/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:
org.apache.hadoop.yarn.label.TestFileSystemNodeLabelManager
org.apache.hadoop.yarn.label.TestNodeLabelManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4843//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4843//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4843//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4843//console
This message is automatically generated.
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2494) [YARN-796] Node label manager API and storage implementations
[
https://issues.apache.org/jira/browse/YARN-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126493#comment-14126493
]
Hadoop QA commented on YARN-2494:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667321/YARN-2494.patch
against trunk revision 7498dd7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4853//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4853//console
This message is automatically generated.
[YARN-796] Node label manager API and storage implementations
-
Key: YARN-2494
URL: https://issues.apache.org/jira/browse/YARN-2494
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2494.patch, YARN-2494.patch
This JIRA includes APIs and storage implementations of node label manager,
NodeLabelManager is an abstract class used to manage labels of nodes in the
cluster, it has APIs to query/modify
- Nodes according to given label
- Labels according to given hostname
- Add/remove labels
- Set labels of nodes in the cluster
- Persist/recover changes of labels/labels-on-nodes to/from storage
And it has two implementations to store modifications
- Memory based storage: It will not persist changes, so all labels will be
lost when RM restart
- FileSystem based storage: It will persist/recover to/from FileSystem (like
HDFS), and all labels and labels-on-nodes will be recovered upon RM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
