YARN-7056. [Partial backport] Document Resource Profiles feature. (Sunil G via wangda)
Change-Id: I5f5d52cdf7aba4ac09684aab573e6916dbad183e (cherry picked from commit 6b88cd1901a65b977fed759e322dcd75a2cd33b6) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/335e3a63 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/335e3a63 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/335e3a63 Branch: refs/heads/branch-3.0 Commit: 335e3a633424c743f9f4bd1658728f979dfe43e0 Parents: 4bf2991 Author: Wangda Tan <wan...@apache.org> Authored: Thu Sep 7 10:07:35 2017 -0700 Committer: Daniel Templeton <templ...@apache.org> Committed: Tue Oct 31 15:06:17 2017 -0700 ---------------------------------------------------------------------- hadoop-project/src/site/site.xml | 1 + .../src/site/markdown/ResourceProfiles.md | 79 ++++++++++++++++++++ 2 files changed, 80 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/335e3a63/hadoop-project/src/site/site.xml ---------------------------------------------------------------------- diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index 35e743a..2aa1da7 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -144,6 +144,7 @@ <item name="Opportunistic Containers" href="hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html"/> <item name="YARN Federation" href="hadoop-yarn/hadoop-yarn-site/Federation.html"/> <item name="Shared Cache" href="hadoop-yarn/hadoop-yarn-site/SharedCache.html"/> + <item name="Resource Profiles" href="hadoop-yarn/hadoop-yarn-site/ResourceProfiles.html"/> </menu> <menu name="YARN REST APIs" inherit="top"> http://git-wip-us.apache.org/repos/asf/hadoop/blob/335e3a63/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md new file mode 100644 index 0000000..e7b38e1 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md @@ -0,0 +1,79 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Hadoop: YARN Resource Types +=========================== + +Overview +-------- +Resource types support in YARN helps to extend the YARN resource model to a more flexible model which makes it easier to add new countable resourceÂtypes. This solution also helps the users to submit jobs with ease to specify the resources they need. + +Resource model of YARN +----------------------- +Resource Manager will load a new configuration file named `resource-types.xml` to determine the set of resource Âtypes for which scheduling is enabled. Sample XML will look like below. + +```xml +<configuration> + <property> + <name>yarn.resource-types</name> + <value>resource1, resource2</value> + </property> + + <property> + <name>yarn.resource-types.resource1.units</name> + <value>G</value> + </property> +</configuration> +``` + +Similarly, a new configuration file `nodeÂ-resources.xml` will also be loaded by Node Manager where the resource capabilities of a node can be specified. + +```xml +<configuration> + <property> + <name>yarn.nodemanager.resource-type.resource1</name> + <value>5G</value> + </property> + + <property> + <name>yarn.nodemanager.resource-type.resource2</name> + <value>2m</value> + </property> + +</configuration> +``` + +Node Manager will use these custom resource types and will register it's capability to Resource Manager. + +Configurations +------------- + +Please note that, `resource-types.xml` and `nodeÂ-resources.xml` file also need to be placed in conf directory if new resources are to be added to YARN. + +*In `resource-types.xml`* + +| Configuration Property | Value | Description | +|:---- |:---- |:---- | +| `yarn.resource-types` | resource1 | Custom resource | +| `yarn.resource-types.resource1.units` | G | Default unit for resource1 type | + +*In `nodeÂ-resources.xml`* + +| Configuration Property | Value | Description | +|:---- |:---- |:---- | +| `yarn.nodemanager.resource-type.resource1` | 5G | Resource capability for resource named 'resource1'. | + --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org