Author: sandy
Date: Mon Aug 26 19:59:12 2013
New Revision: 1517669
URL: http://svn.apache.org/r1517669
Log:
YARN-1093. Corrections to Fair Scheduler documentation (Wing Yew Poon via Sandy
Ryza)
Modified:
hadoop/common/branches/branch-2/hadoop-yarn-project/CHANGES.txt
hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
Modified: hadoop/common/branches/branch-2/hadoop-yarn-project/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-yarn-project/CHANGES.txt?rev=1517669&r1=1517668&r2=1517669&view=diff
==============================================================================
--- hadoop/common/branches/branch-2/hadoop-yarn-project/CHANGES.txt (original)
+++ hadoop/common/branches/branch-2/hadoop-yarn-project/CHANGES.txt Mon Aug 26
19:59:12 2013
@@ -33,6 +33,9 @@ Release 2.1.1-beta - UNRELEASED
YARN-1074. Cleaned up YARN CLI application list to only display running
applications by default. (Xuan Gong via vinodkv)
+ YARN-1093. Corrections to Fair Scheduler documentation (Wing Yew Poon via
+ Sandy Ryza)
+
OPTIMIZATIONS
BUG FIXES
Modified:
hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm?rev=1517669&r1=1517668&r2=1517669&view=diff
==============================================================================
---
hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
(original)
+++
hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
Mon Aug 26 19:59:12 2013
@@ -47,7 +47,7 @@ Hadoop MapReduce Next Generation - Fair
The scheduler organizes apps further into "queues", and shares resources
fairly between these queues. By default, all users share a single queue,
- called âdefaultâ. If an app specifically lists a queue in a container
resource
+ named âdefaultâ. If an app specifically lists a queue in a container
resource
request, the request is submitted to that queue. It is also possible to
assign
queues based on the user name included with the request through
configuration. Within each queue, a scheduling policy is used to share
@@ -85,7 +85,7 @@ Hadoop MapReduce Next Generation - Fair
their parents in the fair scheduler configuration file.
A queue's name starts with the names of its parents, with periods as
- separators. So a queue named "queue1" under the root named, would be referred
+ separators. So a queue named "queue1" under the root queue, would be referred
to as "root.queue1", and a queue named "queue2" under a queue named "parent1"
would be referred to as "root.parent1.queue2". When referring to queues, the
root part of the name is optional, so queue1 could be referred to as just
@@ -118,22 +118,23 @@ Hadoop MapReduce Next Generation - Fair
Customizing the Fair Scheduler typically involves altering two files. First,
scheduler-wide options can be set by adding configuration properties in the
- fair-scheduler.xml file in your existing configuration directory. Second, in
+ yarn-site.xml file in your existing configuration directory. Second, in
most cases users will want to create a manifest file listing which queues
exist and their respective weights and capacities. The location of this file
- is flexible - but it must be declared in fair-scheduler.xml.
+ is flexible - but it must be declared in yarn-site.xml.
* <<<yarn.scheduler.fair.allocation.file>>>
* Path to allocation file. An allocation file is an XML manifest describing
queues and their properties, in addition to certain policy defaults. This
file
must be in XML format as described in the next section.
+ Defaults to fair-scheduler.xml in configuration directory.
* <<<yarn.scheduler.fair.user-as-default-queue>>>
* Whether to use the username associated with the allocation as the
default
queue name, in the event that a queue name is not specified. If this is
set
- to "false" or unset, all jobs have a shared default queue, called
"default".
+ to "false" or unset, all jobs have a shared default queue, named
"default".
Defaults to true.
* <<<yarn.scheduler.fair.preemption>>>
@@ -178,14 +179,15 @@ Hadoop MapReduce Next Generation - Fair
Allocation file format
- The allocation file must be in XML format. The format contains four types of
+ The allocation file must be in XML format. The format contains five types of
elements:
* <<Queue elements>>, which represent queues. Each may contain the following
properties:
* minResources: minimum resources the queue is entitled to, in the form
- "X mb, Y vcores". If a queue's minimum share is not satisfied, it will be
+ "X mb, Y vcores". For the single-resource fairness policy, the vcores
+ value is ignored. If a queue's minimum share is not satisfied, it will be
offered available resources before any other queue under the same parent.
Under the single-resource fairness policy, a queue
is considered unsatisfied if its memory usage is below its minimum memory
@@ -199,7 +201,8 @@ Allocation file format
may be using those resources.
* maxResources: maximum resources a queue is allowed, in the form
- "X mb, Y vcores". A queue will never be assigned a container that would
+ "X mb, Y vcores". For the single-resource fairness policy, the vcores
+ value is ignored. A queue will never be assigned a container that would
put its aggregate usage over this limit.
* maxRunningApps: limit the number of apps from the queue to run at once
@@ -234,19 +237,23 @@ Allocation file format
its fair share before it will try to preempt containers to take resources
from
other queues.
+ * <<A defaultQueueSchedulingPolicy element>>, which sets the default
scheduling
+ policy for queues; overriden by the schedulingPolicy element in each queue
+ if specified. Defaults to "fair".
+
An example allocation file is given here:
---
<?xml version="1.0"?>
<allocations>
<queue name="sample_queue">
- <minResources>10000 mb</minResources>
- <maxResources>90000 mb</maxResources>
+ <minResources>10000 mb,0vcores</minResources>
+ <maxResources>90000 mb,0vcores</maxResources>
<maxRunningApps>50</maxRunningApps>
<weight>2.0</weight>
<schedulingPolicy>fair</schedulingPolicy>
<queue name="sample_sub_queue">
- <minResources>5000 mb</minResources>
+ <minResources>5000 mb,0vcores</minResources>
</queue>
</queue>
<user name="sample_user">
