[
https://issues.apache.org/jira/browse/YARN-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Ferguson updated YARN-3:
-------------------------------
Attachment: YARN-3-lce_only-v1.patch
This patch augments the LinuxContainerExecutor with an LCEResourcesHandler,
which can be used to enforce resource limits using either cgroups (in this
patch) or sched_setaffinity/taskset (future patch). A
DefaultLCEResourcesHandler is also provided which does not enforce any new
resource limits.
The LCEResourcesHandler interface (and concrete classes) are introduced to keep
the LinuxContainerExecutor java class simple, and to separate the cgroups and
future sched_setaffinity logic.
The resources handler code is split across the introduced Java classes, and the
existing container-executor native binary. This is done to minimize the amount
of code added to the native binary, to provide easy logging via Java
mechanisms, and because a singleton is needed to track CPU assignments when
using sched_setaffinity/taskset.
The handler operates synchronously with the execution of the container.
The changes to the native code are:
1) A resources option has been added to the LaunchContainer command. This
option is used to convey a list of cgroups into which the container should be
placed before the user command is launched, and, in the future, will be used to
alternatively covey a list of CPUs which the process should be pinned to, if
using sched_setaffinity instead of cgroups.
2) A --mount-cgroups command has been added to the native code. This command
will mount cgroups controllers and create hierarchies for the NodeManager to
manage. This feature is optional (see below), and exposed to the Java code via
a new method in LinuxContainerExecutor.java.
The following configuration options are introduced:
yarn.nodemanager.linux-container-executor.resources-handler.class -- The class
which should assist the LCE in handling resources.
yarn.nodemanager.linux-container-executor.cgroups.hierarchy -- The cgroups
hierarchy under which to place YARN proccesses (cannot contain commas). Only
used when the LCE resources handler is set to the CgroupsLCEResourcesHandler.
yarn.nodemanager.linux-container-executor.cgroups.mount -- Whether the LCE
should attempt to mount cgroups if not found. Only used when the LCE resources
handler is set to the CgroupsLCEResourcesHandler.
yarn.nodemanager.linux-container-executor.cgroups.mount-path -- Where the LCE
should attempt to mount cgroups if not found. Common locations include
/sys/fs/cgroup and /cgroup. The path must exist before the NodeManager is
launched. Only used when the LCE resources handler is set to the
CgroupsLCEResourcesHandler.
> Add support for CPU isolation/monitoring of containers
> ------------------------------------------------------
>
> Key: YARN-3
> URL: https://issues.apache.org/jira/browse/YARN-3
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Arun C Murthy
> Assignee: Andrew Ferguson
> Attachments: MAPREDUCE-4334-executor-v1.patch,
> MAPREDUCE-4334-executor-v2.patch, MAPREDUCE-4334-executor-v3.patch,
> MAPREDUCE-4334-executor-v4.patch, MAPREDUCE-4334-pre1.patch,
> MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch,
> MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch,
> MAPREDUCE-4334-v1.patch, MAPREDUCE-4334-v2.patch, YARN-3-lce_only-v1.patch,
> mapreduce-4334-design-doc-v2.txt, mapreduce-4334-design-doc.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
