[ https://issues.apache.org/jira/browse/YARN-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Ferguson updated YARN-3: ------------------------------- Attachment: YARN-3-lce_only-v1.patch This patch augments the LinuxContainerExecutor with an LCEResourcesHandler, which can be used to enforce resource limits using either cgroups (in this patch) or sched_setaffinity/taskset (future patch). A DefaultLCEResourcesHandler is also provided which does not enforce any new resource limits. The LCEResourcesHandler interface (and concrete classes) are introduced to keep the LinuxContainerExecutor java class simple, and to separate the cgroups and future sched_setaffinity logic. The resources handler code is split across the introduced Java classes, and the existing container-executor native binary. This is done to minimize the amount of code added to the native binary, to provide easy logging via Java mechanisms, and because a singleton is needed to track CPU assignments when using sched_setaffinity/taskset. The handler operates synchronously with the execution of the container. The changes to the native code are: 1) A resources option has been added to the LaunchContainer command. This option is used to convey a list of cgroups into which the container should be placed before the user command is launched, and, in the future, will be used to alternatively covey a list of CPUs which the process should be pinned to, if using sched_setaffinity instead of cgroups. 2) A --mount-cgroups command has been added to the native code. This command will mount cgroups controllers and create hierarchies for the NodeManager to manage. This feature is optional (see below), and exposed to the Java code via a new method in LinuxContainerExecutor.java. The following configuration options are introduced: yarn.nodemanager.linux-container-executor.resources-handler.class -- The class which should assist the LCE in handling resources. yarn.nodemanager.linux-container-executor.cgroups.hierarchy -- The cgroups hierarchy under which to place YARN proccesses (cannot contain commas). Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler. yarn.nodemanager.linux-container-executor.cgroups.mount -- Whether the LCE should attempt to mount cgroups if not found. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler. yarn.nodemanager.linux-container-executor.cgroups.mount-path -- Where the LCE should attempt to mount cgroups if not found. Common locations include /sys/fs/cgroup and /cgroup. The path must exist before the NodeManager is launched. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler. > Add support for CPU isolation/monitoring of containers > ------------------------------------------------------ > > Key: YARN-3 > URL: https://issues.apache.org/jira/browse/YARN-3 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Arun C Murthy > Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4334-executor-v1.patch, > MAPREDUCE-4334-executor-v2.patch, MAPREDUCE-4334-executor-v3.patch, > MAPREDUCE-4334-executor-v4.patch, MAPREDUCE-4334-pre1.patch, > MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, > MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch, > MAPREDUCE-4334-v1.patch, MAPREDUCE-4334-v2.patch, YARN-3-lce_only-v1.patch, > mapreduce-4334-design-doc-v2.txt, mapreduce-4334-design-doc.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira