[
https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413913#comment-13413913
]
Andrew Ferguson commented on MAPREDUCE-4334:
--------------------------------------------
hi all, I think there are pros and cons to both approaches, which I will try to
outline below.
Cgroups:
- they provide a coherent path for future resource management: network
bandwidth, CPU upper- and lower-bounds, block I/O priorities and limits, etc.
[1]
- can be integrated with resource management for other applications, drawing
upon a single resource budget for a group of users
- cgroup's hierarchies are key to this. in a taskset-only world, the NM would
need to be given a fixed allocation of the node's CPUs to manage
- cgroups are not persistent across reboots. this is unfortunate. however, 1)
anyone using them needs to mount them on startup, so they will need to make a
change to their startup process already, and 2) there are extensive,
cross-distro tools to create and manage cgroups automatically on reboot (RHEL 6
has great docs on them [2])
- some clusters are already using Cgroups, without any support from
Hadoop/YARN. for example, StumbleUpon [3]
Taskset:
- compatible with RHEL 5
- does not require changes to node startup
- can be implemented with a SUID root binary, as LCE is today
My first design for this JIRA had the LCE create the cgroups. This turned out
to be the wrong approach for several reasons:
- What if I wanted to use the regular container executor with cgroups? An admin
may not allow me to have a setuid root binary, but may be willing to create a
cgroup hierarchy for me (after all, this is one advantage of the hierarchy:
delegation)
- Conversely, what if I wanted to use the LCE without cgroups?
- There needs to be a part of the NM responsible for deleting unused cgroups,
and the other tasks of a ContainersManager I described in MAPREDUCE-4351. Some
of those are specific to how resource enforcement is being done; it seemed best
to keep that code together in the ContainersManager, then spread across a
ContainersManager and the LCE.
- Putting the resource enforcement "smarts" in the ContainersMonitor (which is
already receiving events from the RM), allows it to dynamically adjust the
resource enforcement
- On startup, the JVM can appear to be using twice as much memory as it
actually is (see comment in ContainersMonitorImpl.java). By starting the JVM
within the cgroup, rather than allowing it to start outside the cgroup and
moving it into the cgroup with a ContainersMonitor as my patch above does, the
kernel may kill the JVM inadvertently.
I really like the flexibility of keeping the LCE and resource enforcement
separate. In my mind, the LCE is for starting processes, and should stick to
doing that. Resource enforcement is a separate job.
My recommendation is the following:
1) Keep the LCE as it is.
2) Support pluggable ContainersMonitors (MAPREDUCE-4351)
3) For cgroups, we can start with the patch above. It is best for admins who
already use cgroups on their nodes and want to have YARN take advantage of
them. (This is the point of the yarn.nodemanger.cgroups.path config option I
added)
4) For cgroups, we could have a second ContainersMonitor plugin which uses a
setuid root binary to also mount & create cgroups, freeing the admin from
managing them at all.
5) For taskset, we can implement a ContainersMonitor which uses a setuid root
binary (potentially the LCE, but perhaps better if it's something else, just to
keep the security footprint down) to pin processes to CPUs. This
ContainersMonitor will also need the memory enforcement code from the current
ContainersMonitorImpl
I've done 1-3 (well, #1 is a freebie :-) ... and I can definitely do #5 as well.
Arun, does this design appeal to you?
[1]
http://www.linux-kongress.org/2010/slides/seyfried-cgroups-linux-kongress-2010-presentation.pdf
[2]
https://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/index.html
[3] http://inchoate-clatter.blogspot.com/2012/03/hbase-ops-automation.html
> Add support for CPU isolation/monitoring of containers
> ------------------------------------------------------
>
> Key: MAPREDUCE-4334
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Reporter: Arun C Murthy
> Assignee: Andrew Ferguson
> Attachments: MAPREDUCE-4334-pre1.patch,
> MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch,
> MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch
>
>
> Once we get in MAPREDUCE-4327, it will be important to actually enforce
> limits on CPU consumption of containers.
> Several options spring to mind:
> # taskset (RHEL5+)
> # cgroups (RHEL6+)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira