[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413913#comment-13413913
 ] 

Andrew Ferguson commented on MAPREDUCE-4334:
--------------------------------------------

hi all, I think there are pros and cons to both approaches, which I will try to 
outline below.

Cgroups:
- they provide a coherent path for future resource management: network 
bandwidth, CPU upper- and lower-bounds, block I/O priorities and limits, etc. 
[1]
- can be integrated with resource management for other applications, drawing 
upon a single resource budget for a group of users
- cgroup's hierarchies are key to this. in a taskset-only world, the NM would 
need to be given a fixed allocation of the node's CPUs to manage
- cgroups are not persistent across reboots. this is unfortunate. however, 1) 
anyone using them needs to mount them on startup, so they will need to make a 
change to their startup process already, and 2) there are extensive, 
cross-distro tools to create and manage cgroups automatically on reboot (RHEL 6 
has great docs on them [2])
- some clusters are already using Cgroups, without any support from 
Hadoop/YARN. for example, StumbleUpon [3]


Taskset:
- compatible with RHEL 5
- does not require changes to node startup
- can be implemented with a SUID root binary, as LCE is today


My first design for this JIRA had the LCE create the cgroups. This turned out 
to be the wrong approach for several reasons:
- What if I wanted to use the regular container executor with cgroups? An admin 
may not allow me to have a setuid root binary, but may be willing to create a 
cgroup hierarchy for me (after all, this is one advantage of the hierarchy: 
delegation)
- Conversely, what if I wanted to use the LCE without cgroups?
- There needs to be a part of the NM responsible for deleting unused cgroups, 
and the other tasks of a ContainersManager I described in MAPREDUCE-4351. Some 
of those are specific to how resource enforcement is being done; it seemed best 
to keep that code together in the ContainersManager, then spread across a 
ContainersManager and the LCE.
- Putting the resource enforcement "smarts" in the ContainersMonitor (which is 
already receiving events from the RM), allows it to dynamically adjust the 
resource enforcement
- On startup, the JVM can appear to be using twice as much memory as it 
actually is (see comment in ContainersMonitorImpl.java). By starting the JVM 
within the cgroup, rather than allowing it to start outside the cgroup and 
moving it into the cgroup with a ContainersMonitor as my patch above does, the 
kernel may kill the JVM inadvertently.
 

I really like the flexibility of keeping the LCE and resource enforcement 
separate. In my mind, the LCE is for starting processes, and should stick to 
doing that. Resource enforcement is a separate job.



My recommendation is the following:
1) Keep the LCE as it is.
2) Support pluggable ContainersMonitors (MAPREDUCE-4351)
3) For cgroups, we can start with the patch above. It is best for admins who 
already use cgroups on their nodes and want to have YARN take advantage of 
them. (This is the point of the yarn.nodemanger.cgroups.path config option I 
added)
4) For cgroups, we could have a second ContainersMonitor plugin which uses a 
setuid root binary to also mount & create cgroups, freeing the admin from 
managing them at all.
5) For taskset, we can implement a ContainersMonitor which uses a setuid root 
binary (potentially the LCE, but perhaps better if it's something else, just to 
keep the security footprint down) to pin processes to CPUs. This 
ContainersMonitor will also need the memory enforcement code from the current 
ContainersMonitorImpl


I've done 1-3 (well, #1 is a freebie :-) ... and I can definitely do #5 as well.


Arun, does this design appeal to you?



[1] 
http://www.linux-kongress.org/2010/slides/seyfried-cgroups-linux-kongress-2010-presentation.pdf
[2] 
https://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/index.html
[3] http://inchoate-clatter.blogspot.com/2012/03/hbase-ops-automation.html

                
> Add support for CPU isolation/monitoring of containers
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-4334
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>            Reporter: Arun C Murthy
>            Assignee: Andrew Ferguson
>         Attachments: MAPREDUCE-4334-pre1.patch, 
> MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, 
> MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch
>
>
> Once we get in MAPREDUCE-4327, it will be important to actually enforce 
> limits on CPU consumption of containers. 
> Several options spring to mind:
> # taskset (RHEL5+)
> # cgroups (RHEL6+)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Reply via email to