[ https://issues.apache.org/jira/browse/YARN-799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13681489#comment-13681489 ]
Chris Riccomini commented on YARN-799: -------------------------------------- Fair enough. One thing that I notice is, when I patch to write to /tasks, the tasks file has >1 PID in it, while the cgroup.procs file has only 1 (as expected). This suggests to me that the tasks file contains all child PIDs, so I'm a bit confused about the comment in YARN-3. Nevertheless, it'd be worth verifying. > CgroupsLCEResourcesHandler tries to write to cgroup.procs > --------------------------------------------------------- > > Key: YARN-799 > URL: https://issues.apache.org/jira/browse/YARN-799 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Affects Versions: 2.0.4-alpha, 2.0.5-alpha > Reporter: Chris Riccomini > > The implementation of > bq. > ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java > Tells the container-executor to write PIDs to cgroup.procs: > {code} > public String getResourcesOption(ContainerId containerId) { > String containerName = containerId.toString(); > StringBuilder sb = new StringBuilder("cgroups="); > if (isCpuWeightEnabled()) { > sb.append(pathForCgroup(CONTROLLER_CPU, containerName) + > "/cgroup.procs"); > sb.append(","); > } > if (sb.charAt(sb.length() - 1) == ',') { > sb.deleteCharAt(sb.length() - 1); > } > return sb.toString(); > } > {code} > Apparently, this file has not always been writeable: > https://patchwork.kernel.org/patch/116146/ > http://lkml.indiana.edu/hypermail/linux/kernel/1004.1/00536.html > https://lists.linux-foundation.org/pipermail/containers/2009-July/019679.html > The RHEL version of the Linux kernel that I'm using has a CGroup module that > has a non-writeable cgroup.procs file. > {quote} > $ uname -a > Linux criccomi-ld 2.6.32-131.4.1.el6.x86_64 #1 SMP Fri Jun 10 10:54:26 EDT > 2011 x86_64 x86_64 x86_64 GNU/Linux > {quote} > As a result, when the container-executor tries to run, it fails with this > error message: > bq. fprintf(LOGFILE, "Failed to write pid %s (%d) to file %s - %s\n", > This is because the executor is given a resource by the > CgroupsLCEResourcesHandler that includes cgroup.procs, which is non-writeable: > {quote} > $ pwd > /cgroup/cpu/hadoop-yarn/container_1370986842149_0001_01_000001 > $ ls -l > total 0 > -r--r--r-- 1 criccomi eng 0 Jun 11 14:43 cgroup.procs > -rw-r--r-- 1 criccomi eng 0 Jun 11 14:43 cpu.rt_period_us > -rw-r--r-- 1 criccomi eng 0 Jun 11 14:43 cpu.rt_runtime_us > -rw-r--r-- 1 criccomi eng 0 Jun 11 14:43 cpu.shares > -rw-r--r-- 1 criccomi eng 0 Jun 11 14:43 notify_on_release > -rw-r--r-- 1 criccomi eng 0 Jun 11 14:43 tasks > {quote} > I patched CgroupsLCEResourcesHandler to use /tasks instead of /cgroup.procs, > and this appears to have fixed the problem. > I can think of several potential resolutions to this ticket: > 1. Ignore the problem, and make people patch YARN when they hit this issue. > 2. Write to /tasks instead of /cgroup.procs for everyone > 3. Check permissioning on /cgroup.procs prior to writing to it, and fall back > to /tasks. > 4. Add a config to yarn-site that lets admins specify which file to write to. > Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira