[jira] [Commented] (TEZ-1698) Cut down on ResourceCalculatorProcessTree overheads in Tez
[ https://issues.apache.org/jira/browse/TEZ-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191204#comment-14191204 ] Gopal V commented on TEZ-1698: -- +1 - Thanks Rajesh, this looks good. This isn't on by default, so it should be good for 0.5.2 as well. > Cut down on ResourceCalculatorProcessTree overheads in Tez > -- > > Key: TEZ-1698 > URL: https://issues.apache.org/jira/browse/TEZ-1698 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.5.2 >Reporter: Gopal V >Assignee: Rajesh Balamohan > Attachments: ProcfsBasedProcessTree.png, ProcfsFiles.png, > TEZ-1698.1.patch, TEZ-1698.2.patch, TEZ-1698.3.patch, TEZ-1698.4.patch > > > ResourceCalculatorProcessTree scraps all of /proc/ for PIDs which are part of > the current task's process group. > This is mostly wasted in Tez, since unlike YARN which has to do this since it > has the PID for the container-executor process (bash) and has to trace the > bash -> java spawn inheritance. > !ProcfsBasedProcessTree.png! > The latency effect of this is less clearly visible with the profiler turned > on as this is primarily related to rate of syscalls + overhead in the kernel > (via the following codepath in YARN). > !ProcfsFiles.png! > {code} > private List getProcessList() { > String[] processDirs = (new File(procfsDir)).list(); > ... > for (String dir : processDirs) { > try { > if ((new File(procfsDir, dir)).isDirectory()) { > processList.add(dir); > } > ... > public void updateProcessTree() { > if (!pid.equals(deadPid)) { > // Get the list of processes > List processList = getProcessList(); > ... > for (String proc : processList) { > // Get information for each process > ProcessInfo pInfo = new ProcessInfo(proc); > if (constructProcessInfo(pInfo, procfsDir) != null) { > allProcessInfo.put(proc, pInfo); > if (proc.equals(this.pid)) { > me = pInfo; // cache 'me' > processTree.put(proc, pInfo); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1698) Cut down on ResourceCalculatorProcessTree overheads in Tez
[ https://issues.apache.org/jira/browse/TEZ-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191458#comment-14191458 ] Siddharth Seth commented on TEZ-1698: - [~rajesh.balamohan] - I think the license needs to be at the top of files. Before the package / imports. > Cut down on ResourceCalculatorProcessTree overheads in Tez > -- > > Key: TEZ-1698 > URL: https://issues.apache.org/jira/browse/TEZ-1698 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.5.2 >Reporter: Gopal V >Assignee: Rajesh Balamohan > Fix For: 0.5.2 > > Attachments: ProcfsBasedProcessTree.png, ProcfsFiles.png, > TEZ-1698.1.patch, TEZ-1698.2.patch, TEZ-1698.3.patch, TEZ-1698.4.patch > > > ResourceCalculatorProcessTree scraps all of /proc/ for PIDs which are part of > the current task's process group. > This is mostly wasted in Tez, since unlike YARN which has to do this since it > has the PID for the container-executor process (bash) and has to trace the > bash -> java spawn inheritance. > !ProcfsBasedProcessTree.png! > The latency effect of this is less clearly visible with the profiler turned > on as this is primarily related to rate of syscalls + overhead in the kernel > (via the following codepath in YARN). > !ProcfsFiles.png! > {code} > private List getProcessList() { > String[] processDirs = (new File(procfsDir)).list(); > ... > for (String dir : processDirs) { > try { > if ((new File(procfsDir, dir)).isDirectory()) { > processList.add(dir); > } > ... > public void updateProcessTree() { > if (!pid.equals(deadPid)) { > // Get the list of processes > List processList = getProcessList(); > ... > for (String proc : processList) { > // Get information for each process > ProcessInfo pInfo = new ProcessInfo(proc); > if (constructProcessInfo(pInfo, procfsDir) != null) { > allProcessInfo.put(proc, pInfo); > if (proc.equals(this.pid)) { > me = pInfo; // cache 'me' > processTree.put(proc, pInfo); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)