[
https://issues.apache.org/jira/browse/TEZ-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201116#comment-16201116
]
TezQA commented on TEZ-3852:
----------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12891568/TEZ-3852.003.patch
against master revision c82b2ea.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.history.TestHistoryParser
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2660//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2660//console
This message is automatically generated.
> Optimize ContainerContext.isSuperSet to speed container reuse decisions
> -----------------------------------------------------------------------
>
> Key: TEZ-3852
> URL: https://issues.apache.org/jira/browse/TEZ-3852
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Attachments: TEZ-3852.001.patch, TEZ-3852.002.patch,
> TEZ-3852.003.patch
>
>
> Found an AM that was consuming high CPU. The stack trace below shows that
> container reuse compatibility check with a high number of local resources was
> the culprit.
> {noformat:title=task scheduler compatibility check}
> "DelayedContainerManager" #112 prio=5 os_prio=0 tid=0x0000000003b59800
> nid=0x1edba runnable [0x00007fe13c232000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.putVal(HashMap.java:628)
> at java.util.HashMap.putMapEntries(HashMap.java:514)
> at java.util.HashMap.<init>(HashMap.java:489)
> at
> org.apache.tez.dag.app.ContainerContext.localResourcesCompatible(ContainerContext.java:132)
> at
> org.apache.tez.dag.app.ContainerContext.isSuperSet(ContainerContext.java:116)
> at
> org.apache.tez.dag.app.rm.container.ContainerContextMatcher.isSuperSet(ContainerContextMatcher.java:50)
> at
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService.canAssignTaskToContainer(YarnTaskSchedulerService.java:1543)
> at
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService.getMatchingRequestWithoutPriority(YarnTaskSchedulerService.java:1492)
> at
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService.access$500(YarnTaskSchedulerService.java:85)
> at
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService$NodeLocalContainerAssigner.assignReUsedContainer(YarnTaskSchedulerService.java:1870)
> at
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService.assignReUsedContainerWithLocation(YarnTaskSchedulerService.java:1754)
> - locked <0x00000006e0d12600> (a
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService)
> at
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService.assignReUsedContainersWithLocation(YarnTaskSchedulerService.java:1712)
> - locked <0x00000006e0d12600> (a
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService)
> at
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService.tryAssignReUsedContainers(YarnTaskSchedulerService.java:578)
> - locked <0x00000006e0d12600> (a
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService)
> at
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService.access$800(YarnTaskSchedulerService.java:85)
> at
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService$DelayedContainerManager.doAssignAll(YarnTaskSchedulerService.java:2103)
> - locked <0x00000006e0d12600> (a
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService)
> at
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService$DelayedContainerManager.mainLoop(YarnTaskSchedulerService.java:1984)
> at
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService$DelayedContainerManager.run(YarnTaskSchedulerService.java:1974)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
