[
https://issues.apache.org/jira/browse/TEZ-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377115#comment-14377115
]
Bikas Saha commented on TEZ-2217:
---------------------------------
The existing debug logs should be enough if enabled. What is intriguing is that
at this point in time there are pending task requests that have not already
been matched to the containers because I am guessing that the job already has
all the containers it will ever get. If that was not the case then it would hit
the changed code path (AM is idle or there are no pending requests).
What is the min expiry time compared to the delays between node-rack-star
matching? Hoping that the containers have been tried to be matched upto star
before the min expiry elapses. So all tasks should have been matched to some
containers leading to empty task requests.
> The min-held-containers constraint is not enforced during query runtime
> ------------------------------------------------------------------------
>
> Key: TEZ-2217
> URL: https://issues.apache.org/jira/browse/TEZ-2217
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Gopal V
> Assignee: Bikas Saha
> Attachments: TEZ-2217.1.patch, TEZ-2217.txt.bz2
>
>
> The min-held containers constraint is respected during query idle times, but
> is not respected when a query is actually in motion.
> The AM releases unused containers during dag execution without checking for
> min-held containers.
> {code}
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing
> container, containerId=container_1424502260528_1348_01_000013,
> containerExpiryTime=1426891313264, idleTimeoutMin=5000
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Releasing unused container:
> container_1424502260528_1348_01_000013
> {code}
> This is actually useful only after the AM has received a soft pre-emption
> message, doing it on an idle cluster slows down one of the most common query
> patterns in BI systems.
> {code}
> create temporary table smalltable as ...;
> select ... bigtable JOIN smalltable ON ...;
> {code}
> The smaller query in the beginning throws away the pre-warmed capacity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
