[ https://issues.apache.org/jira/browse/TEZ-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377068#comment-14377068 ]
Gopal V commented on TEZ-2217: ------------------------------ I quickly cross-checked, this - it seems to be still letting go of containers despite min-held being > queue size. The containers were observed as being released during the getSplits() operation. {code} 2015-03-23 18:35:14,865 INFO [InputInitializer [Map 1] #0] io.HiveInputFormat: Generating splits 2015-03-23 18:35:14,870 INFO [InputInitializer [Map 1] #0] log.PerfLogger: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl> 2015-03-23 18:35:14,889 INFO [DelayedContainerManager] rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing container, containerId=container_1424502260528_1391_01_000310, containerExpiryTime=1427160914665, idleTimeoutMin=5000 2015-03-23 18:35:14,889 INFO [DelayedContainerManager] rm.YarnTaskSchedulerService: Releasing unused container: container_1424502260528_1391_01_000310 2015-03-23 18:35:14,889 INFO [Dispatcher thread: Central] history.HistoryEventHandler: [HISTORY][DAG:dag_1424502260528_1391_11][Event:CONTAINER_STOPPED]: containerId=container_1424502260528_1391_01_000310, stoppedTime=1427160914889, exitStatus=0 2015-03-23 18:35:14,889 INFO [Dispatcher thread: Central] container.AMContainerImpl: AMContainer container_1424502260528_1391_01_000310 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST 2015-03-23 18:35:14,890 INFO [ContainerLauncher #25] launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_STOP_REQUEST {code} > The min-held-containers constraint is not enforced during query runtime > ------------------------------------------------------------------------ > > Key: TEZ-2217 > URL: https://issues.apache.org/jira/browse/TEZ-2217 > Project: Apache Tez > Issue Type: Bug > Affects Versions: 0.6.0, 0.7.0 > Reporter: Gopal V > Assignee: Bikas Saha > Attachments: TEZ-2217.1.patch, TEZ-2217.txt.bz2 > > > The min-held containers constraint is respected during query idle times, but > is not respected when a query is actually in motion. > The AM releases unused containers during dag execution without checking for > min-held containers. > {code} > 2015-03-20 15:41:53,475 INFO [DelayedContainerManager] > rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing > container, containerId=container_1424502260528_1348_01_000013, > containerExpiryTime=1426891313264, idleTimeoutMin=5000 > 2015-03-20 15:41:53,475 INFO [DelayedContainerManager] > rm.YarnTaskSchedulerService: Releasing unused container: > container_1424502260528_1348_01_000013 > {code} > This is actually useful only after the AM has received a soft pre-emption > message, doing it on an idle cluster slows down one of the most common query > patterns in BI systems. > {code} > create temporary table smalltable as ...; > select ... bigtable JOIN smalltable ON ...; > {code} > The smaller query in the beginning throws away the pre-warmed capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)