[jira] [Updated] (TEZ-2217) The min-held-containers constraint is not enforced during query runtime
[
https://issues.apache.org/jira/browse/TEZ-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikas Saha updated TEZ-2217:
Attachment: TEZ-2217.4.patch
Attaching commit patch.
> The min-held-containers constraint is not enforced during query runtime
>
>
> Key: TEZ-2217
> URL: https://issues.apache.org/jira/browse/TEZ-2217
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Gopal V
>Assignee: Bikas Saha
> Attachments: TEZ-2217-debug.txt.bz2, TEZ-2217.1.patch,
> TEZ-2217.2.patch, TEZ-2217.3.patch, TEZ-2217.4.patch, TEZ-2217.txt.bz2
>
>
> The min-held containers constraint is respected during query idle times, but
> is not respected when a query is actually in motion.
> The AM releases unused containers during dag execution without checking for
> min-held containers.
> {code}
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing
> container, containerId=container_1424502260528_1348_01_13,
> containerExpiryTime=1426891313264, idleTimeoutMin=5000
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Releasing unused container:
> container_1424502260528_1348_01_13
> {code}
> This is actually useful only after the AM has received a soft pre-emption
> message, doing it on an idle cluster slows down one of the most common query
> patterns in BI systems.
> {code}
> create temporary table smalltable as ...;
> select ... bigtable JOIN smalltable ON ...;
> {code}
> The smaller query in the beginning throws away the pre-warmed capacity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2217) The min-held-containers constraint is not enforced during query runtime
[
https://issues.apache.org/jira/browse/TEZ-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikas Saha updated TEZ-2217:
Attachment: TEZ-2217.3.patch
> The min-held-containers constraint is not enforced during query runtime
>
>
> Key: TEZ-2217
> URL: https://issues.apache.org/jira/browse/TEZ-2217
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Gopal V
>Assignee: Bikas Saha
> Attachments: TEZ-2217-debug.txt.bz2, TEZ-2217.1.patch,
> TEZ-2217.2.patch, TEZ-2217.3.patch, TEZ-2217.txt.bz2
>
>
> The min-held containers constraint is respected during query idle times, but
> is not respected when a query is actually in motion.
> The AM releases unused containers during dag execution without checking for
> min-held containers.
> {code}
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing
> container, containerId=container_1424502260528_1348_01_13,
> containerExpiryTime=1426891313264, idleTimeoutMin=5000
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Releasing unused container:
> container_1424502260528_1348_01_13
> {code}
> This is actually useful only after the AM has received a soft pre-emption
> message, doing it on an idle cluster slows down one of the most common query
> patterns in BI systems.
> {code}
> create temporary table smalltable as ...;
> select ... bigtable JOIN smalltable ON ...;
> {code}
> The smaller query in the beginning throws away the pre-warmed capacity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2217) The min-held-containers constraint is not enforced during query runtime
[
https://issues.apache.org/jira/browse/TEZ-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikas Saha updated TEZ-2217:
Attachment: (was: TEZ-2217.3.patch)
> The min-held-containers constraint is not enforced during query runtime
>
>
> Key: TEZ-2217
> URL: https://issues.apache.org/jira/browse/TEZ-2217
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Gopal V
>Assignee: Bikas Saha
> Attachments: TEZ-2217-debug.txt.bz2, TEZ-2217.1.patch,
> TEZ-2217.2.patch, TEZ-2217.txt.bz2
>
>
> The min-held containers constraint is respected during query idle times, but
> is not respected when a query is actually in motion.
> The AM releases unused containers during dag execution without checking for
> min-held containers.
> {code}
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing
> container, containerId=container_1424502260528_1348_01_13,
> containerExpiryTime=1426891313264, idleTimeoutMin=5000
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Releasing unused container:
> container_1424502260528_1348_01_13
> {code}
> This is actually useful only after the AM has received a soft pre-emption
> message, doing it on an idle cluster slows down one of the most common query
> patterns in BI systems.
> {code}
> create temporary table smalltable as ...;
> select ... bigtable JOIN smalltable ON ...;
> {code}
> The smaller query in the beginning throws away the pre-warmed capacity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2217) The min-held-containers constraint is not enforced during query runtime
[
https://issues.apache.org/jira/browse/TEZ-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikas Saha updated TEZ-2217:
Attachment: TEZ-2217.3.patch
> The min-held-containers constraint is not enforced during query runtime
>
>
> Key: TEZ-2217
> URL: https://issues.apache.org/jira/browse/TEZ-2217
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Gopal V
>Assignee: Bikas Saha
> Attachments: TEZ-2217-debug.txt.bz2, TEZ-2217.1.patch,
> TEZ-2217.2.patch, TEZ-2217.3.patch, TEZ-2217.txt.bz2
>
>
> The min-held containers constraint is respected during query idle times, but
> is not respected when a query is actually in motion.
> The AM releases unused containers during dag execution without checking for
> min-held containers.
> {code}
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing
> container, containerId=container_1424502260528_1348_01_13,
> containerExpiryTime=1426891313264, idleTimeoutMin=5000
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Releasing unused container:
> container_1424502260528_1348_01_13
> {code}
> This is actually useful only after the AM has received a soft pre-emption
> message, doing it on an idle cluster slows down one of the most common query
> patterns in BI systems.
> {code}
> create temporary table smalltable as ...;
> select ... bigtable JOIN smalltable ON ...;
> {code}
> The smaller query in the beginning throws away the pre-warmed capacity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2217) The min-held-containers constraint is not enforced during query runtime
[
https://issues.apache.org/jira/browse/TEZ-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikas Saha updated TEZ-2217:
Attachment: TEZ-2217.2.patch
New patch. The problem was that the expire time was not update until the min
held container expire time actually elapsed. But if task requests would come in
just before the update would happen, then in the next allocation cycle the min
held containers would be released because they just crossed the expire time
boundary. Looks like the timing of the next dag is currently hitting that race
condition and probably was not hitting it earlier.
Can you please try this out?
> The min-held-containers constraint is not enforced during query runtime
>
>
> Key: TEZ-2217
> URL: https://issues.apache.org/jira/browse/TEZ-2217
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Gopal V
>Assignee: Bikas Saha
> Attachments: TEZ-2217-debug.txt.bz2, TEZ-2217.1.patch,
> TEZ-2217.2.patch, TEZ-2217.txt.bz2
>
>
> The min-held containers constraint is respected during query idle times, but
> is not respected when a query is actually in motion.
> The AM releases unused containers during dag execution without checking for
> min-held containers.
> {code}
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing
> container, containerId=container_1424502260528_1348_01_13,
> containerExpiryTime=1426891313264, idleTimeoutMin=5000
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Releasing unused container:
> container_1424502260528_1348_01_13
> {code}
> This is actually useful only after the AM has received a soft pre-emption
> message, doing it on an idle cluster slows down one of the most common query
> patterns in BI systems.
> {code}
> create temporary table smalltable as ...;
> select ... bigtable JOIN smalltable ON ...;
> {code}
> The smaller query in the beginning throws away the pre-warmed capacity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2217) The min-held-containers constraint is not enforced during query runtime
[
https://issues.apache.org/jira/browse/TEZ-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated TEZ-2217:
-
Attachment: TEZ-2217-debug.txt.bz2
Debug logs attached.
{code}
$ grep "Releasing unused" app-log.txt | wc -l
111
{code}
I always use {{--hiveconf tez.am.log.level="INFO;=DEBUG"}}, that
seems to have worked.
> The min-held-containers constraint is not enforced during query runtime
>
>
> Key: TEZ-2217
> URL: https://issues.apache.org/jira/browse/TEZ-2217
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Gopal V
>Assignee: Bikas Saha
> Attachments: TEZ-2217-debug.txt.bz2, TEZ-2217.1.patch,
> TEZ-2217.txt.bz2
>
>
> The min-held containers constraint is respected during query idle times, but
> is not respected when a query is actually in motion.
> The AM releases unused containers during dag execution without checking for
> min-held containers.
> {code}
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing
> container, containerId=container_1424502260528_1348_01_13,
> containerExpiryTime=1426891313264, idleTimeoutMin=5000
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Releasing unused container:
> container_1424502260528_1348_01_13
> {code}
> This is actually useful only after the AM has received a soft pre-emption
> message, doing it on an idle cluster slows down one of the most common query
> patterns in BI systems.
> {code}
> create temporary table smalltable as ...;
> select ... bigtable JOIN smalltable ON ...;
> {code}
> The smaller query in the beginning throws away the pre-warmed capacity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2217) The min-held-containers constraint is not enforced during query runtime
[
https://issues.apache.org/jira/browse/TEZ-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikas Saha updated TEZ-2217:
Attachment: TEZ-2217.1.patch
Attaching a fix that ensures that when there are no further pending container
requests then new containers are not released if they have been added to the
min held list. This should be safe because there are no pending requests.
[~gopalv] Can you please try this out and see if this fixes your case? If so,
then a review would be great :) The code change is minimal and explained above.
The test was a pain to write :P
> The min-held-containers constraint is not enforced during query runtime
>
>
> Key: TEZ-2217
> URL: https://issues.apache.org/jira/browse/TEZ-2217
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Gopal V
>Assignee: Bikas Saha
> Attachments: TEZ-2217.1.patch, TEZ-2217.txt.bz2
>
>
> The min-held containers constraint is respected during query idle times, but
> is not respected when a query is actually in motion.
> The AM releases unused containers during dag execution without checking for
> min-held containers.
> {code}
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing
> container, containerId=container_1424502260528_1348_01_13,
> containerExpiryTime=1426891313264, idleTimeoutMin=5000
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Releasing unused container:
> container_1424502260528_1348_01_13
> {code}
> This is actually useful only after the AM has received a soft pre-emption
> message, doing it on an idle cluster slows down one of the most common query
> patterns in BI systems.
> {code}
> create temporary table smalltable as ...;
> select ... bigtable JOIN smalltable ON ...;
> {code}
> The smaller query in the beginning throws away the pre-warmed capacity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2217) The min-held-containers constraint is not enforced during query runtime
[
https://issues.apache.org/jira/browse/TEZ-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated TEZ-2217:
-
Attachment: TEZ-2217.txt.bz2
Logs for a 600-container min-held + ~50 containers in capacity.
The AM should not be releasing any containers it gets, but it does.
> The min-held-containers constraint is not enforced during query runtime
>
>
> Key: TEZ-2217
> URL: https://issues.apache.org/jira/browse/TEZ-2217
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Gopal V
>Assignee: Bikas Saha
> Attachments: TEZ-2217.txt.bz2
>
>
> The min-held containers constraint is respected during query idle times, but
> is not respected when a query is actually in motion.
> The AM releases unused containers during dag execution without checking for
> min-held containers.
> {code}
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing
> container, containerId=container_1424502260528_1348_01_13,
> containerExpiryTime=1426891313264, idleTimeoutMin=5000
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Releasing unused container:
> container_1424502260528_1348_01_13
> {code}
> This is actually useful only after the AM has received a soft pre-emption
> message, doing it on an idle cluster slows down one of the most common query
> patterns in BI systems.
> {code}
> create temporary table smalltable as ...;
> select ... bigtable JOIN smalltable ON ...;
> {code}
> The smaller query in the beginning throws away the pre-warmed capacity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
