Hello Philip Zeyliger, Bikramjeet Vig, Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10058
to look at the new patch set (#7).
Change subject: IMPALA-6847: work around high memory estimates for AC
......................................................................
IMPALA-6847: work around high memory estimates for AC
Adds MAX_MEM_LIMIT_FOR_ADMISSION query option, which takes
effect if and only if
* Memory-based admission control is enabled for the pool
* No mem_limit is set
In that case the planner sets mem_limit to the value of the
new option and uses min(mem_limit, mem estimate) for admission
control.
The idea is that queries with very large estimated consumption can run
even if they exceed the pool resources, but the mem_limit applied by
the planner prevents runaway memory consumption. This is still
a flawed solution because low memory estimates can lead to overcommittal
of memory and high memory estimates can result in underutilization,
but gives a safety valve to escape the worst consequences of poor planner
estimates.
Testing:
Added an admission control test to verify that query option allows
queries with high estimates to run but sets a mem_limit.
Also tested manually on a minicluster started with:
start-impala-cluster.py --impalad_args='-vmodule admission-controller=3 \
-default_pool_mem_limit 12884901888'
Change-Id: Ia5fc32a507ad0f00f564dfe4f954a829ac55d14e
---
M be/src/scheduling/query-schedule.cc
M be/src/service/impala-server.cc
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/service/Frontend.java
A
testdata/workloads/functional-query/queries/QueryTest/admission-reject-mem-estimate.test
M tests/custom_cluster/test_admission_controller.py
11 files changed, 162 insertions(+), 37 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/10058/7
--
To view, visit http://gerrit.cloudera.org:8080/10058
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia5fc32a507ad0f00f564dfe4f954a829ac55d14e
Gerrit-Change-Number: 10058
Gerrit-PatchSet: 7
Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <phi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
