Hello Andrew Sherman, Lars Volker, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13740
to look at the new patch set (#13).
Change subject: IMPALA-7486: Add specialized estimation scheme for dedicated
coordinators
......................................................................
IMPALA-7486: Add specialized estimation scheme for dedicated coordinators
This patch computes two memory estimates in the frontend:
an estimate for any host that is an executor (including
a combined coordinator and executor) and an estimate
for a dedicated coordinator. This is computed regardless
of whether it is a dedicated coordinator or not.
Admission control then, in the case when the coordinator
is dedicated, uses the coordinator memory estimate for
the coordinator node and the executor memory estimate
for all other nodes.
Other highlights:
- if MEM_LIMIT query option is set, it is applied to all backends
- the min_query_mem_limit pool config is not enforced on the
dedicated coordinator estimates unless MEM_LIMIT query option is set
- the lower cap on estimates and the admission checks based on the
min mem limit required for reservation are applied separately on
coordinator's and executors' mem requirements.
- Added a hidden startup option 'use_dedicated_coordinator_estimates'
which if set to false, reverts to previous estimation behavior.
Testing:
- Added unit test for admission/rejection in dedicated
coordinator clusters.
- Added end to end tests.
Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21
---
M be/src/runtime/coordinator.cc
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/query-schedule.cc
M be/src/scheduling/query-schedule.h
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/impala-http-handler.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
A
testdata/workloads/functional-query/queries/QueryTest/dedicated-coord-mem-estimates.test
M tests/custom_cluster/test_admission_controller.py
M www/admission_controller.tmpl
19 files changed, 776 insertions(+), 108 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/13740/13
--
To view, visit http://gerrit.cloudera.org:8080/13740
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21
Gerrit-Change-Number: 13740
Gerrit-PatchSet: 13
Gerrit-Owner: Bikramjeet Vig <bikramjeet....@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Lars Volker <l...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
