Tim Armstrong has uploaded a new patch set (#7) to the change originally
created by Bikramjeet Vig. ( http://gerrit.cloudera.org:8080/13740 )
Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord
......................................................................
[WIP] IMPALA-7486: Prototype for admitting less mem on coord
This patch computes two memory estimates in the frontend:
an estimate for any host that is an executor (including
a combined coordinator and executor) and an estimate
for a dedicated coordinator. This is computed regardless
of whether it is a dedicated coordinator or not.
Admission control then, in the case when the coordinator
is dedicated, uses the coordinator memory estimate for
the coordinator node and the executor memory estimate
for all other nodes.
Complete:
- Calculate the mem needed for the coord fragment + accounting
for the runtime filter agg mem required + a safety buffer of 100MB
- changes to get the mem_to_admit/limit for coord, accesible through
the QuerySchedule
- add/remove/modify checks in both reject and admit methods to
seperate the coord and executor checks
- Update memory accounting in admission controller
- use coord's mem limit for the coord queryState
- added BE unit tests for most cases
- Updated the Admission debug page to show the coord mem values
TODO:
- add e2e tests:
- Simple case (done)
- query with num_node = 1
- Any additional test cases not covered by BE unit tests
or need an equivalent e2e test
- e2e tests are not written with dedicated coord,
so all should work fine
Testing:
* Added unit test for admission/rejection in dedicated
coordinator clusters.
Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21
---
M be/src/runtime/coordinator.cc
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/query-schedule.cc
M be/src/scheduling/query-schedule.h
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/impala-http-handler.cc
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test
M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test
M
testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M
testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level0.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level1.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test
M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test
M tests/custom_cluster/test_admission_controller.py
M www/admission_controller.tmpl
27 files changed, 2,172 insertions(+), 1,318 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/13740/7
--
To view, visit http://gerrit.cloudera.org:8080/13740
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21
Gerrit-Change-Number: 13740
Gerrit-PatchSet: 7
Gerrit-Owner: Bikramjeet Vig <bikramjeet....@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Lars Volker <l...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
