Tim Armstrong has uploaded a new patch set (#6) to the change originally created by Bikramjeet Vig. ( http://gerrit.cloudera.org:8080/13740 )
Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord ...................................................................... [WIP] IMPALA-7486: Prototype for admitting less mem on coord This patch computes two memory estimates in the frontend: an estimate for any host that is an executor (including a combined coordinator and executor) and an estimate for a dedicated coordinator. This is computed regardless of whether it is a dedicated coordinator or not. Admission control then, in the case when the coordinator is dedicated, uses the coordinator memory estimate for the coordinator node and the executor memory estimate for all other nodes. Complete: - Calculate the mem needed for the coord fragment + accounting for the runtime filter agg mem required + a safety buffer of 100MB - changes to get the mem_to_admit/limit for coord, accesible through the QuerySchedule - add/remove/modify checks in both reject and admit methods to seperate the coord and executor checks - Update memory accounting in admission controller - use coord's mem limit for the coord queryState - added BE unit tests for most cases - Updated the Admission debug page to show the coord mem values TODO: - add e2e tests: - Simple case (done) - query with num_node = 1 - Any additional test cases not covered by BE unit tests or need an equivalent e2e test - e2e tests are not written with dedicated coord, so all should work fine Testing: * Added unit test for admission/rejection in dedicated coordinator clusters. Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 --- M be/src/runtime/coordinator.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M be/src/service/impala-http-handler.cc M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M tests/custom_cluster/test_admission_controller.py M www/admission_controller.tmpl 14 files changed, 570 insertions(+), 96 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/13740/6 -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 6 Gerrit-Owner: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com> Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Lars Volker <l...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>