Abhishek Rawat has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/19533 )
Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission ...................................................................... IMPALA-11858: Cap per backend memory estimate to its memory limit for admission Admission controller caps memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. A config option 'clamp_query_mem_limit_backend_mem_limit' is also added to revert to the old behavior where queries requesting more memory than backend's admission limit get rejected. The memory requested by a query when MEM_LIMIT or MEM_LIMIT_EXECUTORS is set is also capped to the memory limit for admission on the backends. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Updated custom_cluster tests in test_admission_controller.py - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M tests/custom_cluster/test_admission_controller.py 11 files changed, 298 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/8 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 8 Gerrit-Owner: Abhishek Rawat <ara...@cloudera.com> Gerrit-Reviewer: Abhishek Rawat <ara...@cloudera.com> Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Reviewer: Yida Wu <wydbaggio...@gmail.com>
