[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission Admission controller caps memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. A config option 'clamp_query_mem_limit_backend_mem_limit' is also added to revert to the old behavior where queries requesting more memory than backend's admission limit get rejected. The memory requested by a query when MEM_LIMIT or MEM_LIMIT_EXECUTORS is set is also capped to the memory limit for admission on the backends. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Updated custom_cluster tests in test_admission_controller.py - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Reviewed-on: http://gerrit.cloudera.org:8080/19533 Reviewed-by: Abhishek Rawat Tested-by: Impala Public Jenkins --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M be/src/scheduling/scheduler-test-util.cc M tests/custom_cluster/test_admission_controller.py 12 files changed, 299 insertions(+), 92 deletions(-) Approvals: Abhishek Rawat: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 16 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 15: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 15 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 13 Mar 2023 07:52:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9139/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 15 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 13 Mar 2023 02:37:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 15: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9138/ -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 15 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 13 Mar 2023 01:56:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 15: Code-Review+2 Carry forward after be test case fix. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 15 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sun, 12 Mar 2023 20:42:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9138/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 15 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sun, 12 Mar 2023 20:43:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12608/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 15 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sun, 12 Mar 2023 20:32:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission Admission controller caps memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. A config option 'clamp_query_mem_limit_backend_mem_limit' is also added to revert to the old behavior where queries requesting more memory than backend's admission limit get rejected. The memory requested by a query when MEM_LIMIT or MEM_LIMIT_EXECUTORS is set is also capped to the memory limit for admission on the backends. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Updated custom_cluster tests in test_admission_controller.py - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M be/src/scheduling/scheduler-test-util.cc M tests/custom_cluster/test_admission_controller.py 12 files changed, 299 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/15 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 15 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 14: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9137/ -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 14 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sun, 12 Mar 2023 14:40:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9137/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 14 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sun, 12 Mar 2023 09:26:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12607/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 14 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sun, 12 Mar 2023 09:23:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 14: Code-Review+2 Python 3 related fixes in tests/custom_cluster/test_admission_controller.py Carry over +2 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 14 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sun, 12 Mar 2023 09:04:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission Admission controller caps memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. A config option 'clamp_query_mem_limit_backend_mem_limit' is also added to revert to the old behavior where queries requesting more memory than backend's admission limit get rejected. The memory requested by a query when MEM_LIMIT or MEM_LIMIT_EXECUTORS is set is also capped to the memory limit for admission on the backends. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Updated custom_cluster tests in test_admission_controller.py - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M tests/custom_cluster/test_admission_controller.py 11 files changed, 298 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/14 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 14 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 13: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9136/ -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 13 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sun, 12 Mar 2023 07:09:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9136/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 13 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sun, 12 Mar 2023 01:56:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 13: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9135/ -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 13 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sun, 12 Mar 2023 01:35:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9135/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 13 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sat, 11 Mar 2023 20:22:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 13 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sat, 11 Mar 2023 20:22:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 12: Code-Review+2 LGTM -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 12 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sat, 11 Mar 2023 16:37:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 12: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 12 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 10 Mar 2023 18:38:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12605/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 12 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 10 Mar 2023 17:12:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12604/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 11 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 10 Mar 2023 17:07:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission Admission controller caps memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. A config option 'clamp_query_mem_limit_backend_mem_limit' is also added to revert to the old behavior where queries requesting more memory than backend's admission limit get rejected. The memory requested by a query when MEM_LIMIT or MEM_LIMIT_EXECUTORS is set is also capped to the memory limit for admission on the backends. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Updated custom_cluster tests in test_admission_controller.py - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M tests/custom_cluster/test_admission_controller.py 11 files changed, 298 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/12 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 12 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission Admission controller caps memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. A config option 'clamp_query_mem_limit_backend_mem_limit' is also added to revert to the old behavior where queries requesting more memory than backend's admission limit get rejected. The memory requested by a query when MEM_LIMIT or MEM_LIMIT_EXECUTORS is set is also capped to the memory limit for admission on the backends. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Updated custom_cluster tests in test_admission_controller.py - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M tests/custom_cluster/test_admission_controller.py 11 files changed, 300 insertions(+), 93 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/11 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 11 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 10: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 10 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Mar 2023 03:24:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12593/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 10 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Mar 2023 03:07:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission Admission controller caps memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. A config option 'clamp_query_mem_limit_backend_mem_limit' is also added to revert to the old behavior where queries requesting more memory than backend's admission limit get rejected. The memory requested by a query when MEM_LIMIT or MEM_LIMIT_EXECUTORS is set is also capped to the memory limit for admission on the backends. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Updated custom_cluster tests in test_admission_controller.py - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M tests/custom_cluster/test_admission_controller.py 11 files changed, 298 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/10 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 10 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 9: (2 comments) http://gerrit.cloudera.org:8080/#/c/19533/8/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/19533/8/tests/custom_cluster/test_admission_controller.py@491 PS8, Line 491: > flake8: E131 continuation line unaligned for hanging indent Done http://gerrit.cloudera.org:8080/#/c/19533/8/tests/custom_cluster/test_admission_controller.py@517 PS8, Line 517: > flake8: E131 continuation line unaligned for hanging indent Done -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 9 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Mar 2023 02:47:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 9: Please fix the warning from Jenkins. Others look good to me. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 9 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Mar 2023 02:30:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12592/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 9 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Mar 2023 02:03:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12591/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 8 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Mar 2023 01:58:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission Admission controller caps memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. A config option 'clamp_query_mem_limit_backend_mem_limit' is also added to revert to the old behavior where queries requesting more memory than backend's admission limit get rejected. The memory requested by a query when MEM_LIMIT or MEM_LIMIT_EXECUTORS is set is also capped to the memory limit for admission on the backends. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Updated custom_cluster tests in test_admission_controller.py - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M tests/custom_cluster/test_admission_controller.py 11 files changed, 298 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/9 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 9 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 9: (2 comments) http://gerrit.cloudera.org:8080/#/c/19533/9/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/19533/9/tests/custom_cluster/test_admission_controller.py@491 PS9, Line 491: + flake8: E131 continuation line unaligned for hanging indent http://gerrit.cloudera.org:8080/#/c/19533/9/tests/custom_cluster/test_admission_controller.py@517 PS9, Line 517: + flake8: E131 continuation line unaligned for hanging indent -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 9 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Mar 2023 01:43:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 8: (2 comments) http://gerrit.cloudera.org:8080/#/c/19533/8/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/19533/8/tests/custom_cluster/test_admission_controller.py@491 PS8, Line 491: + flake8: E131 continuation line unaligned for hanging indent http://gerrit.cloudera.org:8080/#/c/19533/8/tests/custom_cluster/test_admission_controller.py@517 PS8, Line 517: + flake8: E131 continuation line unaligned for hanging indent -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 8 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Mar 2023 01:38:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission Admission controller caps memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. A config option 'clamp_query_mem_limit_backend_mem_limit' is also added to revert to the old behavior where queries requesting more memory than backend's admission limit get rejected. The memory requested by a query when MEM_LIMIT or MEM_LIMIT_EXECUTORS is set is also capped to the memory limit for admission on the backends. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Updated custom_cluster tests in test_admission_controller.py - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M tests/custom_cluster/test_admission_controller.py 11 files changed, 298 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/8 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 8 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 7: (3 comments) http://gerrit.cloudera.org:8080/#/c/19533/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19533/7//COMMIT_MSG@9 PS7, Line 9: admission controller caps memory estimates for a given query to > Nit: Admission Done http://gerrit.cloudera.org:8080/#/c/19533/7/be/src/scheduling/schedule-state.h File be/src/scheduling/schedule-state.h: http://gerrit.cloudera.org:8080/#/c/19533/7/be/src/scheduling/schedule-state.h@288 PS7, Line 288: /// admitted based on the pool configuration passed to it. Must be called at least once > Nit: update the comment to cover the new parameters Done http://gerrit.cloudera.org:8080/#/c/19533/7/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/19533/7/tests/custom_cluster/test_admission_controller.py@504 PS7, Line 504: # A bit too much memory to run on coordinator. mem_limit will be clamped > Are we losing coverage on the "A bit too much memory to run on coordinator" Added a new test case for testing the 'clamp_query_mem_limit_backend_mem_limit' flag. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 7 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Mar 2023 01:37:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 7: Code-Review+1 (3 comments) I've read through and this looks good http://gerrit.cloudera.org:8080/#/c/19533/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19533/7//COMMIT_MSG@9 PS7, Line 9: admission controller caps memory estimates for a given query to Nit: Admission http://gerrit.cloudera.org:8080/#/c/19533/7/be/src/scheduling/schedule-state.h File be/src/scheduling/schedule-state.h: http://gerrit.cloudera.org:8080/#/c/19533/7/be/src/scheduling/schedule-state.h@288 PS7, Line 288: /// admitted based on the pool configuration passed to it. Must be called at least once Nit: update the comment to cover the new parameters http://gerrit.cloudera.org:8080/#/c/19533/7/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/19533/7/tests/custom_cluster/test_admission_controller.py@504 PS7, Line 504: # A bit too much memory to run on coordinator. mem_limit will be clamped Are we losing coverage on the "A bit too much memory to run on coordinator" case? I suppose we could ruin with -clamp_query_mem_limit_backend_mem_limit=false to get the old behavior. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 7 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 08 Mar 2023 01:16:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 7: (1 comment) Thanks for the feedback and review. http://gerrit.cloudera.org:8080/#/c/19533/7/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/19533/7/be/src/scheduling/admission-controller.cc@68 PS7, Line 68: However, if this flag is set, such a query gets " : "admitted with backend's memory limit > This flag is set by default. It seems like a behavior change for query admi Yeah we can document it for sure. However, it's only applicable for the case when resource pool's max query mem_limit is either not set or set to a value higher than backend's mem limit for admission. Basically, in most cases if pool configs are set properly we won't even run into this change of behavior. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 7 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 07 Mar 2023 20:02:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 7: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/19533/7/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/19533/7/be/src/scheduling/admission-controller.cc@68 PS7, Line 68: However, if this flag is set, such a query gets " : "admitted with backend's memory limit This flag is set by default. It seems like a behavior change for query admission. Should we document it? -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 7 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 07 Mar 2023 18:11:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12546/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 7 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 06 Mar 2023 01:17:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12545/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 6 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 06 Mar 2023 01:06:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission admission controller caps memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. A config option 'clamp_query_mem_limit_backend_mem_limit' is also added to revert to the old behavior where queries requesting more memory than backend's admission limit get rejected. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Updated custom_cluster tests in test_admission_controller.py - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M tests/custom_cluster/test_admission_controller.py 11 files changed, 266 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/7 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 7 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has posted comments on this change. (
http://gerrit.cloudera.org:8080/19533 )
Change subject: IMPALA-11858: Cap per backend memory estimate to its memory
limit for admission
..
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19533/6/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/19533/6/be/src/scheduling/admission-controller.cc@
PS6, Line : } else if (be_state.be_desc.admit_mem_limit() <
min_executor_admit_mem_limit.second) {
> line too long (92 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/19533/6/tests/custom_cluster/test_admission_controller.py
File tests/custom_cluster/test_admission_controller.py:
http://gerrit.cloudera.org:8080/#/c/19533/6/tests/custom_cluster/test_admission_controller.py@700
PS6, Line 700: +
> flake8: W504 line break after binary operator
Done
--
To view, visit http://gerrit.cloudera.org:8080/19533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c
Gerrit-Change-Number: 19533
Gerrit-PatchSet: 6
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Mon, 06 Mar 2023 01:01:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has posted comments on this change. (
http://gerrit.cloudera.org:8080/19533 )
Change subject: IMPALA-11858: Cap per backend memory estimate to its memory
limit for admission
..
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.cc
File be/src/scheduling/executor-group.cc:
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.cc@100
PS4, Line 100: if (per_executor_admit_mem_limit_ > 0) {
: per_executor_admit_mem_limit_ =
: std::min(be_desc.admit_mem_limit(),
per_executor_admit_mem_limit_);
: } else if (per_executor_admit_mem_limit_ == 0) {
: per_executor_admit_mem_limit_ = be_desc.admit_mem_limit();
: } else {
: DCHECK(false) << "Invalid negative value of
per_executor_admit_mem_limit_ "
: << per_executor_admit_mem_limit_;
: }
> I see.
Done. be_desc.admit_mem_limit() must be greater than 0.
--
To view, visit http://gerrit.cloudera.org:8080/19533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c
Gerrit-Change-Number: 19533
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Mon, 06 Mar 2023 00:46:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/19533 )
Change subject: IMPALA-11858: Cap per backend memory estimate to its memory
limit for admission
..
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19533/6/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/19533/6/be/src/scheduling/admission-controller.cc@
PS6, Line : } else if (be_state.be_desc.admit_mem_limit() <
min_executor_admit_mem_limit.second) {
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19533/6/tests/custom_cluster/test_admission_controller.py
File tests/custom_cluster/test_admission_controller.py:
http://gerrit.cloudera.org:8080/#/c/19533/6/tests/custom_cluster/test_admission_controller.py@700
PS6, Line 700: +
flake8: W504 line break after binary operator
--
To view, visit http://gerrit.cloudera.org:8080/19533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c
Gerrit-Change-Number: 19533
Gerrit-PatchSet: 6
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Mon, 06 Mar 2023 00:46:35 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission admission controller caps memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. A config option 'clamp_query_mem_limit_backend_mem_limit' is also added to revert to the old behavior where queries requesting more memory than backend's admission limit get rejected. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Updated custom_cluster tests in test_admission_controller.py - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M tests/custom_cluster/test_admission_controller.py 11 files changed, 265 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/6 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 6 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/19533 )
Change subject: IMPALA-11858: Cap per backend memory estimate to its memory
limit for admission
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.cc
File be/src/scheduling/executor-group.cc:
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.cc@100
PS4, Line 100: if (per_executor_admit_mem_limit_ > 0) {
: per_executor_admit_mem_limit_ =
: std::min(be_desc.admit_mem_limit(),
per_executor_admit_mem_limit_);
: } else if (per_executor_admit_mem_limit_ == 0) {
: per_executor_admit_mem_limit_ = be_desc.admit_mem_limit();
: } else {
: DCHECK(false) << "Invalid negative value of
per_executor_admit_mem_limit_ "
: << per_executor_admit_mem_limit_;
: }
> When adding the first executor, per_executor_admit_mem_limit_ is 0 so need
I see.
How about move DCHECK to line #100 as DCHECK(be_desc.admit_mem_limit() >= 0) ?
If group size is 1 and be_desc.admit_mem_limit() < 0, DCHECK will not be hit.
--
To view, visit http://gerrit.cloudera.org:8080/19533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c
Gerrit-Change-Number: 19533
Gerrit-PatchSet: 5
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Fri, 03 Mar 2023 07:11:09 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12529/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 03 Mar 2023 06:27:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission admission controller was capping memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h 10 files changed, 201 insertions(+), 42 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/5 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has posted comments on this change. (
http://gerrit.cloudera.org:8080/19533 )
Change subject: IMPALA-11858: Cap per backend memory estimate to its memory
limit for admission
..
Patch Set 4:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/admission-controller-test.cc
File be/src/scheduling/admission-controller-test.cc:
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/admission-controller-test.cc@20
PS4, Line 20: #include "scheduling/cluster-membership-test-util.h"
> nit: move to line 28
Done
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/admission-controller-test.cc@110
PS4, Line 110: Scheduler::ExecutorConfig group_config = {group.first,
group.second};
> it's not necessary to define group_config, we can get mem_limit_for_admin f
Done
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/admission-controller-test.cc@139
PS4, Line 139: }
> should we check if coordinator is not found?
Done
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/admission-controller.cc@1914
PS4, Line 1914: const Scheduler::ExecutorConfig group_config
> it's not necessary to define group_config, we can get mem_limit_for_admin f
Done
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.h
File be/src/scheduling/executor-group.h:
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.h@130
PS4, Line 130: int64_t
> nit: don't need return value
Done
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.h@152
PS4, Line 152: memory limit
> nit: memory limit in bytes
Done
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.h@153
PS4, Line 153: /// controller to bound the per backend memory limit for a
given query
> add '.' in the end of sentence
Done
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.cc
File be/src/scheduling/executor-group.cc:
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.cc@100
PS4, Line 100: if (per_executor_admit_mem_limit_ > 0) {
: per_executor_admit_mem_limit_ =
: std::min(be_desc.admit_mem_limit(),
per_executor_admit_mem_limit_);
: } else if (per_executor_admit_mem_limit_ == 0) {
: per_executor_admit_mem_limit_ = be_desc.admit_mem_limit();
: } else {
: DCHECK(false) << "Invalid negative value of
per_executor_admit_mem_limit_ "
: << per_executor_admit_mem_limit_;
: }
> Simplify as:
When adding the first executor, per_executor_admit_mem_limit_ is 0 so need
special case for handling that.
Empty executor group is a valid case for coordinator only queries and in such
cases GetPerExecutorMemLimitForAdmission() should return 0.
--
To view, visit http://gerrit.cloudera.org:8080/19533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c
Gerrit-Change-Number: 19533
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Fri, 03 Mar 2023 06:06:23 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/19533 )
Change subject: IMPALA-11858: Cap per backend memory estimate to its memory
limit for admission
..
Patch Set 4:
(8 comments)
Looks good to me. Just some minor comments.
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/admission-controller-test.cc
File be/src/scheduling/admission-controller-test.cc:
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/admission-controller-test.cc@20
PS4, Line 20: #include "scheduling/cluster-membership-test-util.h"
nit: move to line 28
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/admission-controller-test.cc@110
PS4, Line 110: Scheduler::ExecutorConfig group_config = {group.first,
group.second};
it's not necessary to define group_config, we can get mem_limit_for_admin from
executor group and coordinator directly when call
state->UpdateMemoryRequirements().
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/admission-controller-test.cc@139
PS4, Line 139: }
should we check if coordinator is not found?
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/admission-controller.cc@1914
PS4, Line 1914: const Scheduler::ExecutorConfig group_config
it's not necessary to define group_config, we can get mem_limit_for_admin from
executor group and coordinator directly when call
state->UpdateMemoryRequirements().
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.h
File be/src/scheduling/executor-group.h:
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.h@130
PS4, Line 130: int64_t
nit: don't need return value
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.h@152
PS4, Line 152: memory limit
nit: memory limit in bytes
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.h@153
PS4, Line 153: /// controller to bound the per backend memory limit for a
given query
add '.' in the end of sentence
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.cc
File be/src/scheduling/executor-group.cc:
http://gerrit.cloudera.org:8080/#/c/19533/4/be/src/scheduling/executor-group.cc@100
PS4, Line 100: if (per_executor_admit_mem_limit_ > 0) {
: per_executor_admit_mem_limit_ =
: std::min(be_desc.admit_mem_limit(),
per_executor_admit_mem_limit_);
: } else if (per_executor_admit_mem_limit_ == 0) {
: per_executor_admit_mem_limit_ = be_desc.admit_mem_limit();
: } else {
: DCHECK(false) << "Invalid negative value of
per_executor_admit_mem_limit_ "
: << per_executor_admit_mem_limit_;
: }
Simplify as:
DCHECK(be_desc.admit_mem_limit() >= 0) << "Invalid negative value of executor
admin_mem_limit" << be_desc.admit_mem_limit();
per_executor_admit_mem_limit_ = std::min(be_desc.admit_mem_limit(),
per_executor_admit_mem_limit_);
--
To view, visit http://gerrit.cloudera.org:8080/19533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c
Gerrit-Change-Number: 19533
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Fri, 03 Mar 2023 02:29:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12527/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 03 Mar 2023 01:31:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12526/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 03 Mar 2023 01:25:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission admission controller was capping memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M be/src/scheduling/scheduler.h 11 files changed, 218 insertions(+), 42 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/4 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission admission controller was capping memory estimates for a given query to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). With this patch, for a given query schedule, the Coordinator backend's memory limit is used for capping memory to admit on coordinator and min of all executor backend's memory limit is used for capping mem to admit on executors. Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Added new checks in existing tests in executor-group-test.cc - Ran exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/cluster-membership-test-util.h M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M be/src/scheduling/scheduler.h 11 files changed, 217 insertions(+), 42 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/3 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/19533 )
Change subject: IMPALA-11858: Cap per backend memory estimate to its memory
limit for admission
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/19533/3/be/src/scheduling/cluster-membership-test-util.cc
File be/src/scheduling/cluster-membership-test-util.cc:
http://gerrit.cloudera.org:8080/#/c/19533/3/be/src/scheduling/cluster-membership-test-util.cc@75
PS3, Line 75: BackendDescriptorPB MakeBackendDescriptor(int idx, int
port_offset, int64_t admit_mem_limit) {
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/19533/3/be/src/scheduling/executor-group-test.cc
File be/src/scheduling/executor-group-test.cc:
http://gerrit.cloudera.org:8080/#/c/19533/3/be/src/scheduling/executor-group-test.cc@70
PS3, Line 70: const BackendDescriptorPB& executor1 = MakeBackendDescriptor(1,
group1, /* port_offset=*/0,
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/19533/3/be/src/scheduling/executor-group-test.cc@74
PS3, Line 74: const BackendDescriptorPB& executor2 = MakeBackendDescriptor(2,
group1, /* port_offset=*/0,
line too long (93 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/19533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c
Gerrit-Change-Number: 19533
Gerrit-PatchSet: 3
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 03 Mar 2023 01:05:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/19533/2/be/src/scheduling/schedule-state.h File be/src/scheduling/schedule-state.h: http://gerrit.cloudera.org:8080/#/c/19533/2/be/src/scheduling/schedule-state.h@288 PS2, Line 288: The assumption is that all : /// executors have the same memory limit for admission and all the coordinators have the : /// same memory limit for admission. > Is the assumption working if one node is set as both executor and coordinat Thanks Wenzhe, that's a good point. I think we could have couple of cases: - Impala with dedicated coordinators and executors - Impala with all nodes acting as both coordinators and executors - Impala with some nodes acting as dedicated coordinators and dedicated executors while others acting as both I think the third case is a bit tricky to handle. Let me think about this a bit more and rework the patch. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Feb 2023 21:00:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/19533/2/be/src/scheduling/admission-controller-test.cc File be/src/scheduling/admission-controller-test.cc: http://gerrit.cloudera.org:8080/#/c/19533/2/be/src/scheduling/admission-controller-test.cc@877 PS2, Line 877: set_is_executor(false); If the node is set as executor and coordinator, the assumption mat not work. http://gerrit.cloudera.org:8080/#/c/19533/2/be/src/scheduling/schedule-state.h File be/src/scheduling/schedule-state.h: http://gerrit.cloudera.org:8080/#/c/19533/2/be/src/scheduling/schedule-state.h@288 PS2, Line 288: The assumption is that all : /// executors have the same memory limit for admission and all the coordinators have the : /// same memory limit for admission. Is the assumption working if one node is set as both executor and coordinator? -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Feb 2023 19:16:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12434/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Feb 2023 09:13:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12433/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Feb 2023 09:13:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission admission controller was capping memory estimates for the backends to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Also Running exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h 4 files changed, 37 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/2 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19533 ) Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/19533/1/be/src/scheduling/admission-controller-test.cc File be/src/scheduling/admission-controller-test.cc: http://gerrit.cloudera.org:8080/#/c/19533/1/be/src/scheduling/admission-controller-test.cc@912 PS1, Line 912: int64_t coordinator_admit_mem_limit = schedule_state->GetPerBackendMemLimitForAdmission(true); line too long (96 > 90) http://gerrit.cloudera.org:8080/#/c/19533/1/be/src/scheduling/admission-controller-test.cc@913 PS1, Line 913: int64_t executor_admit_mem_limit = schedule_state->GetPerBackendMemLimitForAdmission(false); line too long (94 > 90) http://gerrit.cloudera.org:8080/#/c/19533/1/be/src/scheduling/schedule-state.cc File be/src/scheduling/schedule-state.cc: http://gerrit.cloudera.org:8080/#/c/19533/1/be/src/scheduling/schedule-state.cc@329 PS1, Line 329: per_backend_mem_to_admit = min(per_backend_mem_to_admit, GetPerBackendMemLimitForAdmission(false)); line too long (101 > 90) http://gerrit.cloudera.org:8080/#/c/19533/1/be/src/scheduling/schedule-state.cc@330 PS1, Line 330: coord_backend_mem_to_admit = min(coord_backend_mem_to_admit, GetPerBackendMemLimitForAdmission(true)); line too long (104 > 90) -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Feb 2023 08:50:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Abhishek Rawat has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19533 Change subject: IMPALA-11858: Cap per backend memory estimate to its memory limit for admission .. IMPALA-11858: Cap per backend memory estimate to its memory limit for admission admission controller was capping memory estimates for the backends to its physical memory. The memory estimates should instead be capped to the backend's memory limit for admission, which is computed during daemon initialization in ExecEnv::Init(). Also fixed the issue related to excessive logging in query profiles when using global admission controller. If the query was queued the remote admission controller client was logging 'Queued' status in profile every time it checked the query status and it hadn't changed. Testing: - Updated existing unit tests in admission-controller-test.cc - Also Running exhaustive tests Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h 4 files changed, 32 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19533/1 -- To view, visit http://gerrit.cloudera.org:8080/19533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c Gerrit-Change-Number: 19533 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
