[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17683 ) Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 28 Jul 2021 05:33:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17683 ) Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. IMPALA-8762: Track host level admission stats across all coordinators This patch adds the ability to share the per-host stats for locally admitted queries across all coordinators. This helps to get a more consolidated view of the cluster for stats like slots_in_use and mem_admitted when making local admission decisions. Testing: Added e2e py test Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Reviewed-on: http://gerrit.cloudera.org:8080/17683 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M common/thrift/StatestoreService.thrift M common/thrift/metrics.json M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_catalog_wait.py M tests/custom_cluster/test_executor_groups.py M tests/custom_cluster/test_runtime_profile.py M tests/custom_cluster/test_scratch_disk.py 9 files changed, 261 insertions(+), 86 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 5 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17683 ) Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9193/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 27 Jul 2021 23:45:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17683 ) Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7354/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 27 Jul 2021 23:23:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17683 ) Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 27 Jul 2021 23:23:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/17683 ) Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. Patch Set 3: Code-Review+2 Carrying over Andrew's +2 -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 27 Jul 2021 23:22:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/17683 ) Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/17683/2/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/17683/2/be/src/scheduling/admission-controller.cc@257 PS2, Line 257: << "All admission topic key prefixes should be of the same size"; > Nit: "All admission topic keys" is better, as this may make it clearer for Done http://gerrit.cloudera.org:8080/#/c/17683/2/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/17683/2/tests/custom_cluster/test_executor_groups.py@573 PS2, Line 573: # identical but this will at least test that code path as a sanity check. > Nit: "at least" "code path" Done -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 27 Jul 2021 23:22:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Hello Andrew Sherman, Joe McDonnell, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17683 to look at the new patch set (#3). Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. IMPALA-8762: Track host level admission stats across all coordinators This patch adds the ability to share the per-host stats for locally admitted queries across all coordinators. This helps to get a more consolidated view of the cluster for stats like slots_in_use and mem_admitted when making local admission decisions. Testing: Added e2e py test Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 --- M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M common/thrift/StatestoreService.thrift M common/thrift/metrics.json M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_catalog_wait.py M tests/custom_cluster/test_executor_groups.py M tests/custom_cluster/test_runtime_profile.py M tests/custom_cluster/test_scratch_disk.py 9 files changed, 261 insertions(+), 86 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/17683/3 -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/17683 ) Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. Patch Set 2: Code-Review+2 (2 comments) LGTM http://gerrit.cloudera.org:8080/#/c/17683/2/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/17683/2/be/src/scheduling/admission-controller.cc@257 PS2, Line 257: << "Both admission topic keys prefixes should be of the same size"; Nit: "All admission topic keys" is better, as this may make it clearer for any future prefix additions. http://gerrit.cloudera.org:8080/#/c/17683/2/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/17683/2/tests/custom_cluster/test_executor_groups.py@573 PS2, Line 573: # identical but this will atleast test that codepath as a sanity check. Nit: "at least" "code path" -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sun, 25 Jul 2021 22:52:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17683 ) Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 21 Jul 2021 21:58:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17683 ) Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9128/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 20 Jul 2021 21:16:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/17683 ) Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. Patch Set 2: (8 comments) http://gerrit.cloudera.org:8080/#/c/17683/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17683/1//COMMIT_MSG@7 PS1, Line 7: coordinators > nit: coordinators Done http://gerrit.cloudera.org:8080/#/c/17683/1/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/17683/1/be/src/scheduling/admission-controller.h@440 PS1, Line 440: // This maps a backends's id(host/port id) to its host level statistics which are used > Nit: Its not a struct any more Done http://gerrit.cloudera.org:8080/#/c/17683/1/be/src/scheduling/admission-controller.h@442 PS1, Line 442: kends. > nit: what's the map key? host_id? Done http://gerrit.cloudera.org:8080/#/c/17683/1/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/17683/1/be/src/scheduling/admission-controller.cc@75 PS1, Line 75: // "". "!" is used because the backend id > Nit: is it clearer to say it is of the form? Done http://gerrit.cloudera.org:8080/#/c/17683/1/be/src/scheduling/admission-controller.cc@244 PS1, Line 244: // Parses the topic key to separate the prefix that helps recognize the kind of update > Nit: separate Done http://gerrit.cloudera.org:8080/#/c/17683/1/be/src/scheduling/admission-controller.cc@245 PS1, Line 245: // received. > Nit: received Done http://gerrit.cloudera.org:8080/#/c/17683/1/be/src/scheduling/admission-controller.cc@1652 PS1, Line 1652: if (topic_backend_id == host_id_) continue; > This is skipping updates about the current host? this skips the update that was added by the current host in its previous statestore update, since the current host's local version is more up to date. Similar to what we do in L1635 above http://gerrit.cloudera.org:8080/#/c/17683/1/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/17683/1/tests/custom_cluster/test_executor_groups.py@96 PS1, Line 96: self.num_impalads = 2 > why is this 2? this is a mis-nomer, it basically refers to the num of impalad instances that self._start_impala_cluster will look for. See L66 and L80. I'll update the naming of the variables to represent their true meaning. -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 20 Jul 2021 20:50:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Hello Andrew Sherman, Joe McDonnell, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17683 to look at the new patch set (#2). Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. IMPALA-8762: Track host level admission stats across all coordinators This patch adds the ability to share the per-host stats for locally admitted queries across all coordinators. This helps to get a more consolidated view of the cluster for stats like slots_in_use and mem_admitted when making local admission decisions. Testing: Added e2e py test Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 --- M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M common/thrift/StatestoreService.thrift M common/thrift/metrics.json M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_catalog_wait.py M tests/custom_cluster/test_executor_groups.py M tests/custom_cluster/test_runtime_profile.py M tests/custom_cluster/test_scratch_disk.py 9 files changed, 261 insertions(+), 86 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/17683/2 -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
