[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Riza Suminto has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. IMPALA-12005: Describe executor group set selection in query profile This patch adds new profile counters under the Frontend profile node to describe executor group set selection during query planning. It modifies FrontendProfile.java to allow one level of TRuntimeProfileNode nesting under the Frontend profile node. This makes it possible to group profile counters specific to each executor group set in consideration. "fragment-costs" hint is renamed to "segment-costs". A new "cpu-comparison-result" hint is added after "segment-costs" to help navigate how cpu sizing decision is made. This patch also adds some function overloading in runtime-profile.cc to hide TotalTime and InactiveTotalTime that is meaningless for anything under the Frontend profile node. Additional context also added into AnalysisException threw when none of the executor group sets fits the query requirement. This is how the Frontend profile node looks like after running TestExecutorGroups::test_query_cpu_count_divisor_fraction Frontend: Referenced Tables: tpcds_parquet.store_sales - CpuCountDivisor: 0.20 - ExecutorGroupsConsidered: 3 (3) Executor group 1 (root.tiny): Verdict: not enough cpu cores - CpuAsk: 15 (15) - CpuMax: 2 (2) - EffectiveParallelism: 3 (3) - MemoryAsk: 36.83 MB (38617088) - MemoryMax: 64.00 MB (67108864) Executor group 2 (root.small): Verdict: not enough cpu cores - CpuAsk: 25 (25) - CpuMax: 16 (16) - EffectiveParallelism: 5 (5) - MemoryAsk: 36.83 MB (38624004) - MemoryMax: 64.00 MB (67108864) Executor group 3 (root.large): Verdict: Match - CpuAsk: 35 (35) - CpuMax: 192 (192) - EffectiveParallelism: 7 (7) - MemoryAsk: 36.84 MB (38633570) - MemoryMax: 8388608.00 GB (9007199254740992) Testing: - Pass core tests Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Reviewed-on: http://gerrit.cloudera.org:8080/19628 Reviewed-by: Riza Suminto Tested-by: Riza Suminto --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test M tests/custom_cluster/test_executor_groups.py 12 files changed, 668 insertions(+), 474 deletions(-) Approvals: Riza Suminto: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 7 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 6: Verified+1 Code-Review+2 Carry votes from tip of changes. -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 12:47:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 22 Mar 2023 20:20:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12651/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Mar 2023 23:56:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Mar 2023 23:46:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/19628/4/be/src/util/runtime-profile.h File be/src/util/runtime-profile.h: http://gerrit.cloudera.org:8080/#/c/19628/4/be/src/util/runtime-profile.h@315 PS4, Line 315: RuntimeProfileBase(ObjectPool* pool, const std::string& name, : Counter* total_time_counter, Counter* inactive_timer, : bool add_default_counters = true); : > We could combine two overloaded functions as one by defining the default va Good point. Thanks for the reminder! http://gerrit.cloudera.org:8080/#/c/19628/4/fe/src/main/java/org/apache/impala/service/FrontendProfile.java File fe/src/main/java/org/apache/impala/service/FrontendProfile.java: http://gerrit.cloudera.org:8080/#/c/19628/4/fe/src/main/java/org/apache/impala/service/FrontendProfile.java@187 PS4, Line 187: Preconditions.checkState(profile_ != null, "already emitted profile"); > it's already called in addToCounter() and setToCounter() This is different from addToCounter() and setToCounter(). That one append counter into profile_. addChildrenProfile() intend to add child TRuntimeProfileNode under profile_. This still need to check that profile_ != null (that thrift profile has not been emitted). -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Mar 2023 23:38:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Hello Quanlong Huang, Abhishek Rawat, David Rorke, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19628 to look at the new patch set (#5). Change subject: IMPALA-12005: Describe executor group set selection in query profile .. IMPALA-12005: Describe executor group set selection in query profile This patch adds new profile counters under the Frontend profile node to describe executor group set selection during query planning. It modifies FrontendProfile.java to allow one level of TRuntimeProfileNode nesting under the Frontend profile node. This makes it possible to group profile counters specific to each executor group set in consideration. "fragment-costs" hint is renamed to "segment-costs". A new "cpu-comparison-result" hint is added after "segment-costs" to help navigate how cpu sizing decision is made. This patch also adds some function overloading in runtime-profile.cc to hide TotalTime and InactiveTotalTime that is meaningless for anything under the Frontend profile node. Additional context also added into AnalysisException threw when none of the executor group sets fits the query requirement. This is how the Frontend profile node looks like after running TestExecutorGroups::test_query_cpu_count_divisor_fraction Frontend: Referenced Tables: tpcds_parquet.store_sales - CpuCountDivisor: 0.20 - ExecutorGroupsConsidered: 3 (3) Executor group 1 (root.tiny): Verdict: not enough cpu cores - CpuAsk: 15 (15) - CpuMax: 2 (2) - EffectiveParallelism: 3 (3) - MemoryAsk: 36.83 MB (38617088) - MemoryMax: 64.00 MB (67108864) Executor group 2 (root.small): Verdict: not enough cpu cores - CpuAsk: 25 (25) - CpuMax: 16 (16) - EffectiveParallelism: 5 (5) - MemoryAsk: 36.83 MB (38624004) - MemoryMax: 64.00 MB (67108864) Executor group 3 (root.large): Verdict: Match - CpuAsk: 35 (35) - CpuMax: 192 (192) - EffectiveParallelism: 7 (7) - MemoryAsk: 36.84 MB (38633570) - MemoryMax: 8388608.00 GB (9007199254740992) Testing: - Pass core tests Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test M tests/custom_cluster/test_executor_groups.py 12 files changed, 668 insertions(+), 474 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/19628/5 -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/19628/4/be/src/util/runtime-profile.h File be/src/util/runtime-profile.h: http://gerrit.cloudera.org:8080/#/c/19628/4/be/src/util/runtime-profile.h@315 PS4, Line 315: RuntimeProfileBase(ObjectPool* pool, const std::string& name, : Counter* total_time_counter, Counter* inactive_timer); : RuntimeProfileBase(ObjectPool* pool, const std::string& name, : Counter* total_time_counter, Counter* inactive_timer, bool add_default_counters); We could combine two overloaded functions as one by defining the default value of "add_default_counters" as true. http://gerrit.cloudera.org:8080/#/c/19628/4/fe/src/main/java/org/apache/impala/service/FrontendProfile.java File fe/src/main/java/org/apache/impala/service/FrontendProfile.java: http://gerrit.cloudera.org:8080/#/c/19628/4/fe/src/main/java/org/apache/impala/service/FrontendProfile.java@187 PS4, Line 187: Preconditions.checkState(profile_ != null, "already emitted profile"); it's already called in addToCounter() and setToCounter() -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sun, 19 Mar 2023 07:24:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12642/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 17 Mar 2023 00:27:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12641/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 17 Mar 2023 00:16:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/19628 )
Change subject: IMPALA-12005: Describe executor group set selection in query
profile
..
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19628/3/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/19628/3/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@744
PS3, Line 744: builder.append(" (sum children))]")
> Missing one closing parenthesis.
Done
--
To view, visit http://gerrit.cloudera.org:8080/19628
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a
Gerrit-Change-Number: 19628
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 17 Mar 2023 00:09:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Hello Quanlong Huang, Abhishek Rawat, David Rorke, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19628 to look at the new patch set (#4). Change subject: IMPALA-12005: Describe executor group set selection in query profile .. IMPALA-12005: Describe executor group set selection in query profile This patch adds new profile counters under the Frontend profile node to describe executor group set selection during query planning. It modifies FrontendProfile.java to allow one level of TRuntimeProfileNode nesting under the Frontend profile node. This makes it possible to group profile counters specific to each executor group set in consideration. "fragment-costs" hint is renamed to "segment-costs". A new "cpu-comparison-result" hint is added after "segment-costs" to help navigate how cpu sizing decision is made. This patch also adds some function overloading in runtime-profile.cc to hide TotalTime and InactiveTotalTime that is meaningless for anything under the Frontend profile node. Additional context also added into AnalysisException threw when none of the executor group sets fits the query requirement. This is how the Frontend profile node looks like after running TestExecutorGroups::test_query_cpu_count_divisor_fraction Frontend: Referenced Tables: tpcds_parquet.store_sales - CpuCountDivisor: 0.20 - ExecutorGroupsConsidered: 3 (3) Executor group 1 (root.tiny): Verdict: not enough cpu cores - CpuAsk: 15 (15) - CpuMax: 2 (2) - EffectiveParallelism: 3 (3) - MemoryAsk: 36.83 MB (38617088) - MemoryMax: 64.00 MB (67108864) Executor group 2 (root.small): Verdict: not enough cpu cores - CpuAsk: 25 (25) - CpuMax: 16 (16) - EffectiveParallelism: 5 (5) - MemoryAsk: 36.83 MB (38624004) - MemoryMax: 64.00 MB (67108864) Executor group 3 (root.large): Verdict: Match - CpuAsk: 35 (35) - CpuMax: 192 (192) - EffectiveParallelism: 7 (7) - MemoryAsk: 36.84 MB (38633570) - MemoryMax: 8388608.00 GB (9007199254740992) Testing: - Pass core tests Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test M tests/custom_cluster/test_executor_groups.py 12 files changed, 678 insertions(+), 466 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/19628/4 -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/19628 )
Change subject: IMPALA-12005: Describe executor group set selection in query
profile
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19628/3/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/19628/3/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@744
PS3, Line 744: builder.append(" (sum children)]");
Missing one closing parenthesis.
--
To view, visit http://gerrit.cloudera.org:8080/19628
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a
Gerrit-Change-Number: 19628
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 17 Mar 2023 00:05:06 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Hello Quanlong Huang, Abhishek Rawat, David Rorke, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19628 to look at the new patch set (#3). Change subject: IMPALA-12005: Describe executor group set selection in query profile .. IMPALA-12005: Describe executor group set selection in query profile This patch adds new profile counters under the Frontend profile node to describe executor group set selection during query planning. It modifies FrontendProfile.java to allow one level of TRuntimeProfileNode nesting under the Frontend profile node. This makes it possible to group profile counters specific to each executor group set in consideration. "fragment-costs" hint is renamed to "segment-costs". A new "cpu-comparison-result" hint is added after "segment-costs" to help navigate how cpu sizing decision is made. This patch also adds some function overloading in runtime-profile.cc to hide TotalTime and InactiveTotalTime that is meaningless for anything under the Frontend profile node. Additional context also added into AnalysisException threw when none of the executor group sets fits the query requirement. This is how the Frontend profile node looks like after running TestExecutorGroups::test_query_cpu_count_divisor_fraction Frontend: Referenced Tables: tpcds_parquet.store_sales - CpuCountDivisor: 0.20 - ExecutorGroupsConsidered: 3 (3) Executor group 1 (root.tiny): Verdict: not enough cpu cores - CpuAsk: 15 (15) - CpuMax: 2 (2) - EffectiveParallelism: 3 (3) - MemoryAsk: 36.83 MB (38617088) - MemoryMax: 64.00 MB (67108864) Executor group 2 (root.small): Verdict: not enough cpu cores - CpuAsk: 25 (25) - CpuMax: 16 (16) - EffectiveParallelism: 5 (5) - MemoryAsk: 36.83 MB (38624004) - MemoryMax: 64.00 MB (67108864) Executor group 3 (root.large): Verdict: Match - CpuAsk: 35 (35) - CpuMax: 192 (192) - EffectiveParallelism: 7 (7) - MemoryAsk: 36.84 MB (38633570) - MemoryMax: 8388608.00 GB (9007199254740992) Testing: - Pass core tests Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test M tests/custom_cluster/test_executor_groups.py 12 files changed, 678 insertions(+), 466 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/19628/3 -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12640/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 16 Mar 2023 23:59:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/19628/1/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/19628/1/fe/src/main/java/org/apache/impala/service/Frontend.java@245 PS1, Line 245: private static final String EXECUTOR_GROUPS_CONSIDERED = "ExecutorGroupsConsidered > Nit: Make plural - "ExecutorGroupsConsidered" Done http://gerrit.cloudera.org:8080/#/c/19628/1/fe/src/main/java/org/apache/impala/service/Frontend.java@2102 PS1, Line 2102: addCounter(groupSetProfile, : new TCounter(EFFECTIVE_PARALLELISM, TUnit.UNIT, cores_ > Since the Effective Parallelism can vary for each group set/replan, maybe i Done -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 16 Mar 2023 23:40:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Hello Quanlong Huang, Abhishek Rawat, David Rorke, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19628 to look at the new patch set (#2). Change subject: IMPALA-12005: Describe executor group set selection in query profile .. IMPALA-12005: Describe executor group set selection in query profile This patch adds new profile counters under the Frontend profile node to describe executor group set selection during query planning. It modifies FrontendProfile.java to allow one level of TRuntimeProfileNode nesting under the Frontend profile node. This makes it possible to group profile counters specific to each executor group set in consideration. A new "cpu-comparison" hint is added besides "fragment-cost" to help navigate how cpu sizing decision is made. This patch also adds some function overloading in runtime-profile.cc to hide TotalTime and InactiveTotalTime that is meaningless for anything under the Frontend profile node. Additional context also added into AnalysisException threw when none of the executor group sets fits the query requirement. This is how the Frontend profile node looks like after running TestExecutorGroups::test_query_cpu_count_divisor_fraction Frontend: Referenced Tables: tpcds_parquet.store_sales - CpuCountDivisor: 0.20 - ExecutorGroupsConsidered: 3 (3) Executor group 1 (root.tiny): Verdict: not enough cpu cores - CpuAsk: 15 (15) - CpuMax: 2 (2) - EffectiveParallelism: 3 (3) - MemoryAsk: 36.83 MB (38617088) - MemoryMax: 64.00 MB (67108864) Executor group 2 (root.small): Verdict: not enough cpu cores - CpuAsk: 25 (25) - CpuMax: 16 (16) - EffectiveParallelism: 5 (5) - MemoryAsk: 36.83 MB (38624004) - MemoryMax: 64.00 MB (67108864) Executor group 3 (root.large): Verdict: Match - CpuAsk: 35 (35) - CpuMax: 192 (192) - EffectiveParallelism: 7 (7) - MemoryAsk: 36.84 MB (38633570) - MemoryMax: 8388608.00 GB (9007199254740992) Testing: - Pass core tests Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test M tests/custom_cluster/test_executor_groups.py 12 files changed, 418 insertions(+), 208 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/19628/2 -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
David Rorke has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/19628/1/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/19628/1/fe/src/main/java/org/apache/impala/service/Frontend.java@245 PS1, Line 245: private static final String EXECUTOR_GROUP_CONSIDERED = "ExecutorGroupConsidered"; Nit: Make plural - "ExecutorGroupsConsidered" http://gerrit.cloudera.org:8080/#/c/19628/1/fe/src/main/java/org/apache/impala/service/Frontend.java@2102 PS1, Line 2102: FrontendProfile.getCurrent().setToCounter( : EFFECTIVE_PARALLELISM, TUnit.UNIT, cores_requirement); Since the Effective Parallelism can vary for each group set/replan, maybe it would be better to add this as a counter to each groupSetProfile so the values get displayed separately for each group? We already do that for CpuAsk so maybe it's redundant to print both for each group but overall I think it would be clearer. The current approach of just printing EffectiveParallelism once at the top suggests that the value is the same for all groups which is misleading. -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 16 Mar 2023 23:13:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12635/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 16 Mar 2023 15:28:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19628 Change subject: IMPALA-12005: Describe executor group set selection in query profile .. IMPALA-12005: Describe executor group set selection in query profile This patch adds new profile counters under Frontend profile node to describe executor group set selection during query planning. It modifies FrontendProfile.java to allow one level of TRuntimeProfileNode nesting under Frontend profile node. This makes it possible to group profile counters specific to individual executor group set in consideration. This patch also adds some function overloading in runtime-profile.cc to hide TotalTime and InactiveTotalTime that is meaningless for anything under Frontend profile node. Additional context also added into AnalysisException thrown when none of executor group set fits the query requirement. This is how Frontend profile node look like after running TestExecutorGroups::test_query_cpu_count_divisor_fraction Frontend: Referenced Tables: tpcds_parquet.store_sales - CpuCountDivisor: 0.20 - EffectiveParallelism: 7 (7) - ExecutorGroupConsidered: 3 (3) Executor group 1 (root.tiny): Verdict: not enough cpu cores - CpuAsk: 15 (15) - CpuMax: 2 (2) - MemoryAsk: 36.83 MB (38617088) - MemoryMax: 64.00 MB (67108864) Executor group 2 (root.small): Verdict: not enough cpu cores - CpuAsk: 25 (25) - CpuMax: 16 (16) - MemoryAsk: 36.83 MB (38624004) - MemoryMax: 64.00 MB (67108864) Executor group 3 (root.large): Verdict: Match - CpuAsk: 35 (35) - CpuMax: 192 (192) - MemoryAsk: 36.84 MB (38633570) - MemoryMax: 8388608.00 GB (9007199254740992) Testing: - Pass core tests Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M tests/custom_cluster/test_executor_groups.py 10 files changed, 249 insertions(+), 53 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/19628/1 -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto
