Hello Quanlong Huang, Abhishek Rawat, David Rorke, Wenzhe Zhou, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19628 to look at the new patch set (#5). Change subject: IMPALA-12005: Describe executor group set selection in query profile ...................................................................... IMPALA-12005: Describe executor group set selection in query profile This patch adds new profile counters under the Frontend profile node to describe executor group set selection during query planning. It modifies FrontendProfile.java to allow one level of TRuntimeProfileNode nesting under the Frontend profile node. This makes it possible to group profile counters specific to each executor group set in consideration. "fragment-costs" hint is renamed to "segment-costs". A new "cpu-comparison-result" hint is added after "segment-costs" to help navigate how cpu sizing decision is made. This patch also adds some function overloading in runtime-profile.cc to hide TotalTime and InactiveTotalTime that is meaningless for anything under the Frontend profile node. Additional context also added into AnalysisException threw when none of the executor group sets fits the query requirement. This is how the Frontend profile node looks like after running TestExecutorGroups::test_query_cpu_count_divisor_fraction Frontend: Referenced Tables: tpcds_parquet.store_sales - CpuCountDivisor: 0.20 - ExecutorGroupsConsidered: 3 (3) Executor group 1 (root.tiny): Verdict: not enough cpu cores - CpuAsk: 15 (15) - CpuMax: 2 (2) - EffectiveParallelism: 3 (3) - MemoryAsk: 36.83 MB (38617088) - MemoryMax: 64.00 MB (67108864) Executor group 2 (root.small): Verdict: not enough cpu cores - CpuAsk: 25 (25) - CpuMax: 16 (16) - EffectiveParallelism: 5 (5) - MemoryAsk: 36.83 MB (38624004) - MemoryMax: 64.00 MB (67108864) Executor group 3 (root.large): Verdict: Match - CpuAsk: 35 (35) - CpuMax: 192 (192) - EffectiveParallelism: 7 (7) - MemoryAsk: 36.84 MB (38633570) - MemoryMax: 8388608.00 GB (9007199254740992) Testing: - Pass core tests Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test M tests/custom_cluster/test_executor_groups.py 12 files changed, 668 insertions(+), 474 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/19628/5 -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Abhishek Rawat <ara...@cloudera.com> Gerrit-Reviewer: David Rorke <dro...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>