Riza Suminto has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/19628 )
Change subject: IMPALA-12005: Describe executor group set selection in query
profile
......................................................................
IMPALA-12005: Describe executor group set selection in query profile
This patch adds new profile counters under the Frontend profile node to
describe executor group set selection during query planning. It modifies
FrontendProfile.java to allow one level of TRuntimeProfileNode nesting
under the Frontend profile node. This makes it possible to group profile
counters specific to each executor group set in consideration.
"fragment-costs" hint is renamed to "segment-costs". A new
"cpu-comparison-result" hint is added after "segment-costs" to help
navigate how cpu sizing decision is made.
This patch also adds some function overloading in runtime-profile.cc to
hide TotalTime and InactiveTotalTime that is meaningless for anything
under the Frontend profile node. Additional context also added into
AnalysisException threw when none of the executor group sets fits the
query requirement.
This is how the Frontend profile node looks like after running
TestExecutorGroups::test_query_cpu_count_divisor_fraction
Frontend:
Referenced Tables: tpcds_parquet.store_sales
- CpuCountDivisor: 0.20
- ExecutorGroupsConsidered: 3 (3)
Executor group 1 (root.tiny):
Verdict: not enough cpu cores
- CpuAsk: 15 (15)
- CpuMax: 2 (2)
- EffectiveParallelism: 3 (3)
- MemoryAsk: 36.83 MB (38617088)
- MemoryMax: 64.00 MB (67108864)
Executor group 2 (root.small):
Verdict: not enough cpu cores
- CpuAsk: 25 (25)
- CpuMax: 16 (16)
- EffectiveParallelism: 5 (5)
- MemoryAsk: 36.83 MB (38624004)
- MemoryMax: 64.00 MB (67108864)
Executor group 3 (root.large):
Verdict: Match
- CpuAsk: 35 (35)
- CpuMax: 192 (192)
- EffectiveParallelism: 7 (7)
- MemoryAsk: 36.84 MB (38633570)
- MemoryMax: 8388608.00 GB (9007199254740992)
Testing:
- Pass core tests
Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a
Reviewed-on: http://gerrit.cloudera.org:8080/19628
Reviewed-by: Riza Suminto <riza.sumi...@cloudera.com>
Tested-by: Riza Suminto <riza.sumi...@cloudera.com>
---
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M be/src/util/runtime-profile.cc
M be/src/util/runtime-profile.h
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/FrontendProfile.java
M
testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test
M tests/custom_cluster/test_executor_groups.py
12 files changed, 668 insertions(+), 474 deletions(-)
Approvals:
Riza Suminto: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/19628
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a
Gerrit-Change-Number: 19628
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <riza.sumi...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ara...@cloudera.com>
Gerrit-Reviewer: David Rorke <dro...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
