[
https://issues.apache.org/jira/browse/HIVE-27751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramesh Kumar Thangarajan updated HIVE-27751:
--------------------------------------------
Description:
Query Compilation summary is very useful for reading and collecting all the
measures of compile time in a single place. It is also useful in debugging a
performance issue in the query compilation phase and also to report and compare
with various runs
In order to run test this. Please set the config hive.compile.print.summary to
true in any q file and run the test to see the Query Compilation Summary in the
logs. One example of the output is below. The order of operations are
maintained while print the summary too:
{code:java}
Query Compilation Summary
----------------------------------------------------------------------------------------------
waitCompile
0 ms
parse
4 ms
getTableConstraints - HS2-cache
69 ms
optimizer - Calcite: Plan generation
257 ms
optimizer - Calcite: Prejoin ordering transformation
20 ms
optimizer - Calcite: Postjoin ordering transformation
24 ms
optimizer
705 ms
optimizer - HiveOpConverterPostProc
0 ms
optimizer - Generator
24 ms
optimizer - PartitionColumnsSeparator
1 ms
optimizer - SyntheticJoinPredicate
2 ms
optimizer - SimplePredicatePushDown
8 ms
optimizer - RedundantDynamicPruningConditionsRemoval
0 ms
optimizer - SortedDynPartitionTimeGranularityOptimizer
2 ms
optimizer - PartitionPruner
3 ms
optimizer - PartitionConditionRemover
2 ms
optimizer - GroupByOptimizer
2 ms
optimizer - ColumnPruner
10 ms
optimizer - CountDistinctRewriteProc
1 ms
optimizer - SamplePruner
1 ms
optimizer - MapJoinProcessor
2 ms
optimizer - BucketingSortingReduceSinkOptimizer
2 ms
optimizer - UnionProcessor
2 ms
optimizer - JoinReorder
0 ms
optimizer - FixedBucketPruningOptimizer
2 ms
optimizer - BucketVersionPopulator
2 ms
optimizer - NonBlockingOpDeDupProc
1 ms
optimizer - IdentityProjectRemover
0 ms
optimizer - LimitPushdownOptimizer
2 ms
optimizer - OrderlessLimitPushDownOptimizer
1 ms
optimizer - StatsOptimizer
0 ms
optimizer - SimpleFetchOptimizer
0 ms
TezCompiler - Run top n key optimization
2 ms
TezCompiler - Setup dynamic partition pruning
3 ms
optimizer - Merge single column semi-join reducers to composite
0 ms
partition-retrieving
1 ms
TezCompiler - Setup stats in the operator plan
78 ms
TezCompiler - Sorted dynamic partition optimization
3 ms
TezCompiler - Reduce Sink de-duplication
4 ms
TezCompiler - Run the optimizations that use stats for optimization
5 ms
TezCompiler - Run reduce sink after join algorithm selection
2 ms
TezCompiler - Run remove dynamic pruning by size
1 ms
TezCompiler - Run cycle analysis for partition pruning
0 ms
TezCompiler - Remove redundant semijoin reduction
1 ms
TezCompiler - Shared scans optimization
13 ms
TezCompiler - markOperatorsWithUnstableRuntimeStats
1 ms
TezCompiler - generateTaskTree
31 ms
TezCompiler - optimizeTaskPlan
156 ms
TezCompiler
323 ms
semanticAnalyze
2628 ms
compile
2633 ms
----------------------------------------------------------------------------------------------
{code}
{{}}
was:
Query Compilation summary is very useful for reading and collecting all the
measures of compile time in a single place. It is also useful in debugging a
performance issue in the query compilation phase and also to report and compare
with various runs
After the
> Log Query Compilation summary in an accumulated way
> ---------------------------------------------------
>
> Key: HIVE-27751
> URL: https://issues.apache.org/jira/browse/HIVE-27751
> Project: Hive
> Issue Type: Task
> Components: Hive
> Reporter: Ramesh Kumar Thangarajan
> Assignee: Ramesh Kumar Thangarajan
> Priority: Major
> Labels: pull-request-available
>
> Query Compilation summary is very useful for reading and collecting all the
> measures of compile time in a single place. It is also useful in debugging a
> performance issue in the query compilation phase and also to report and
> compare with various runs
> In order to run test this. Please set the config hive.compile.print.summary
> to true in any q file and run the test to see the Query Compilation Summary
> in the logs. One example of the output is below. The order of operations are
> maintained while print the summary too:
> {code:java}
> Query Compilation Summary
> ----------------------------------------------------------------------------------------------
> waitCompile
> 0 ms
> parse
> 4 ms
> getTableConstraints - HS2-cache
> 69 ms
> optimizer - Calcite: Plan generation
> 257 ms
> optimizer - Calcite: Prejoin ordering transformation
> 20 ms
> optimizer - Calcite: Postjoin ordering transformation
> 24 ms
> optimizer
> 705 ms
> optimizer - HiveOpConverterPostProc
> 0 ms
> optimizer - Generator
> 24 ms
> optimizer - PartitionColumnsSeparator
> 1 ms
> optimizer - SyntheticJoinPredicate
> 2 ms
> optimizer - SimplePredicatePushDown
> 8 ms
> optimizer - RedundantDynamicPruningConditionsRemoval
> 0 ms
> optimizer - SortedDynPartitionTimeGranularityOptimizer
> 2 ms
> optimizer - PartitionPruner
> 3 ms
> optimizer - PartitionConditionRemover
> 2 ms
> optimizer - GroupByOptimizer
> 2 ms
> optimizer - ColumnPruner
> 10 ms
> optimizer - CountDistinctRewriteProc
> 1 ms
> optimizer - SamplePruner
> 1 ms
> optimizer - MapJoinProcessor
> 2 ms
> optimizer - BucketingSortingReduceSinkOptimizer
> 2 ms
> optimizer - UnionProcessor
> 2 ms
> optimizer - JoinReorder
> 0 ms
> optimizer - FixedBucketPruningOptimizer
> 2 ms
> optimizer - BucketVersionPopulator
> 2 ms
> optimizer - NonBlockingOpDeDupProc
> 1 ms
> optimizer - IdentityProjectRemover
> 0 ms
> optimizer - LimitPushdownOptimizer
> 2 ms
> optimizer - OrderlessLimitPushDownOptimizer
> 1 ms
> optimizer - StatsOptimizer
> 0 ms
> optimizer - SimpleFetchOptimizer
> 0 ms
> TezCompiler - Run top n key optimization
> 2 ms
> TezCompiler - Setup dynamic partition pruning
> 3 ms
> optimizer - Merge single column semi-join reducers to composite
> 0 ms
> partition-retrieving
> 1 ms
> TezCompiler - Setup stats in the operator plan
> 78 ms
> TezCompiler - Sorted dynamic partition optimization
> 3 ms
> TezCompiler - Reduce Sink de-duplication
> 4 ms
> TezCompiler - Run the optimizations that use stats for optimization
> 5 ms
> TezCompiler - Run reduce sink after join algorithm selection
> 2 ms
> TezCompiler - Run remove dynamic pruning by size
> 1 ms
> TezCompiler - Run cycle analysis for partition pruning
> 0 ms
> TezCompiler - Remove redundant semijoin reduction
> 1 ms
> TezCompiler - Shared scans optimization
> 13 ms
> TezCompiler - markOperatorsWithUnstableRuntimeStats
> 1 ms
> TezCompiler - generateTaskTree
> 31 ms
> TezCompiler - optimizeTaskPlan
> 156 ms
> TezCompiler
> 323 ms
> semanticAnalyze
> 2628 ms
> compile
> 2633 ms
> ----------------------------------------------------------------------------------------------
> {code}
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
