[
https://issues.apache.org/jira/browse/HIVE-27751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809764#comment-17809764
]
Ramesh Kumar Thangarajan commented on HIVE-27751:
-------------------------------------------------
Hi [~zabetak]
Thank you very much for reviewing this. I have updated the description with the
sample output.
Usually the debug logs are all spread across multiple places and we do not have
a easy way to get the details from user when they run into performance issues.
As part of this PR, main idea is to output the information in the command line
output too. This will be done only if the config is turned on. That is what I
meant by accumulated as we get all the details related to Query Compilation at
one single place and its visible to the user as part of the query output.
Also I have addressed your comments, can you let me know what you think about
the latest patch?
> Log Query Compilation summary in an accumulated way
> ---------------------------------------------------
>
> Key: HIVE-27751
> URL: https://issues.apache.org/jira/browse/HIVE-27751
> Project: Hive
> Issue Type: Task
> Components: Hive
> Reporter: Ramesh Kumar Thangarajan
> Assignee: Ramesh Kumar Thangarajan
> Priority: Major
> Labels: pull-request-available
>
> Query Compilation summary is very useful for reading and collecting all the
> measures of compile time in a single place. It is also useful in debugging a
> performance issue in the query compilation phase and also to report and
> compare with various runs
> In order to run test this. Please set the config hive.compile.print.summary
> to true in any q file and run the test to see the Query Compilation Summary
> in the logs. One example of the output is below. The order of operations are
> maintained while print the summary too:
> {code:java}
> Query Compilation Summary
> ----------------------------------------------------------------------------------------------
> waitCompile
> 0 ms
> parse
> 4 ms
> getTableConstraints - HS2-cache
> 69 ms
> optimizer - Calcite: Plan generation
> 257 ms
> optimizer - Calcite: Prejoin ordering transformation
> 20 ms
> optimizer - Calcite: Postjoin ordering transformation
> 24 ms
> optimizer
> 705 ms
> optimizer - HiveOpConverterPostProc
> 0 ms
> optimizer - Generator
> 24 ms
> optimizer - PartitionColumnsSeparator
> 1 ms
> optimizer - SyntheticJoinPredicate
> 2 ms
> optimizer - SimplePredicatePushDown
> 8 ms
> optimizer - RedundantDynamicPruningConditionsRemoval
> 0 ms
> optimizer - SortedDynPartitionTimeGranularityOptimizer
> 2 ms
> optimizer - PartitionPruner
> 3 ms
> optimizer - PartitionConditionRemover
> 2 ms
> optimizer - GroupByOptimizer
> 2 ms
> optimizer - ColumnPruner
> 10 ms
> optimizer - CountDistinctRewriteProc
> 1 ms
> optimizer - SamplePruner
> 1 ms
> optimizer - MapJoinProcessor
> 2 ms
> optimizer - BucketingSortingReduceSinkOptimizer
> 2 ms
> optimizer - UnionProcessor
> 2 ms
> optimizer - JoinReorder
> 0 ms
> optimizer - FixedBucketPruningOptimizer
> 2 ms
> optimizer - BucketVersionPopulator
> 2 ms
> optimizer - NonBlockingOpDeDupProc
> 1 ms
> optimizer - IdentityProjectRemover
> 0 ms
> optimizer - LimitPushdownOptimizer
> 2 ms
> optimizer - OrderlessLimitPushDownOptimizer
> 1 ms
> optimizer - StatsOptimizer
> 0 ms
> optimizer - SimpleFetchOptimizer
> 0 ms
> TezCompiler - Run top n key optimization
> 2 ms
> TezCompiler - Setup dynamic partition pruning
> 3 ms
> optimizer - Merge single column semi-join reducers to composite
> 0 ms
> partition-retrieving
> 1 ms
> TezCompiler - Setup stats in the operator plan
> 78 ms
> TezCompiler - Sorted dynamic partition optimization
> 3 ms
> TezCompiler - Reduce Sink de-duplication
> 4 ms
> TezCompiler - Run the optimizations that use stats for optimization
> 5 ms
> TezCompiler - Run reduce sink after join algorithm selection
> 2 ms
> TezCompiler - Run remove dynamic pruning by size
> 1 ms
> TezCompiler - Run cycle analysis for partition pruning
> 0 ms
> TezCompiler - Remove redundant semijoin reduction
> 1 ms
> TezCompiler - Shared scans optimization
> 13 ms
> TezCompiler - markOperatorsWithUnstableRuntimeStats
> 1 ms
> TezCompiler - generateTaskTree
> 31 ms
> TezCompiler - optimizeTaskPlan
> 156 ms
> TezCompiler
> 323 ms
> semanticAnalyze
> 2628 ms
> compile
> 2633 ms
> ----------------------------------------------------------------------------------------------
> {code}
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
