Riza Suminto has uploaded a new patch set (#30) to the change originally created by Qifan Chen. ( http://gerrit.cloudera.org:8080/19033 )
Change subject: IMPALA-11604 Planner changes for CPU usage ...................................................................... IMPALA-11604 Planner changes for CPU usage This patch augments IMPALA-10992 by establishing an infrastructure to allow weighted total amount of data to process to be used as a new factor in the definition and selection of an executor group. In this patch the weight component is set to 1. Two flavors of the weighted amount of data processed are enabled in this patch: the minimal amount and the maximal amount. The former is the processing cost along a path with the largest sum of processing costs, reflecting the best level of parallelism among fragments. The latter is the sum of that of every fragment, reflecting the worst level of parallelism. The geometric mean of the minimal and the maximal is used as the estimated processing cost of the query. A fragment's total amount of data processed is the sum of that of every node in the fragment. The weighted amount of data processed is computed with a general formula as follows. Processing cost is a pair: PC(D, N), where D = I * C * W where D is the weighted amount of data processed N is number of instances I is input cardinality C is expression evaluation cost per row, set to 1 W is average row size A description of the computation for each kind of plan node is given below. 1. Aggregation node: C and W are the sum of the costs and partial row widths for each AggregateInfo object. 2. AnalyticEval node: C is sum of the evaluation costs for analytic functions, partition by equal and order by equal predicate; 3. CardinalityCheck node: Both C and I are 1; 4. DataSource scan node: C is computed from a subset of the selection predicates excluding data source accepted predicates; 5. EmptySet node: I is 0; 6. Exchange node: A modification of the general formula when in broadcast mode: D = D * number of receivers; 7. Hash join node: probe side = PC(I0 * C(equi-join predicate) * W, N) + PC(output cardinality * C(other join predicate) * W, N) build side = PC(I1 * C(equi-join predicate) * W, N) 8. Hbase scan node: N is 1 9. Hdfs and Kudu scan node: N is mt_dop when query option mt_dop >= 1, otherwise N is number of nodes * max scan threads; 10. Nested loop join node: When the right child is not a SingularRowSrc node: probe side = PC(I0 * C(equi-join predicate) * W, N) + PC(output cardinality * C(other join predicate) * W, N) build side = PC(I1 * C(equi-join predicate) * W, N) When the right child is a SingularRowSrc node: probe side = PC(I0 * W, N) build side = PC(I0 * I1 * W, N) 11. Select node: Use the general formula; 12. SingularRowSrc node: Since the node is involved once per input in nested loop join, the contribution of this node is computed in nested loop join; 13. Sort node: C is the evalation cost for the sort expression and W is the width of the intermediate tuple being sorted; 14. Subplan node: C is 1. I is the multiplication of the cardinality of the left and the right child; 15. Union node: C is the cost of materializing rows from all non pass-through children. W is the width of all non pass-through children; 16. Unnest node: I is the cardinality of the containing subplan node and C is 1. The processing cost for the data sink of a fragment is also computed and added to that of the fragment. This patch also assumes that the number of instances of execution overlaps each other when there is a discrepancy of instances among nodes in a single fragment. For example when there are 6 scan thread and 3 aggregation threads in a single fragment, 3 threads are used for both scan and aggregate, and 3 other threads are used for the scan. As an example, the best and worst processing cost for TPCDS large query q14a and tiny q19 are as follows. Processing cost for query: q14a: Best case: total=1271804127, numInstances=9, perInstance=141311569 Worst case: total=5169193752, numInstances=12, perInstance=430766146 q19: Best case: total=1082950, numInstances=15, perInstance=72196 Worst case: total=1082950, numInstances=15, perInstance=72196 Testing: 1. Unit testing by examining the best and the worst processing cost computed for all plan nodes in all fragments for a small set of queries; 2. Core tests. Change-Id: If32dc770dfffcdd0be2b5555a789a7720952c68a --- M common/thrift/Frontend.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduTableSink.java M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/PlanRootSink.java M fe/src/main/java/org/apache/impala/planner/Planner.java A fe/src/main/java/org/apache/impala/planner/ProcessingCost.java M fe/src/main/java/org/apache/impala/planner/ResourceProfile.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/SelectNode.java M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M fe/src/main/java/org/apache/impala/planner/SubplanNode.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/planner/UnnestNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java A fe/src/main/java/org/apache/impala/util/AutoScaleUtil.java M fe/src/main/java/org/apache/impala/util/ExprUtil.java M fe/src/main/java/org/apache/impala/util/MathUtil.java 38 files changed, 743 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19033/30 -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If32dc770dfffcdd0be2b5555a789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 30 Gerrit-Owner: Qifan Chen <qfc...@hotmail.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qfc...@hotmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>