Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22091 )
Change subject: IMPALA-13531: Calcite CTE frontend ...................................................................... IMPALA-13531: Calcite CTE frontend Support identifying CTEs in the Calcite planner for single-node plans. CTE detection is based on HIVE-28259. Adds new logical nodes Sequence, CTEProducer, and CTEConsumer, matching constructs outlined in "Optimization of Common Table Expressions in MPP Database Systems" by El-Helw et al. Implements them with physical SequenceNode, CTEProducerNode, and CTEConsumerNode. After optimization steps and before converting to Impala physical plan nodes, runs a CTE program to identify CTE candidates and replace them with ImpalaCTEConsumer nodes using VolcanoPlanner with materializations. After materializing CTEs, identifies all CTEs from ImpalaCTEConsumers in the plan tree and constructs a top-level ImpalaSequence containing the plan tree and ImpalaCTEProducers to materialize CTE results for consumers. Adds cte_threshold query option to control when CTE replacement occurs. CTE replacement will only occur when number of occurences exceeds the threshold. Defaults to -1 so CTEs are disabled. Adds cte_suggester_class command-line option to enable providing a different Suggester implementation. Defaults to Calcite's RelCommonExpressionBasicSuggester. Adds basic explain tests. Change-Id: Id0840c0859d2fe25628d799a18d302cee1eb36e8 Reviewed-on: http://gerrit.cloudera.org:8080/22091 Reviewed-by: Joe McDonnell <[email protected]> Tested-by: Joe McDonnell <[email protected]> --- M be/src/service/frontend.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift M common/thrift/Query.thrift A fe/src/main/java/org/apache/impala/planner/CTEConsumerNode.java A fe/src/main/java/org/apache/impala/planner/CTEProducerNode.java A fe/src/main/java/org/apache/impala/planner/SequenceNode.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/coercenodes/CoerceNodes.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaCTEConsumer.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaCTEProducer.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaSequence.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/NodeCreationUtils.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/CTERuleConfig.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/CombineValuesNodesRule.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ImpalaJoinProjectTransposeRule.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ImpalaLoptOptimizeJoinRule.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/RelUtil.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/RemoveInfrequentCTERule.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/FilterSelectivityEstimator.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdDistinctRowCount.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdRowCount.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdSize.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CTESuggesterFactory.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteSingleNodePlanner.java M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TestLoptJoinRule.java M java/calcite-planner/src/test/java/org/apache/impala/calcite/service/CalciteOptimizerTest.java A testdata/workloads/functional-query/queries/QueryTest/cte.test M tests/query_test/test_calcite_planner.py 36 files changed, 1,244 insertions(+), 64 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/22091 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id0840c0859d2fe25628d799a18d302cee1eb36e8 Gerrit-Change-Number: 22091 Gerrit-PatchSet: 42 Gerrit-Owner: Michael Smith <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Anonymous Coward (816) Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Stamatis Zampetakis <[email protected]> Gerrit-Reviewer: Steve Carlin <[email protected]>
