Hello Aman Sinha, Quanlong Huang, Joe McDonnell, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21109 to look at the new patch set (#25). Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries ...................................................................... IMPALA-12872: Use Calcite for optimization - part 1: simple queries This is the first commit to use the Calcite library to parse, analyze, and optimize queries. The hook for the planner is through an override of the JniFrontend. The CalciteJniFrontend class is the driver that walks through each of the Calcite steps which are as follows: CalciteQueryParser: Takes the string query and outputs an AST in the form of Calcite's SqlNode object. CalciteMetadataHandler: Iterate through the SqlNode from the previous step and make sure all essential table metadata is retrieved from catalogd. CalciteValidator: Validate the SqlNode tree, akin to the Impala Analyzer. CalciteRelNodeConverter: Change the AST into a logical plan. In this first commit, the only logical nodes used are LogicalTableScan and LogicalProject. The LogicalTableScan will serve as the node that reads from an Hdfs Table and the LogicalProject will only project out the used columns in the query. In later versions, the LogicalProject will also handle function changes. CalciteOptimizer: This step is to optimize the query. In this cut, it will be a nop, but in later versions, it will perform logical optimizations via Calcite's rule mechanism. CalcitePhysPlanCreator: Converts the Calcite RelNode logical tree into Impala's PlanNode physical tree ExecRequestCreator: Implement the existing Impala steps that turn a Single Node Plan into a Distributed Plan. It will also create the TExecRequest object needed by the runtime server. Only some very basic queries will work with this commit. These include: select * from tbl <-- only needs the LogicalTableScan select c1 from tbl <-- Also uses the LogicalProject In the CalciteJniFrontend, there is some basic checks to make sure only select statements will get processed. Any non-query statement will revert back to the current Impala planner. In this iteration, any queries besides the minimal ones listed above will result in a caught exception which will then be run through the current Impala planner. The tests that do work can be found in calcite.test and run through the custom cluster test test_experimental_planner.py This iteration should support all types with the exception of complex types. Calcite does not have a STRING type, so the string type is represented as VARCHAR(MAXINT) similar to how Hive represents their STRING type. The ImpalaTypeConverter file is used to convert the Impala Type object to corresponding Calcite objects. Authorization is not yet working with this current commit. A Jira has been filed (IMPALA-13011) to deal with this. Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 --- M bin/set-classpath.sh M bin/start-impala-cluster.py M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A java/calcite-planner/pom.xml A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ConvertToImpalaRelRules.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaProjectRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/NodeWithExprs.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/phys/ImpalaHdfsScanNode.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/util/CreateExprVisitor.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaCalciteCatalogReader.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CompilerStep.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/validate/ImpalaConformance.java M java/pom.xml A testdata/workloads/functional-query/queries/QueryTest/calcite.test A tests/custom_cluster/test_calcite_planner.py 33 files changed, 3,102 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21109/25 -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 25 Gerrit-Owner: Steve Carlin <scar...@cloudera.com> Gerrit-Reviewer: Aman Sinha <amsi...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Steve Carlin <scar...@cloudera.com>