Yi Hu created CALCITE-7101:
------------------------------
Summary: Query parsed to LogicalJoin changed to LogicalFilter
causing ClassCastException then CannotPlanException in newer versions
Key: CALCITE-7101
URL: https://issues.apache.org/jira/browse/CALCITE-7101
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.39.0, 1.34.0
Environment: Java11
Reporter: Yi Hu
We are trying to upgrade Apache Calcite version in our project (Apache Beam
https://github.com/apache/beam/pull/35588). Found a potential breaking change.
Prior to Calcite <=1.33, the following query works for the test
{quote}"select * from CUSTOMER "
+ " where exists ( "
+ " select * from ORDERS "
+ " where o_custkey = c_custkey )";{quote}
However, since Calcite 1.34 (to 1.38), the following error is observed
{quote}java.lang.RuntimeException: Error while applying rule FilterToCalcRule,
args [rel#55:LogicalFilter.NONE(input=RelSubset#54,condition=EXISTS({
LogicalFilter(condition=[=($1, $cor0.c_custkey)])
BeamIOSourceRel(table=[[beam, ORDERS]])
}),variablesSet=[$cor0])]
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:250)
at
org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:59)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:523)
at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:317)
at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:385)
at
org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:210)
Caused by: java.lang.ClassCastException: class
org.apache.calcite.rex.RexSubQuery cannot be cast to class
org.apache.calcite.rex.RexLocalRef (org.apache.calcite.rex.RexSubQuery and
org.apache.calcite.rex.RexLocalRef are in unnamed module of loader 'app')
at
org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuilder.java:304)
{quote}
It is appeared due to CALCITE-6874, which is fixed in Calcite 1.39. However the
fix does not fix our use case In 1.39/1.40, a different error is seen:
{quote}org.apache.beam.sdk.extensions.sql.impl.SqlConversionException: Unable
to convert query select * from CUSTOMER where exists ( select * from ORDERS
where o_custkey = c_custkey )
at
app//org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:214)
at
app//org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:116)
Caused by: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are
not enough rules to produce a node with desired properties:
convention=BEAM_LOGICAL.
Missing conversion is LogicalFilter[convention: NONE -> BEAM_LOGICAL]
at
app//org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit(RelSubset.java:718)
at
app//org.apache.calcite.plan.volcano.RelSubset.buildCheapestPlan(RelSubset.java:391)
at
app//org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:535)
at app//org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:353)
at app//org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:385)
at
app//org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:210)
... 48 more
{quote}
Taking a closer look, In Calcite<=1.33 the query is parsed (planner.parse)
as
{quote}LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2])
LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2],
o_custkey=[CAST($3):INTEGER], $f1=[CAST($4):BOOLEAN])
LogicalJoin(condition=[=($0, $3)], joinType=[inner])
BeamIOSourceRel(table=[[beam, CUSTOMER]])
LogicalAggregate(group=[\{0}], agg#0=[MIN($1)])
LogicalProject(o_custkey=[$1], $f0=[true])
BeamIOSourceRel(table=[[beam, ORDERS]])
{quote}
Then send to planner.rel(). After Calcite 1.34, the query parsed differently
{quote}LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2])
LogicalFilter(condition=[EXISTS(\{ LogicalFilter(condition=[=($1,
$cor0.c_custkey)]) BeamIOSourceRel(table=[[beam, ORDERS]]) })],
variablesSet=[[$cor0]]) BeamIOSourceRel(table=[[beam, CUSTOMER]]){quote}
such that a LogicalFilter no longer converted to LogicalJoin. Then later on
planner.rel() fails.
What might have caused this change (between Calcite 1.33 -> 1.34). This is
currently blocking upgrade. Is it possible to mitigate this without
implementing a custom "LogicalFilter[convention: NONE -> BEAM_LOGICAL]" rule
such that Calcite still knows to apply FilterToJoinRule?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
