Hi Jin, your contribution is awesome! thanks.
I tested it and works like a charm in most cases. Definitely clearer than the
approach that I was trying. However, on the application I'm working on, I found
a weird issue that I wasn't able to reproduce as a unit test on the
calcite-core project. Btw, I'm using your changes after Haisheng's revision.
So, it is as follows, I have this simple query: select l_extendedprice-1 from
lineitem
where the type of l_extendedprice is DOUBLE. Before applying
ProjectTableScanRule the logical plan is:
LogicalProject(EXPR$0=[-($5, 1)])
LogicalTableScan(table=[[main, lineitem]])
So far all is fine, but when the ProjectTableScanRule was applied I got:
java.lang.AssertionError: type mismatch:
ref:
DOUBLE
input:
BIGINT
at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2000)
at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125)
at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57)
at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:140)
at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:57)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:191)
at org.apache.calcite.rel.core.Project.isValid(Project.java:192)
at org.apache.calcite.rel.core.Project.<init>(Project.java:83)
at
org.apache.calcite.rel.logical.LogicalProject.<init>(LogicalProject.java:62)
at
org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:112)
at
org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:100)
at
org.apache.calcite.rel.core.RelFactories$ProjectFactoryImpl.createProject(RelFactories.java:158)
at org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1414)
at org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1252)
at org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1241)
at
com.blazingdb.calcite.rules.ProjectTableScanRule.apply(ProjectTableScanRule.java:147)
at
com.blazingdb.calcite.rules.ProjectTableScanRule$1.onMatch(ProjectTableScanRule.java:65)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:319)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:560)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:374)
at
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:436)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:256)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:215)
at
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:202)
at
com.blazingdb.calcite.application.RelationalAlgebraGenerator.getOptimizedRelationalAlgebra(RelationalAlgebraGenerator.java:202)
at
com.blazingdb.calcite.catalog.BlazingRulesTest.generateSQLTest(BlazingRulesTest.java:274)
Any idea why could this be happening?
Thank you in advance.
On 2019/10/12 09:12:20, XING JIN <jinxing.co...@gmail.com> wrote:
> Hi Stamatis,
> In current code, BindableTableScan is only created by rules of
> ProjectTableScanRule and FilterTableScanRule. I think it's better to keep
> as it is?
> I made a PR for CALCITE-3405 -- https://github.com/apache/calcite/pull/1500
>
> The idea of the PR is quite straightforward:
> 1. Analyze the parent Project -- collect all the needed fields;
> 2. Column pruning by pushing down the needed fields to BindableTableScan;
> 3. Adjust RexInputRefs in parent Project
>
> @Haisheng @Stamatis It would be great if you can give a review when you
> have time ~~~ Thanks a lot !
>
> Best,
> Jin
>
>
> Stamatis Zampetakis <zabe...@gmail.com> 于2019年10月12日周六 下午3:00写道：
>
> > Hi Rommel,
> >
> > I was hoping that this could be done at least by RelFieldTrimmer [1]. Are
> > you using it already?
> >
> > Best,
> > Stamatis
> >
> > [1]
> >
> > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java
> >
> > On Sat, Oct 12, 2019 at 6:06 AM XING JIN <jinxing.co...@gmail.com> wrote:
> >
> > > Filed a JIRA:
> > > https://issues.apache.org/jira/browse/CALCITE-3405
> > >
> > > Haisheng Yuan <h.y...@alibaba-inc.com> 于2019年10月12日周六 上午4:34写道：
> > >
> > > > Yes, definitely.
> > > >
> > > > You can go through the project expression with InputFinder to find all
> > > the
> > > > used columns, create a logical project with those columns, and remap
> > the
> > > > top project with new column indexes.
> > > >
> > > > On the other hand, instead of creating a new intermidiate pogical
> > > project,
> > > > we can also update ProjectTableScanRule to accept LogicalProject that
> > is
> > > > not a simple mapping, and do the same task I mentioned above.
> > > >
> > > > - Haisheng
> > > >
> > > > ------------------------------------------------------------------
> > > > 发件人：Rommel Quintanilla<rom...@blazingdb.com>
> > > > 日 期：2019年10月12日 03:15:31
> > > > 收件人：<dev@calcite.apache.org>
> > > > 主 题：[QUESTION] Pushing up evaluations from LogicalProjects
> > > >
> > > > Hi, maybe you can help me.
> > > > I have this portion from a larger logical plan:
> > > > ..
> > > > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], *=[*($5, -(1,
> > > > $6))])
> > > > LogicalTableScan(table=[[main, lineitem]])
> > > > ..
> > > >
> > > > Because the LogicalProject above contains an evaluation, the
> > > > ProjectTableScanRule can't convert it to a BindableTableScan.
> > > >
> > > > I wonder if somehow the evaluation could be pushed up more or less like
> > > > this:
> > > > ..
> > > > LogicalProject(l_orderkey=[$0], l_suppkey=[$1], *=[*($2, -(1, $3))])
> > > > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], l_extendedprice=[$5],
> > > > l_discount=[$6]])
> > > > LogicalTableScan(table=[[main, lineitem]])
> > > > ..
> > > >
> > > > Regards.
> > > >
> > >
> >
>
