Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
I mean, testPushNonSimpleMappingProject [1] runs with RelFieldTrimmer enabled, which is done by [2] [1] https://github.com/apache/calcite/pull/1500/files#diff-3c834a32d46b821b5241e132f2ae6bfaR324 [2] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L249 XING JIN 于2019年10月14日周一 下午4:03写道： > Yes, that's how testPushNonSimpleMappingProject [1] works > > [1] > https://github.com/apache/calcite/pull/1500/files#diff-3c834a32d46b821b5241e132f2ae6bfaR324 > > Danny Chan 于2019年10月14日周一 下午3:36写道： > >> > But why the final BindableTableScan is not pruned ? >> >> The RelFieldTrimmer default is turned off, you should open it explicitly. >> >> Best, >> Danny Chan >> 在 2019年10月14日 +0800 AM11:51，dev@calcite.apache.org，写道： >> > >> > BINDABLE_TABLE_SCAN_RULE >> >
Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
Yes, that's how testPushNonSimpleMappingProject [1] works [1] https://github.com/apache/calcite/pull/1500/files#diff-3c834a32d46b821b5241e132f2ae6bfaR324 Danny Chan 于2019年10月14日周一 下午3:36写道： > > But why the final BindableTableScan is not pruned ? > > The RelFieldTrimmer default is turned off, you should open it explicitly. > > Best, > Danny Chan > 在 2019年10月14日 +0800 AM11:51，dev@calcite.apache.org，写道： > > > > BINDABLE_TABLE_SCAN_RULE >
Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
> But why the final BindableTableScan is not pruned ? The RelFieldTrimmer default is turned off, you should open it explicitly. Best, Danny Chan 在 2019年10月14日 +0800 AM11:51，dev@calcite.apache.org，写道： > > BINDABLE_TABLE_SCAN_RULE
Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
Also regarding that Projects merging is common in Calcite optimization rules, we should always remember to avoid merging for cases like the one given by Rommel. I think that would be hard. Best, Jin XING JIN 于2019年10月14日周一 上午11:51写道： > Hi, Stamatis, Danny~ > > Thanks for explain ~ > > > "The consumer in the case of P1 is the project which only needs $0, $2, > $5, > $6 so the trimmer could slim down the scan by projecting only these > fields." > > I think RelFieldTrimmer is already doing this by [1]. > But why the final BindableTableScan is not pruned ? I think the answer is > that RelFieldTrimmer merges projects when trimming fields for Project: > [2][3] > > > "RelFieldTrimmer can be used to transform the plan P1 to > plan P2 and then ProjectTableScanRule can be used to introduce the > BindableTableScan." > > If we wanna go by this approach, shall we avoid the project merging as > mentioned in RelFieldTrimmer [2][3] ? > > Well, as an independent functionality for column pruning, I still suggest > ProjectTableScanRule have the the ability to handle such complex scenarios. > > How do you think? > > Best, > Jin > > > > [1] > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L1054 > [2] > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L416 > > [3] > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1327 > > Danny Chan 于2019年10月14日周一 上午9:42写道： > >> +1, RelFieldTrimmer is the role to trim the unused fields. >> >> Best, >> Danny Chan >> 在 2019年10月13日 +0800 AM6:25，dev@calcite.apache.org，写道： >> > >> > RelFieldTrimmer >> >
Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
Hi, Stamatis, Danny~ Thanks for explain ~ > "The consumer in the case of P1 is the project which only needs $0, $2, $5, $6 so the trimmer could slim down the scan by projecting only these fields." I think RelFieldTrimmer is already doing this by [1]. But why the final BindableTableScan is not pruned ? I think the answer is that RelFieldTrimmer merges projects when trimming fields for Project: [2][3] > "RelFieldTrimmer can be used to transform the plan P1 to plan P2 and then ProjectTableScanRule can be used to introduce the BindableTableScan." If we wanna go by this approach, shall we avoid the project merging as mentioned in RelFieldTrimmer [2][3] ? Well, as an independent functionality for column pruning, I still suggest ProjectTableScanRule have the the ability to handle such complex scenarios. How do you think? Best, Jin [1] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L1054 [2] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L416 [3] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1327 Danny Chan 于2019年10月14日周一 上午9:42写道： > +1, RelFieldTrimmer is the role to trim the unused fields. > > Best, > Danny Chan > 在 2019年10月13日 +0800 AM6:25，dev@calcite.apache.org，写道： > > > > RelFieldTrimmer >
Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
+1, RelFieldTrimmer is the role to trim the unused fields. Best, Danny Chan 在 2019年10月13日 +0800 AM6:25，dev@calcite.apache.org，写道： > > RelFieldTrimmer
Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
I was thinking that RelFieldTrimmer can be used to transform the plan P1 to plan P2 and then ProjectTableScanRule can be used to introduce the BindableTableScan. The consumer in the case of P1 is the project which only needs $0, $2, $5, $6 so the trimmer could slim down the scan by projecting only these fields. I didn't test if it really does this in this case because in some sense the trimming is a bit redundant given that the project just on top has just a subset of the columns of the table. Note that RelFieldTrimmer could be used also during optimization (not only in sql to rel phase) as a separate planning phase. *P1* LogicalProject(l_orderkey=[$0], l_suppkey=[$2], *=[*($5, -(1, $6))]) LogicalTableScan(table=[[main, lineitem]]) *P2* LogicalProject(l_orderkey=[$0], l_suppkey=[$1], *=[*($2, -(1, $3))]) LogicalProject(l_orderkey=[$0], l_suppkey=[$2], l_extendedprice=[$5], l_discount=[$6]]) LogicalTableScan(table=[[main, lineitem]]) On Sat, Oct 12, 2019 at 1:11 PM Wang Yanlin <1989yanlinw...@163.com> wrote: > Just a little curious, > RelFieldTrimmer is used to 'slimmed down' relational expression that > projects only the columns required by its consumer. > How can it used to do this thing? > > > > > -- > > Best, > Wang Yanlin > > > > 在 2019-10-12 18:03:33，"XING JIN" 写道： > >Sure we can ~ > >If we use BindableTableScanRule to derive a BindableTableScan from > >ProjectableFilterableTable, that would happen during a stage of > >optimization run by RelOptPlanner. But RelFieldTrimmer works right during > >conversion of Sql to Rel. > > > >Wang Yanlin <1989yanlinw...@163.com> 于2019年10月12日周六 下午5:56写道： > > > >> Can we just use Bindables.BINDABLE_TABLE_SCAN_RULE to translate the > >> table scan to BindableTableScan ? > >> > >> > >> > >> -- > >> > >> Best, > >> Wang Yanlin > >> > >> > >> > >> At 2019-10-12 17:12:20, "XING JIN" wrote: > >> >Hi Stamatis, > >> >In current code, BindableTableScan is only created by rules of > >> >ProjectTableScanRule and FilterTableScanRule. I think it's better to > keep > >> >as it is? > >> >I made a PR for CALCITE-3405 -- > >> https://github.com/apache/calcite/pull/1500 > >> > > >> >The idea of the PR is quite straightforward: > >> >1. Analyze the parent Project -- collect all the needed fields; > >> >2. Column pruning by pushing down the needed fields to > BindableTableScan; > >> >3. Adjust RexInputRefs in parent Project > >> > > >> >@Haisheng @Stamatis It would be great if you can give a review when you > >> >have time ~~~ Thanks a lot ! > >> > > >> >Best, > >> >Jin > >> > > >> > > >> >Stamatis Zampetakis 于2019年10月12日周六 下午3:00写道： > >> > > >> >> Hi Rommel, > >> >> > >> >> I was hoping that this could be done at least by RelFieldTrimmer [1]. > >> Are > >> >> you using it already? > >> >> > >> >> Best, > >> >> Stamatis > >> >> > >> >> [1] > >> >> > >> >> > >> > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java > >> >> > >> >> On Sat, Oct 12, 2019 at 6:06 AM XING JIN > >> wrote: > >> >> > >> >> > Filed a JIRA: > >> >> > https://issues.apache.org/jira/browse/CALCITE-3405 > >> >> > > >> >> > Haisheng Yuan 于2019年10月12日周六 上午4:34写道： > >> >> > > >> >> > > Yes, definitely. > >> >> > > > >> >> > > You can go through the project expression with InputFinder to > find > >> all > >> >> > the > >> >> > > used columns, create a logical project with those columns, and > >> remap > >> >> the > >> >> > > top project with new column indexes. > >> >> > > > >> >> > > On the other hand, instead of creating a new intermidiate pogical > >> >> > project, > >> >> > > we can also update ProjectTableScanRule to accept LogicalProject > >> that > >> >> is > >> >> > > not a simple mapping, and do the same task I mentioned above. > >> >> > > > >> >> > > - Haisheng > >> >> > > > >> >> > > > -- > >> >> > > 发件人：Rommel Quintanilla > >> >> > > 日 期：2019年10月12日 03:15:31 > >> >> > > 收件人： > >> >> > > 主 题：[QUESTION] Pushing up evaluations from LogicalProjects > >> >> > > > >> >> > > Hi, maybe you can help me. > >> >> > > I have this portion from a larger logical plan: > >> >> > > .. > >> >> > > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], *=[*($5, > >> -(1, > >> >> > > $6))]) > >> >> > > LogicalTableScan(table=[[main, lineitem]]) > >> >> > > .. > >> >> > > > >> >> > > Because the LogicalProject above contains an evaluation, the > >> >> > > ProjectTableScanRule can't convert it to a BindableTableScan. > >> >> > > > >> >> > > I wonder if somehow the evaluation could be pushed up more or > less > >> like > >> >> > > this: > >> >> > > .. > >> >> > > LogicalProject(l_orderkey=[$0], l_suppkey=[$1], *=[*($2, -(1, > >> $3))]) > >> >> > > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], > >> l_extendedprice=[$5], > >> >> > > l_discount=[$6]]) > >> >> > > LogicalTableScan(table=[[main, > lineitem]]) > >> >> > > .. > >> >> > > > >> >> > >
Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
Sure we can ~ If we use BindableTableScanRule to derive a BindableTableScan from ProjectableFilterableTable, that would happen during a stage of optimization run by RelOptPlanner. But RelFieldTrimmer works right during conversion of Sql to Rel. Wang Yanlin <1989yanlinw...@163.com> 于2019年10月12日周六 下午5:56写道： > Can we just use Bindables.BINDABLE_TABLE_SCAN_RULE to translate the > table scan to BindableTableScan ? > > > > -- > > Best, > Wang Yanlin > > > > At 2019-10-12 17:12:20, "XING JIN" wrote: > >Hi Stamatis, > >In current code, BindableTableScan is only created by rules of > >ProjectTableScanRule and FilterTableScanRule. I think it's better to keep > >as it is? > >I made a PR for CALCITE-3405 -- > https://github.com/apache/calcite/pull/1500 > > > >The idea of the PR is quite straightforward: > >1. Analyze the parent Project -- collect all the needed fields; > >2. Column pruning by pushing down the needed fields to BindableTableScan; > >3. Adjust RexInputRefs in parent Project > > > >@Haisheng @Stamatis It would be great if you can give a review when you > >have time ~~~ Thanks a lot ! > > > >Best, > >Jin > > > > > >Stamatis Zampetakis 于2019年10月12日周六 下午3:00写道： > > > >> Hi Rommel, > >> > >> I was hoping that this could be done at least by RelFieldTrimmer [1]. > Are > >> you using it already? > >> > >> Best, > >> Stamatis > >> > >> [1] > >> > >> > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java > >> > >> On Sat, Oct 12, 2019 at 6:06 AM XING JIN > wrote: > >> > >> > Filed a JIRA: > >> > https://issues.apache.org/jira/browse/CALCITE-3405 > >> > > >> > Haisheng Yuan 于2019年10月12日周六 上午4:34写道： > >> > > >> > > Yes, definitely. > >> > > > >> > > You can go through the project expression with InputFinder to find > all > >> > the > >> > > used columns, create a logical project with those columns, and > remap > >> the > >> > > top project with new column indexes. > >> > > > >> > > On the other hand, instead of creating a new intermidiate pogical > >> > project, > >> > > we can also update ProjectTableScanRule to accept LogicalProject > that > >> is > >> > > not a simple mapping, and do the same task I mentioned above. > >> > > > >> > > - Haisheng > >> > > > >> > > -- > >> > > 发件人：Rommel Quintanilla > >> > > 日 期：2019年10月12日 03:15:31 > >> > > 收件人： > >> > > 主 题：[QUESTION] Pushing up evaluations from LogicalProjects > >> > > > >> > > Hi, maybe you can help me. > >> > > I have this portion from a larger logical plan: > >> > > .. > >> > > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], *=[*($5, > -(1, > >> > > $6))]) > >> > > LogicalTableScan(table=[[main, lineitem]]) > >> > > .. > >> > > > >> > > Because the LogicalProject above contains an evaluation, the > >> > > ProjectTableScanRule can't convert it to a BindableTableScan. > >> > > > >> > > I wonder if somehow the evaluation could be pushed up more or less > like > >> > > this: > >> > > .. > >> > > LogicalProject(l_orderkey=[$0], l_suppkey=[$1], *=[*($2, -(1, > $3))]) > >> > > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], > l_extendedprice=[$5], > >> > > l_discount=[$6]]) > >> > > LogicalTableScan(table=[[main, lineitem]]) > >> > > .. > >> > > > >> > > Regards. > >> > > > >> > > >> >
