[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101384#comment-17101384
]
Haisheng Yuan commented on CALCITE-3896:
Jira can't link to github PR at the moment. Here is the PR:
[https://github.com/apache/calcite/pull/1953]
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086015#comment-17086015
]
Haisheng Yuan commented on CALCITE-3896:
{quote}
I hope this "won't and shouldn't" can be enforced by the interface, not just
noted in javadoc.
{quote}
That is impossible. Interface is just a contract, can't control what is done in
the implementation. Car makers won't recommend people drunk-drive, sleep and
drive when selling cars, but some still do. The only thing framework can do is
throw exception when new RelSet is generated.
Here is an example:
https://docs.atlassian.com/software/jira/docs/api/8.4.2/com/atlassian/jira/util/cache/WeakInterner.html
{noformat}
Note: This assumes that your interned objects are sane for interning purposes,
meaning that they are immutable objects with stable hash codes that are
consistent with equals and have a reasonably efficient implementation for
equals. Violate these assumptions at your own risk.
{noformat}
In other words, if you want get things done, do as I tell you.
Regarding your example about plan, as Danny said, it must be generated by your
logical or physical rules, before VolcanoPlanner can pass down required traits.
The framework will do it in a top-down way, from root to leaf. But the
interface still applies.
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085662#comment-17085662
]
Jinpeng Wu commented on CALCITE-3896:
-
> Is this the one of the physical plan after applying all the physical rules?
Yes, Danny. But this plan somehow depends on the passThrough framework. A plan
must be fired
as a candidate before it can win the competition with cost model. I was asking
where can the Project5 get fired.
I am not against the idea itself. Without considering my comments, this
proposal is still a very promotion to current calcite framework. I was just
raising some lessons I ever learnt to make it even better.
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085627#comment-17085627
]
Danny Chen commented on CALCITE-3896:
-
{quote}So how to generate such candidate:
Project4(WithCalculation)<-PhysicalConverter<-Project5(ColumnPruningFromProject1)<-Filter2.
This is most possible the best plan. {quote}
Is this the one of the physical plan after applying all the physical rules ? If
i understand it correctly, what Haisheng proposed is just a promotion to let
the planner has a way to pass through the required traits from root node, it
may generate more candidates in the space, but it does not ensure that the
request trait is definitely to be implemented or the best.
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085500#comment-17085500
]
Jinpeng Wu commented on CALCITE-3896:
-
And I hope this "won't and shouldn't" can be enforced by the interface, not
just noted in javadoc.
For example,
interface RelNode \{
Pair passThrough(RelTraitSet required);
}
It only allows implementations to return the RelTraitSet of Project3 and
AbstractConverter2 in your example.
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085494#comment-17085494
]
Jinpeng Wu commented on CALCITE-3896:
-
When passing through parent requests, it won't and shouldn't generate new child
physical operators
-
So how to generate such candidate:
Project4(WithCalculation)<-PhysicalConverter<-Project5(ColumnPruningFromProject1)<-Filter2.
This is most possible the best plan.
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083523#comment-17083523
]
Haisheng Yuan commented on CALCITE-3896:
Your example is not valid. When passing through parent requests, it won't and
shouldn't generate new child physical operators. e.g.:
Suppose Group 2 is Group 1's child group.
||RelSet||RelSubset ANY||RelSubset Hash[a]||
|Group 1|Project1|AbstractConverter1|
|Group 2|Project2, Filter2| N/A|
What passThrough will do is passing the traitset of AbstractConverter1 through
Project1, obviously, only when Project1 doesn't satisfy AbstractConverter. It
Project decides that it can passThrough, it will return new Project (here is
Project3) with new input subset, so AbstractConverter2 is created.
||RelSet||RelSubset ANY||RelSubset Hash[a]||
|Group 1|Project1|AbstractConverter1, Project3|
|Group 2|Project2, Filter2| AbstractConverter2|
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082903#comment-17082903
]
Jinpeng Wu commented on CALCITE-3896:
-
Hi，[~hyuan] 。
# Got it
# For example, some rule may decide that a logical agg will fire the one phase
agg candidate only when input is small enough or by looking in its input, its
input has already been distributed by the group keys. Well, this case is not
very good. I am just thinking if there may be some exceptions
# An actual case that i have come across，the case AC<-Project(With RexCall),
could generate
## candidate 1：NONE. It is better when calls are generating data with smaller
size (like extract a small part of the data from a big json)
## candidate 2: Project(With RexCall)<-AC. Better when AC is perfectly match
children's delivering traits
## candidate 3: Project(With RexCalls)<-AC<-Project(Column Pruning Only),
better column pruning is available
## candidate 4: Project(Other RexCalls)<-AC<-Project(Containing part of the
rexCalls that may shrink data size), we don' t have the exact cost model here.
So this candidate may produce multiple result that could possibly be the best
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080615#comment-17080615
]
Haisheng Yuan commented on CALCITE-3896:
This will not replace abstract converter, but will make it possible to move on
to remove abstract converter.
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080614#comment-17080614
]
Haisheng Yuan commented on CALCITE-3896:
1. As described in the issue, the redundancy *can't* be avoided given the
current design that physical operator must decide its children's required
traits when it is created. Solving it requires overhaul of planner, and the
implementation rules, which is a big task. This JIRA is not intended to solve
this problem.
2. All the logical and physical rules must be finished before enforcing. All
the alternatives generated by implementation rule are valid.
3. Given a specific required traitSet, passThough generates 1 or none. Can you
give an example to generate multiple candidates?
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077812#comment-17077812
]
Jinpeng Wu commented on CALCITE-3896:
-
Hi， [~hyuan] . Looks like some useful change. Are you intending to use this to
replace AC? I think we need to consider several cases before this：
1. How can the planner know that passing request [a] to MergeJoin[a,b] will
generate exactly the same MergeJoin[a] in order to avoid redundancy. Enforcing
needs only generate an output that satisfies [a], not exactly [a]. Or it could
be another MergeJoin[a] with different inputs, thus different cost.
2. What if the implementation rule generating MergeJoin[a] comes after
passThrough [a] to MergeJoin[a,b]
3. The method passThough could generate more than multiple candidates or none
candidates
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077357#comment-17077357
]
Haisheng Yuan commented on CALCITE-3896:
The cost looks tiny, but if you consider more distribution key combinations
when creating physical operators, multiple different required traits from
parents, join reordering, and property derivation from child operators, the
cost is not so neglectable. However, given Calcite's current design, the
problem is inevitable without overhaul. Anyway, the problem is not this ticket
trying to resolve.
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076967#comment-17076967
]
Junzheng Zheng commented on CALCITE-3896:
-
According to the example, I agree that hash[a] pass through mergeJoin[a,b] and
mergeJoin[a,b,c] are redundant. But I think the cost is tiny, only the new
created mergejoin[a,b] and mergeJoin[a,b,c] are redundant, the propery
enforcement of the inputs of the mergeJoin for hash[a] has been applied before.
Maybe it's a tradeoff between the tiny performance enhancement and involving
the interface which I think maybe a little strange. Correct me if I got
something wrong, thanks!
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076786#comment-17076786
]
Haisheng Yuan commented on CALCITE-3896:
VolcanoPlanner will call the method, after all the rule matches are done.
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076706#comment-17076706
]
Stamatis Zampetakis commented on CALCITE-3896:
--
Thanks for moving this forward [~hyuan]. I do understand the problem but not
completely your proposal. Who is calling the {{passThrough}} method, and when?
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
