[jira] [Commented] (CALCITE-4018) Implement trait propagation for EnumerableValues
[ https://issues.apache.org/jira/browse/CALCITE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112804#comment-17112804 ] Haisheng Yuan commented on CALCITE-4018: Not all, but N, produce at most N collations (where N is the number of columns). > Implement trait propagation for EnumerableValues > - > > Key: CALCITE-4018 > URL: https://issues.apache.org/jira/browse/CALCITE-4018 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > Only passThrough is needed. > Currently, when Values is created, it will enumerate all the possible > collations no matter parent operator requires it or not, it will be a > disaster if the Values has thousands of columns, and the parent operator may > be just a hash aggregate or hashjoin, which doesn't care about its collation. > The collation should be created on demand by calling passThrough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4018) Implement trait propagation for EnumerableValues
[ https://issues.apache.org/jira/browse/CALCITE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112803#comment-17112803 ] Haisheng Yuan commented on CALCITE-4018: Yes, here it is: https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableValues.java#L62 The main logic is in RelMdCollation.values(mq, rowType, tuples) > Implement trait propagation for EnumerableValues > - > > Key: CALCITE-4018 > URL: https://issues.apache.org/jira/browse/CALCITE-4018 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > Only passThrough is needed. > Currently, when Values is created, it will enumerate all the possible > collations no matter parent operator requires it or not, it will be a > disaster if the Values has thousands of columns, and the parent operator may > be just a hash aggregate or hashjoin, which doesn't care about its collation. > The collation should be created on demand by calling passThrough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4017) Implement trait propagation for Enumerable Setop
[ https://issues.apache.org/jira/browse/CALCITE-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112802#comment-17112802 ] Haisheng Yuan commented on CALCITE-4017: Yes. SortUnionTransposeRule will be useless after this JIRA is done. Basically, sort operator should be prohibited to participate any rule transformation. > Implement trait propagation for Enumerable Setop > - > > Key: CALCITE-4017 > URL: https://issues.apache.org/jira/browse/CALCITE-4017 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > Mainly for Union operator, not sure about Minus and Intersect. I haven't > check how is Enumerable Minus, Intersect's executor implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4018) Implement trait propagation for EnumerableValues
[ https://issues.apache.org/jira/browse/CALCITE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112794#comment-17112794 ] Rui Wang commented on CALCITE-4018: --- Could you clarify a bit on: which code/implementation/mechanism make Values "it will enumerate all the possible collations"? It sounds like somewhere must have a for loop to go through all columns in Values, have a collation on each column and add more options to search space? > Implement trait propagation for EnumerableValues > - > > Key: CALCITE-4018 > URL: https://issues.apache.org/jira/browse/CALCITE-4018 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > Only passThrough is needed. > Currently, when Values is created, it will enumerate all the possible > collations no matter parent operator requires it or not, it will be a > disaster if the Values has thousands of columns, and the parent operator may > be just a hash aggregate or hashjoin, which doesn't care about its collation. > The collation should be created on demand by calling passThrough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4016) Implement trait propagation for EnumerableCalc
[ https://issues.apache.org/jira/browse/CALCITE-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112786#comment-17112786 ] Rui Wang edited comment on CALCITE-4016 at 5/21/20, 4:20 AM: - Thanks Chunwei for taking this! You might refer to https://github.com/apache/calcite/pull/1985 in which trait propagation is implemented for project and filter. Also #1985 refactors TopDownTest a bit, which might also be useful when you add new tests to TopDownTest. was (Author: amaliujia): Thanks Chunwei for taking this! You might refer to https://github.com/apache/calcite/pull/1985 in which trait propagation is implemented for project and filter. Also #1985 refactors TopDownTest a bit. > Implement trait propagation for EnumerableCalc > -- > > Key: CALCITE-4016 > URL: https://issues.apache.org/jira/browse/CALCITE-4016 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Assignee: Chunwei Lei >Priority: Major > > It should be similar with EnumerableProject. Maybe same logic. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4017) Implement trait propagation for Enumerable Setop
[ https://issues.apache.org/jira/browse/CALCITE-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112788#comment-17112788 ] Rui Wang commented on CALCITE-4017: --- I think the usefulness of this Jira is still come from pushing down Sort, right? > Implement trait propagation for Enumerable Setop > - > > Key: CALCITE-4017 > URL: https://issues.apache.org/jira/browse/CALCITE-4017 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > Mainly for Union operator, not sure about Minus and Intersect. I haven't > check how is Enumerable Minus, Intersect's executor implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4016) Implement trait propagation for EnumerableCalc
[ https://issues.apache.org/jira/browse/CALCITE-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112786#comment-17112786 ] Rui Wang commented on CALCITE-4016: --- Thanks Chunwei for taking this! You might refer to https://github.com/apache/calcite/pull/1985 in which trait propagation is implemented for project and filter. Also #1985 refactors TopDownTest a bit. > Implement trait propagation for EnumerableCalc > -- > > Key: CALCITE-4016 > URL: https://issues.apache.org/jira/browse/CALCITE-4016 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Assignee: Chunwei Lei >Priority: Major > > It should be similar with EnumerableProject. Maybe same logic. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-2836) Remove maven-compiler-plugin from calcite-plus module pom.xml
[
https://issues.apache.org/jira/browse/CALCITE-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112735#comment-17112735
]
Danny Chen commented on CALCITE-2836:
-
We have migrated from maven to gradle now ~
> Remove maven-compiler-plugin from calcite-plus module pom.xml
> -
>
> Key: CALCITE-2836
> URL: https://issues.apache.org/jira/browse/CALCITE-2836
> Project: Calcite
> Issue Type: Bug
> Components: build
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.19.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> It looks like JDK 11.0.2+ has an issue where 1.8 being specified causes
> javadoc to fail. This has failed for the last few builds on Travis with the
> following error message in the calcite-plus module:
> {code:java}
> javadoc: error - The code being documented uses modules but the packages
> defined in https://docs.oracle.com/javase/8/docs/api/ are in the unnamed
> module.
> {code}
> There is a related JDK bugs about this:
> * [https://bugs.openjdk.java.net/browse/JDK-8212233]
> There is a listed work around but I tested and if we cleanup the leftover
> maven-compiler-plugin part from calcite-plus module this issue goes away.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4014) calcite convert(column using utf8mb4) throws exception
[
https://issues.apache.org/jira/browse/CALCITE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112734#comment-17112734
]
Danny Chen commented on CALCITE-4014:
-
Thanks, [~groobyming] for reporting this ~
I saw the syntax we support from the parser:
```sql
{ s = span(); }
e = Expression(ExprContext.ACCEPT_SUB_QUERY) {
args = startList(e);
}
name = SimpleIdentifier() {
args.add(name);
}
{
return SqlStdOperatorTable.CONVERT.createCall(s.end(this), args);
}
```
That means we did support CONVERT( ... USING ...) clause. But you are right, we
do not translate the charSet correctly, there needs more effort in the convert
table [1], we need to translate the charSet as a constant instead of a column
name.
[1]
https://github.com/apache/calcite/blob/05376d6bff11ccef690addb4f9c40ab16dd20bec/core/src/main/java/org/apache/calcite/rex/RexSqlStandardConvertletTable.java#L103
> calcite convert(column using utf8mb4) throws exception
> --
>
> Key: CALCITE-4014
> URL: https://issues.apache.org/jira/browse/CALCITE-4014
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: groobyming
>Priority: Major
> Attachments: image-2020-05-20-17-00-28-611.png
>
>
> Hello everyone, I found that when performing the following sql
> verification, I will be prompted that the UTF8MB4 field cannot be found in
> the table, but after reading the source code calcite is built-in to support
> this convert function, is it not fully implemented yet?
>
> SQL is as follows:
> select convert(name using utf8mb4) from test b where name = '4'
>
> Error Message:
> !image-2020-05-20-17-00-28-611.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-2997) Avoid pushing down join condition in SqlToRelConverter
[
https://issues.apache.org/jira/browse/CALCITE-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112733#comment-17112733
]
Chunwei Lei edited comment on CALCITE-2997 at 5/21/20, 2:29 AM:
Should we also move other optimization mentioned by [~jinxing6...@126.com] to
RelBuilder too?
was (Author: chunwei lei):
I think It may be less overhead if such optimizations are done in
SqltoRelConverter though it is a little weird. But I'm fine with the PR. Should
we also move other optimization mentioned by [~jinxing6...@126.com] to
RelBuilder too?
> Avoid pushing down join condition in SqlToRelConverter
> --
>
> Key: CALCITE-2997
> URL: https://issues.apache.org/jira/browse/CALCITE-2997
> Project: Calcite
> Issue Type: Bug
>Reporter: Jin Xing
>Assignee: Julian Hyde
>Priority: Major
>
> In current code, *SqlToRelConverter:createJoin* is calling
> *RelOptUtil.pushDownJoinConditions* for optimization. And we can find below
> conversion from *SqlNode* to *RelNode*:
> {code:java}
> SqlNode:
> select * from A join B on A.x = B.x * 2
> RelNode (Logical-Plan):
> Join (condition:col0=col1)
> |-Project(x as col0)
> | |-Scan(A)
> |-Project(x * 2 as col1)
> |-Scan(B){code}
> As we can see the logical plan(*RelNode*) posted above is not the pure
> reflection of the original SQL String(*SqlNode*). The optimization is mixed
> into the phase on which AST is converted to Logical-Plan. Actually optimizing
> rule of JoinPushExpressionsRule is doing exactly the same kind of thing.
> Shall we just keep the optimization inside Optimized-Logical-Plan ? I mean
> shall we avoid calling *RelOptUtil.pushDownJoinConditions* in
> *SqlToRelConverter:createJoin*
> I raised this issue because that we are doing something based on the
> Logical-Plan. And it makes us really confused that the Logical-Plan doesn't
> corresponds to SqlNode.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-2997) Avoid pushing down join condition in SqlToRelConverter
[
https://issues.apache.org/jira/browse/CALCITE-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112733#comment-17112733
]
Chunwei Lei commented on CALCITE-2997:
--
I think It may be less overhead if such optimizations are done in
SqltoRelConverter though it is a little weird. But I'm fine with the PR. Should
we also move other optimization mentioned by [~jinxing6...@126.com] to
RelBuilder too?
> Avoid pushing down join condition in SqlToRelConverter
> --
>
> Key: CALCITE-2997
> URL: https://issues.apache.org/jira/browse/CALCITE-2997
> Project: Calcite
> Issue Type: Bug
>Reporter: Jin Xing
>Assignee: Julian Hyde
>Priority: Major
>
> In current code, *SqlToRelConverter:createJoin* is calling
> *RelOptUtil.pushDownJoinConditions* for optimization. And we can find below
> conversion from *SqlNode* to *RelNode*:
> {code:java}
> SqlNode:
> select * from A join B on A.x = B.x * 2
> RelNode (Logical-Plan):
> Join (condition:col0=col1)
> |-Project(x as col0)
> | |-Scan(A)
> |-Project(x * 2 as col1)
> |-Scan(B){code}
> As we can see the logical plan(*RelNode*) posted above is not the pure
> reflection of the original SQL String(*SqlNode*). The optimization is mixed
> into the phase on which AST is converted to Logical-Plan. Actually optimizing
> rule of JoinPushExpressionsRule is doing exactly the same kind of thing.
> Shall we just keep the optimization inside Optimized-Logical-Plan ? I mean
> shall we avoid calling *RelOptUtil.pushDownJoinConditions* in
> *SqlToRelConverter:createJoin*
> I raised this issue because that we are doing something based on the
> Logical-Plan. And it makes us really confused that the Logical-Plan doesn't
> corresponds to SqlNode.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4016) Implement trait propagation for EnumerableCalc
[ https://issues.apache.org/jira/browse/CALCITE-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112729#comment-17112729 ] Chunwei Lei commented on CALCITE-4016: -- [~hyuan] , I would like to take it. > Implement trait propagation for EnumerableCalc > -- > > Key: CALCITE-4016 > URL: https://issues.apache.org/jira/browse/CALCITE-4016 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > It should be similar with EnumerableProject. Maybe same logic. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-4016) Implement trait propagation for EnumerableCalc
[ https://issues.apache.org/jira/browse/CALCITE-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei reassigned CALCITE-4016: Assignee: Chunwei Lei > Implement trait propagation for EnumerableCalc > -- > > Key: CALCITE-4016 > URL: https://issues.apache.org/jira/browse/CALCITE-4016 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Assignee: Chunwei Lei >Priority: Major > > It should be similar with EnumerableProject. Maybe same logic. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4018) Implement trait propagation for EnumerableValues
Haisheng Yuan created CALCITE-4018: -- Summary: Implement trait propagation for EnumerableValues Key: CALCITE-4018 URL: https://issues.apache.org/jira/browse/CALCITE-4018 Project: Calcite Issue Type: Improvement Components: core Reporter: Haisheng Yuan Only passThrough is needed. Currently, when Values is created, it will enumerate all the possible collations no matter parent operator requires it or not, it will be a disaster if the Values has thousands of columns, and the parent operator may be just a hash aggregate or hashjoin, which doesn't care about its collation. The collation should be created on demand by calling passThrough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4017) Implement trait propagation for Enumerable Setop
Haisheng Yuan created CALCITE-4017: -- Summary: Implement trait propagation for Enumerable Setop Key: CALCITE-4017 URL: https://issues.apache.org/jira/browse/CALCITE-4017 Project: Calcite Issue Type: Improvement Components: core Reporter: Haisheng Yuan Mainly for Union operator, not sure about Minus and Intersect. I haven't check how is Enumerable Minus, Intersect's executor implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4016) Implement trait propagation for EnumerableCalc
Haisheng Yuan created CALCITE-4016: -- Summary: Implement trait propagation for EnumerableCalc Key: CALCITE-4016 URL: https://issues.apache.org/jira/browse/CALCITE-4016 Project: Calcite Issue Type: Improvement Components: core Reporter: Haisheng Yuan It should be similar with EnumerableProject. Maybe same logic. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4011) Implement trait propagation for EnumerableProject and EnumerableFilter
[ https://issues.apache.org/jira/browse/CALCITE-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112660#comment-17112660 ] Rui Wang commented on CALCITE-4011: --- Thanks for your patient and detailed explanation. Now I can that it is "EnumerableConvention.enforce" that will create the EnumerableSort (enforce traits, of course). I have updated https://github.com/apache/calcite/pull/1985 to demonstrate traits pass down in EnumerableProject and EnumerableFilter. Please take a look. I didn't implement traits derivation. (cause I didn't find an example from #1953). Please let me know if you think the traits derivation should be include in the same pull request. > Implement trait propagation for EnumerableProject and EnumerableFilter > -- > > Key: CALCITE-4011 > URL: https://issues.apache.org/jira/browse/CALCITE-4011 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Assignee: Rui Wang >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > So that parent trait can be passed down, and it can derive new traitsets from > child. Most importantly, as a demonstration. So that SortProjectTransposeRule > and ProjectSortTransposeRule can be disabled when topDownOpt is enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4012) Implement trait propagation for EnumerableHashJoin and EnumerableNestedLoopJoin
[ https://issues.apache.org/jira/browse/CALCITE-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112650#comment-17112650 ] Haisheng Yuan commented on CALCITE-4012: [~amaliujia] Thanks for taking this. You can refer EnumerableMergeJoin or EnumerableSortedAggregate as example. Once it is done, SortJoinTranspose and SortJoinCopy rule should be prohibited in topdown mode. > Implement trait propagation for EnumerableHashJoin and > EnumerableNestedLoopJoin > --- > > Key: CALCITE-4012 > URL: https://issues.apache.org/jira/browse/CALCITE-4012 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Assignee: Rui Wang >Priority: Major > > The traitset can be derived from the outer relation of > hashjoin/nestedloopjoin. > They can also pass down parent trait request to their outer child. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-4012) Implement trait propagation for EnumerableHashJoin and EnumerableNestedLoopJoin
[ https://issues.apache.org/jira/browse/CALCITE-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned CALCITE-4012: - Assignee: Rui Wang > Implement trait propagation for EnumerableHashJoin and > EnumerableNestedLoopJoin > --- > > Key: CALCITE-4012 > URL: https://issues.apache.org/jira/browse/CALCITE-4012 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Assignee: Rui Wang >Priority: Major > > The traitset can be derived from the outer relation of > hashjoin/nestedloopjoin. > They can also pass down parent trait request to their outer child. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4011) Implement trait propagation for EnumerableProject and EnumerableFilter
[
https://issues.apache.org/jira/browse/CALCITE-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112445#comment-17112445
]
Haisheng Yuan commented on CALCITE-4011:
It is recommended to override {{passThroughTraits}}, instead of
{{passThrough}}, unless {{passThrough}}'s default implementation doesn't
satisfy the user's needs, which is useful when migrating to top-down
optimization if some system already has its own implementation of trait
propagation based on AbstractConverter.
> Implement trait propagation for EnumerableProject and EnumerableFilter
> --
>
> Key: CALCITE-4011
> URL: https://issues.apache.org/jira/browse/CALCITE-4011
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> So that parent trait can be passed down, and it can derive new traitsets from
> child. Most importantly, as a demonstration. So that SortProjectTransposeRule
> and ProjectSortTransposeRule can be disabled when topDownOpt is enabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4011) Implement trait propagation for EnumerableProject and EnumerableFilter
[
https://issues.apache.org/jira/browse/CALCITE-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112438#comment-17112438
]
Haisheng Yuan edited comment on CALCITE-4011 at 5/20/20, 4:49 PM:
--
Thanks for taking this, [~amaliujia]!
If top-down optimization is enabled, any enforcer (including sort, shuffle)
should not be created explicitly by any rule or relnode, only
Convention.enforce() has the privilege to create enforcer operator, and it is
called by planner. Sort related rules in calcite will be useless, because they
generate many junk alternatives.
During trait propagation, each physical operator just needs to do:
1. If it is required some traits by parent operator, what trait can it deliver
and require its child operator to deliver.
2. If its child can deliver some trait, what trait can it derive from this
child, and require other children to deliver corresponding traits.
For Project, e.g.
{code:java}
select a, b, cnt+2 as c from
(select a, b, count(*) cnt from foo group by a,b) t
order by b desc, a desc; -- case 1
-- order by b desc; -- case 2
{code}
The initial logical plan before passing to VolcanoPlanner may look like:
{code:java}
LogicalSort
-> LogicalProject
-> LogicalAggregate
-> LogicalTableScan
{code}
After implementation, it may look like:
{code:java}
EnumerableProject
-> EnumerableSortedAggregate
-> EnumerableTableScan
{code}
The root relset has a desired traitset with collation on b desc, a desc.
The project is required by parent (here is the root) with collation [b desc, a
desc].
It can pass the required traits down to child operator, it will return what
traits if will require its child to satisfy, and at the same itself's new
traitset after pass required traitset.
If this process is done, with SortProjectTransposeRule disabled, we should
still be able to see the following plan (for both queries):
{code:java}
EnumerableProject
-> EnumerableSortedAggregate
-> EnumerableSort
-> EnumerableTableScan
{code}
was (Author: hyuan):
Thanks for taking this, [~amaliujia]!
If top-down optimization is enabled, any enforcer (including sort, shuffle)
should not be created explicitly by any rule or relnode, only
Convention.enforce() has the privilege to create enforcer operator, and it is
called by planner. Sort related rules in calcite will be useless, because they
generate many junk alternatives.
During trait propagation, each physical operator just needs to do:
1. If it is required some traits by parent operator, what trait can it deliver
and require its child operator to deliver.
2. If its child can deliver some trait, what trait can it derive from this
child, and require other children to deliver corresponding traits.
For Project, e.g.
{code:java}
select a, b, cnt+2 as c from
(select a, b, count(*) cnt from foo group by a,b) t
order by b desc, a desc; -- case 1
-- order by b desc; -- case 2
{code}
The initial logical plan before passing to VolcanoPlanner may look like:
{code:java}
LogicalSort
-> LogicalProject
-> LogicalAggregate
-> LogicalTableScan
{code}
After implementation, it may look like:
{code:java}
EnumerableProject
-> EnumerableSortedAggregate
-> EnumerableTableScan
{code}
The root relset has a desired traitset with collation on b desc, a desc.
The project is required by parent (here is the root) with collation [b desc, a
desc].
It can pass the required traits down to child operator, it will return what
traits if will require its child to satisfy, and at the same itself's new
traitset after pass required traitset.
If this process is done, with SortProjectTransposeRule disabled, we should
still be able to see the following plan (for both queries):
{code:java}
EnumerableProject
-> EnumerableSortedAggregate
-> EnumerableSort
-> EnumerableTableScan
{code:java}
> Implement trait propagation for EnumerableProject and EnumerableFilter
> --
>
> Key: CALCITE-4011
> URL: https://issues.apache.org/jira/browse/CALCITE-4011
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> So that parent trait can be passed down, and it can derive new traitsets from
> child. Most importantly, as a demonstration. So that So
[jira] [Commented] (CALCITE-4011) Implement trait propagation for EnumerableProject and EnumerableFilter
[
https://issues.apache.org/jira/browse/CALCITE-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112438#comment-17112438
]
Haisheng Yuan commented on CALCITE-4011:
Thanks for taking this, [~amaliujia]!
If top-down optimization is enabled, any enforcer (including sort, shuffle)
should not be created explicitly by any rule or relnode, only
Convention.enforce() has the privilege to create enforcer operator, and it is
called by planner. Sort related rules in calcite will be useless, because they
generate many junk alternatives.
During trait propagation, each physical operator just needs to do:
1. If it is required some traits by parent operator, what trait can it deliver
and require its child operator to deliver.
2. If its child can deliver some trait, what trait can it derive from this
child, and require other children to deliver corresponding traits.
For Project, e.g.
{code:java}
select a, b, cnt+2 as c from
(select a, b, count(*) cnt from foo group by a,b) t
order by b desc, a desc; -- case 1
-- order by b desc; -- case 2
{code}
The initial logical plan before passing to VolcanoPlanner may look like:
{code:java}
LogicalSort
-> LogicalProject
-> LogicalAggregate
-> LogicalTableScan
{code}
After implementation, it may look like:
{code:java}
EnumerableProject
-> EnumerableSortedAggregate
-> EnumerableTableScan
{code}
The root relset has a desired traitset with collation on b desc, a desc.
The project is required by parent (here is the root) with collation [b desc, a
desc].
It can pass the required traits down to child operator, it will return what
traits if will require its child to satisfy, and at the same itself's new
traitset after pass required traitset.
If this process is done, with SortProjectTransposeRule disabled, we should
still be able to see the following plan (for both queries):
{code:java}
EnumerableProject
-> EnumerableSortedAggregate
-> EnumerableSort
-> EnumerableTableScan
{code:java}
> Implement trait propagation for EnumerableProject and EnumerableFilter
> --
>
> Key: CALCITE-4011
> URL: https://issues.apache.org/jira/browse/CALCITE-4011
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> So that parent trait can be passed down, and it can derive new traitsets from
> child. Most importantly, as a demonstration. So that SortProjectTransposeRule
> and ProjectSortTransposeRule can be disabled when topDownOpt is enabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4012) Implement trait propagation for EnumerableHashJoin and EnumerableNestedLoopJoin
[
https://issues.apache.org/jira/browse/CALCITE-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112159#comment-17112159
]
Haisheng Yuan commented on CALCITE-4012:
Yes, it is {{RelDistributionTraitDef}}. In Calcite, it is represented by
RelDistribution with type "HASH". In this JIRA, it has nothing to do with
distribution, because EnumerableConvention doesn't support distributed
computing. So we only need consider collation for the moment.
> Implement trait propagation for EnumerableHashJoin and
> EnumerableNestedLoopJoin
> ---
>
> Key: CALCITE-4012
> URL: https://issues.apache.org/jira/browse/CALCITE-4012
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> The traitset can be derived from the outer relation of
> hashjoin/nestedloopjoin.
> They can also pass down parent trait request to their outer child.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4014) calcite convert(column using utf8mb4) throws exception
[ https://issues.apache.org/jira/browse/CALCITE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] groobyming updated CALCITE-4014: Description: Hello everyone, I found that when performing the following sql verification, I will be prompted that the UTF8MB4 field cannot be found in the table, but after reading the source code calcite is built-in to support this convert function, is it not fully implemented yet? SQL is as follows: select convert(name using utf8mb4) from test b where name = '4' Error Message: !image-2020-05-20-17-00-28-611.png! was: Hello everyone, I found that when performing the following sql verification, I will be prompted that the UTF8MB4 field cannot be found in the table, but after reading the source code calcite is built-in to support this convert function, is it not fully implemented yet? SQL is as follows: select convert(name using utf8mb4) from test b where name = '4' Error Message: !image-2020-05-20-17-00-28-611.png! > calcite convert(column using utf8mb4) throws exception > -- > > Key: CALCITE-4014 > URL: https://issues.apache.org/jira/browse/CALCITE-4014 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.21.0 >Reporter: groobyming >Priority: Major > Attachments: image-2020-05-20-17-00-28-611.png > > > Hello everyone, I found that when performing the following sql > verification, I will be prompted that the UTF8MB4 field cannot be found in > the table, but after reading the source code calcite is built-in to support > this convert function, is it not fully implemented yet? > > SQL is as follows: > select convert(name using utf8mb4) from test b where name = '4' > > Error Message: > !image-2020-05-20-17-00-28-611.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4014) calcite convert(column using utf8mb4) throws exception
[ https://issues.apache.org/jira/browse/CALCITE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] groobyming updated CALCITE-4014: Attachment: image-2020-05-20-17-00-28-611.png > calcite convert(column using utf8mb4) throws exception > -- > > Key: CALCITE-4014 > URL: https://issues.apache.org/jira/browse/CALCITE-4014 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.21.0 >Reporter: groobyming >Priority: Major > Attachments: image-2020-05-20-17-00-28-611.png > > > Sql: > select convert(name using utf8mb4) from test b where name = '4' > > errors: > org.apache.calcite.runtime.CalciteContextException: From line 1, column 27 to > line 1, column 33: Column 'UTF8MB4' not found in any > tableorg.apache.calcite.runtime.CalciteContextException: From line 1, column > 27 to line 1, column 33: Column 'UTF8MB4' not found in any table at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at > org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) > at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:834) at > org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:819) at > org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4870) > at > org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:259) > at > org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5762) > at > org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5747) > at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317) at > org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.visitChild(SqlShuttle.java:134) > at > org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.visitChild(SqlShuttle.java:101) > at org.apache.calcite.sql.SqlOperator.acceptCall(SqlOperator.java:869) at > org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visitScoped(SqlValidatorImpl.java:5780) > at > org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:50) > at > org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:33) > at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139) at > org.apache.calcite.sql.validate.SqlValidatorImpl.expand(SqlValidatorImpl.java:5354) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:450) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4107) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3395) > at > org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) > at > org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1008) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:968) > at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:216) at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:943) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:650) > at > com.tencent.blueking.calcite.udf.FunctionTestTwo.sqlToRelNode(FunctionTestTwo.java:144) > at > com.tencent.blueking.calcite.udf.FunctionTestTwo.main(FunctionTestTwo.java:87)Caused > by: org.apache.calcite.sql.validate.SqlValidatorException: Column 'UTF8MB4' > not found in any table at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at > org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) > at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572) ... 27 > moreException in thread "main" java.lang.RuntimeException: > org.apache.calcite.runtime.CalciteContextException: From line 1, column 27 to > line 1, column 33: Column 'UTF8MB4' not found in any table at > com.tencent.blueking.calcite.udf.FunctionTestTwo.sqlToRelNode(FunctionTestTwo.java:173) > at > com.tencent.blueking.calcite.udf.FunctionTestTwo.ma
[jira] [Updated] (CALCITE-4014) calcite convert(column using utf8mb4) throws exception
[ https://issues.apache.org/jira/browse/CALCITE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] groobyming updated CALCITE-4014: Description: Hello everyone, I found that when performing the following sql verification, I will be prompted that the UTF8MB4 field cannot be found in the table, but after reading the source code calcite is built-in to support this convert function, is it not fully implemented yet? SQL is as follows: select convert(name using utf8mb4) from test b where name = '4' Error Message: !image-2020-05-20-17-00-28-611.png! was: Sql: select convert(name using utf8mb4) from test b where name = '4' errors: org.apache.calcite.runtime.CalciteContextException: From line 1, column 27 to line 1, column 33: Column 'UTF8MB4' not found in any tableorg.apache.calcite.runtime.CalciteContextException: From line 1, column 27 to line 1, column 33: Column 'UTF8MB4' not found in any table at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:834) at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:819) at org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4870) at org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:259) at org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5762) at org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:5747) at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317) at org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.visitChild(SqlShuttle.java:134) at org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.visitChild(SqlShuttle.java:101) at org.apache.calcite.sql.SqlOperator.acceptCall(SqlOperator.java:869) at org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visitScoped(SqlValidatorImpl.java:5780) at org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:50) at org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:33) at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139) at org.apache.calcite.sql.validate.SqlValidatorImpl.expand(SqlValidatorImpl.java:5354) at org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:450) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4107) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3395) at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1008) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:968) at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:216) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:943) at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:650) at com.tencent.blueking.calcite.udf.FunctionTestTwo.sqlToRelNode(FunctionTestTwo.java:144) at com.tencent.blueking.calcite.udf.FunctionTestTwo.main(FunctionTestTwo.java:87)Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Column 'UTF8MB4' not found in any table at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572) ... 27 moreException in thread "main" java.lang.RuntimeException: org.apache.calcite.runtime.CalciteContextException: From line 1, column 27 to line 1, column 33: Column 'UTF8MB4' not found in any table at com.tencent.blueking.calcite.udf.FunctionTestTwo.sqlToRelNode(FunctionTestTwo.java:173) at com.tencent.blueking.calcite.udf.FunctionTestTwo.main(FunctionTestTwo.java:87)Caused by: org.apache.calcite.runtime.CalciteContextException: From line 1, column 27 to line 1, column 33: Column 'UTF8MB4' not found in any table at sun.reflect.NativeConstruct
[jira] [Commented] (CALCITE-4012) Implement trait propagation for EnumerableHashJoin and EnumerableNestedLoopJoin
[ https://issues.apache.org/jira/browse/CALCITE-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111839#comment-17111839 ] Rui Wang commented on CALCITE-4012: --- Question: what is a RelTraitDef for "HASH"? Is it RelDistributionTraitDef (I found a hard time to connect this with a hashing property on a column)? > Implement trait propagation for EnumerableHashJoin and > EnumerableNestedLoopJoin > --- > > Key: CALCITE-4012 > URL: https://issues.apache.org/jira/browse/CALCITE-4012 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > The traitset can be derived from the outer relation of > hashjoin/nestedloopjoin. > They can also pass down parent trait request to their outer child. -- This message was sent by Atlassian Jira (v8.3.4#803005)
