[jira] [Commented] (CALCITE-3926) CannotPlanException when an empty LogicalValues requires a certain collation
[
https://issues.apache.org/jira/browse/CALCITE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084110#comment-17084110
]
Haisheng Yuan commented on CALCITE-3926:
Thanks for reporting, I will take a look.
> CannotPlanException when an empty LogicalValues requires a certain collation
>
>
> Key: CALCITE-3926
> URL: https://issues.apache.org/jira/browse/CALCITE-3926
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Ruben Q L
>Priority: Blocker
>
> The issue can be reproduced with the following test (in RelBuilderTest.java):
> {code:java}
> @Test void testEmptyValues() throws Exception {
> final RelBuilder builder = RelBuilder.create(config().build());
> final RelNode root =
> builder
> .scan("DEPT")
> .filter(builder.literal(false)) // generates an empty
> LogicalValues
> .sort(
> builder.field("DNAME"),
> builder.field("DEPTNO"))
> .build();
> try (PreparedStatement preparedStatement = RelRunners.run(root)) {
> final String s =
> CalciteAssert.toString(preparedStatement.executeQuery());
> final String result = "";
> assertThat(s, is(result));
> }
> }
> {code}
> The test fails with the following message:
> {code:java}
> Caused by: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There
> are not enough rules to produce a node with desired properties:
> convention=ENUMERABLE, sort=[1, 0].
> Missing conversion is LogicalValues[convention: NONE -> ENUMERABLE, sort:
> [[0, 1, 2], [1, 2], [2]] -> [1, 0]]
> There is 1 empty subset: rel#14:Subset#0.ENUMERABLE.[1, 0], the relevant part
> of the original plan is as follows
> 1:LogicalValues(tuples=[[]])
> {code}
> The issue seems to be caused by the empty values: Scan + Filter(false) =>
> LogicalValues(tuples=[[]]).
> This empty LogicalValues has as collation (in case of 3 fields): [[0, 1, 2],
> [1, 2], [2]]; and in the plan we are requesting as sort collation [1, 0].
> This could be achieved by: EnumerableSort(sort=[1,0]) +
> EnumerableValues(tuples=[[]]) ; but for some reason this plan is not
> generated.
> It seems the issue was somehow introduced by CALCITE-3886
>
> [https://github.com/apache/calcite/commit/08f4a9889728f94903ff424ed9c20d940b821af5]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3926) CannotPlanException when an empty LogicalValues requires a certain collation
[
https://issues.apache.org/jira/browse/CALCITE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-3926:
---
Attachment: image-2020-04-15-11-14-41-581.png
> CannotPlanException when an empty LogicalValues requires a certain collation
>
>
> Key: CALCITE-3926
> URL: https://issues.apache.org/jira/browse/CALCITE-3926
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Ruben Q L
>Priority: Blocker
> Attachments: image-2020-04-15-11-14-41-581.png
>
>
> The issue can be reproduced with the following test (in RelBuilderTest.java):
> {code:java}
> @Test void testEmptyValues() throws Exception {
> final RelBuilder builder = RelBuilder.create(config().build());
> final RelNode root =
> builder
> .scan("DEPT")
> .filter(builder.literal(false)) // generates an empty
> LogicalValues
> .sort(
> builder.field("DNAME"),
> builder.field("DEPTNO"))
> .build();
> try (PreparedStatement preparedStatement = RelRunners.run(root)) {
> final String s =
> CalciteAssert.toString(preparedStatement.executeQuery());
> final String result = "";
> assertThat(s, is(result));
> }
> }
> {code}
> The test fails with the following message:
> {code:java}
> Caused by: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There
> are not enough rules to produce a node with desired properties:
> convention=ENUMERABLE, sort=[1, 0].
> Missing conversion is LogicalValues[convention: NONE -> ENUMERABLE, sort:
> [[0, 1, 2], [1, 2], [2]] -> [1, 0]]
> There is 1 empty subset: rel#14:Subset#0.ENUMERABLE.[1, 0], the relevant part
> of the original plan is as follows
> 1:LogicalValues(tuples=[[]])
> {code}
> The issue seems to be caused by the empty values: Scan + Filter(false) =>
> LogicalValues(tuples=[[]]).
> This empty LogicalValues has as collation (in case of 3 fields): [[0, 1, 2],
> [1, 2], [2]]; and in the plan we are requesting as sort collation [1, 0].
> This could be achieved by: EnumerableSort(sort=[1,0]) +
> EnumerableValues(tuples=[[]]) ; but for some reason this plan is not
> generated.
> It seems the issue was somehow introduced by CALCITE-3886
>
> [https://github.com/apache/calcite/commit/08f4a9889728f94903ff424ed9c20d940b821af5]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3926) CannotPlanException when an empty LogicalValues requires a certain collation
[
https://issues.apache.org/jira/browse/CALCITE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-3926:
---
Attachment: (was: image-2020-04-15-11-14-41-581.png)
> CannotPlanException when an empty LogicalValues requires a certain collation
>
>
> Key: CALCITE-3926
> URL: https://issues.apache.org/jira/browse/CALCITE-3926
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Ruben Q L
>Priority: Blocker
>
> The issue can be reproduced with the following test (in RelBuilderTest.java):
> {code:java}
> @Test void testEmptyValues() throws Exception {
> final RelBuilder builder = RelBuilder.create(config().build());
> final RelNode root =
> builder
> .scan("DEPT")
> .filter(builder.literal(false)) // generates an empty
> LogicalValues
> .sort(
> builder.field("DNAME"),
> builder.field("DEPTNO"))
> .build();
> try (PreparedStatement preparedStatement = RelRunners.run(root)) {
> final String s =
> CalciteAssert.toString(preparedStatement.executeQuery());
> final String result = "";
> assertThat(s, is(result));
> }
> }
> {code}
> The test fails with the following message:
> {code:java}
> Caused by: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There
> are not enough rules to produce a node with desired properties:
> convention=ENUMERABLE, sort=[1, 0].
> Missing conversion is LogicalValues[convention: NONE -> ENUMERABLE, sort:
> [[0, 1, 2], [1, 2], [2]] -> [1, 0]]
> There is 1 empty subset: rel#14:Subset#0.ENUMERABLE.[1, 0], the relevant part
> of the original plan is as follows
> 1:LogicalValues(tuples=[[]])
> {code}
> The issue seems to be caused by the empty values: Scan + Filter(false) =>
> LogicalValues(tuples=[[]]).
> This empty LogicalValues has as collation (in case of 3 fields): [[0, 1, 2],
> [1, 2], [2]]; and in the plan we are requesting as sort collation [1, 0].
> This could be achieved by: EnumerableSort(sort=[1,0]) +
> EnumerableValues(tuples=[[]]) ; but for some reason this plan is not
> generated.
> It seems the issue was somehow introduced by CALCITE-3886
>
> [https://github.com/apache/calcite/commit/08f4a9889728f94903ff424ed9c20d940b821af5]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3926) CannotPlanException when an empty LogicalValues requires a certain collation
[
https://issues.apache.org/jira/browse/CALCITE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084254#comment-17084254
]
Haisheng Yuan commented on CALCITE-3926:
Hi [~rubenql], same conclusion here. Please go ahead and open the pull request.
Thanks!
The culprit is that logical operators are allowed to have traits, which is
really a disappointing design decision.
> CannotPlanException when an empty LogicalValues requires a certain collation
>
>
> Key: CALCITE-3926
> URL: https://issues.apache.org/jira/browse/CALCITE-3926
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Ruben Q L
>Priority: Blocker
>
> The issue can be reproduced with the following test (in RelBuilderTest.java):
> {code:java}
> @Test void testEmptyValues() throws Exception {
> final RelBuilder builder = RelBuilder.create(config().build());
> final RelNode root =
> builder
> .scan("DEPT")
> .filter(builder.literal(false)) // generates an empty
> LogicalValues
> .sort(
> builder.field("DNAME"),
> builder.field("DEPTNO"))
> .build();
> try (PreparedStatement preparedStatement = RelRunners.run(root)) {
> final String s =
> CalciteAssert.toString(preparedStatement.executeQuery());
> final String result = "";
> assertThat(s, is(result));
> }
> }
> {code}
> The test fails with the following message:
> {code:java}
> Caused by: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There
> are not enough rules to produce a node with desired properties:
> convention=ENUMERABLE, sort=[1, 0].
> Missing conversion is LogicalValues[convention: NONE -> ENUMERABLE, sort:
> [[0, 1, 2], [1, 2], [2]] -> [1, 0]]
> There is 1 empty subset: rel#14:Subset#0.ENUMERABLE.[1, 0], the relevant part
> of the original plan is as follows
> 1:LogicalValues(tuples=[[]])
> {code}
> The issue seems to be caused by the empty values: Scan + Filter(false) =>
> LogicalValues(tuples=[[]]).
> This empty LogicalValues has as collation (in case of 3 fields): [[0, 1, 2],
> [1, 2], [2]]; and in the plan we are requesting as sort collation [1, 0].
> This could be achieved by: EnumerableSort(sort=[1,0]) +
> EnumerableValues(tuples=[[]]) ; but for some reason this plan is not
> generated.
> It seems the issue was somehow introduced by CALCITE-3886
>
> [https://github.com/apache/calcite/commit/08f4a9889728f94903ff424ed9c20d940b821af5]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3927) RelSubset is not fired for rule when set gets merged
Haisheng Yuan created CALCITE-3927: -- Summary: RelSubset is not fired for rule when set gets merged Key: CALCITE-3927 URL: https://issues.apache.org/jira/browse/CALCITE-3927 Project: Calcite Issue Type: Bug Components: core Reporter: Haisheng Yuan In VolcanoPlanner, when set gets merged, planner fires rules again for RelNodes in both sets, but not for RelSubset. We might miss something because of this. If all the logical transformation rules and physical implementation rules are separated out and physical rules don't do logical work, we might be OK. But the reality is that all the things are mixed together at the moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3927) RelSubset is not fired for rule when set gets merged
[ https://issues.apache.org/jira/browse/CALCITE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3927: --- Description: In VolcanoPlanner, when set gets merged, planner fires rules again for RelNodes in both sets, but not for RelSubset. We might miss something because of this. If all the logical transformation rules and physical implementation rules are separated out in different stage and physical rules don't do logical work, we might be OK. But the reality is that all the things are mixed together at the moment. was: In VolcanoPlanner, when set gets merged, planner fires rules again for RelNodes in both sets, but not for RelSubset. We might miss something because of this. If all the logical transformation rules and physical implementation rules are separated out and physical rules don't do logical work, we might be OK. But the reality is that all the things are mixed together at the moment. > RelSubset is not fired for rule when set gets merged > > > Key: CALCITE-3927 > URL: https://issues.apache.org/jira/browse/CALCITE-3927 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > > In VolcanoPlanner, when set gets merged, planner fires rules again for > RelNodes in both sets, but not for RelSubset. We might miss something because > of this. > If all the logical transformation rules and physical implementation rules are > separated out in different stage and physical rules don't do logical work, we > might be OK. But the reality is that all the things are mixed together at the > moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3927) RelSubset is not fired for rule when set gets merged
[ https://issues.apache.org/jira/browse/CALCITE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084366#comment-17084366 ] Haisheng Yuan commented on CALCITE-3927: That might be challenging. [~botong] Any easy case? > RelSubset is not fired for rule when set gets merged > > > Key: CALCITE-3927 > URL: https://issues.apache.org/jira/browse/CALCITE-3927 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > > In VolcanoPlanner, when set gets merged, planner fires rules again for > RelNodes in both sets, but not for RelSubset. We might miss something because > of this. > If all the logical transformation rules and physical implementation rules are > separated out in different stage and physical rules don't do logical work, we > might be OK. But the reality is that all the things are mixed together at the > moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3149) CACHE in RelDataTypeFactoryImpl can't be garbage collected
[ https://issues.apache.org/jira/browse/CALCITE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084990#comment-17084990 ] Haisheng Yuan commented on CALCITE-3149: You know it is a cache, and cache can be evicted, right? If you think this is a regression, you can log a JIRA to revert the change. Or make the cache thread-local ever live HashMap. > CACHE in RelDataTypeFactoryImpl can't be garbage collected > --- > > Key: CALCITE-3149 > URL: https://issues.apache.org/jira/browse/CALCITE-3149 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Assignee: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Key and Value are pointing to the same object, even with soft value > references, the objects in CACHE will not be garbage collected. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3932) Make data type cache thread local, non-evictable
Haisheng Yuan created CALCITE-3932: -- Summary: Make data type cache thread local, non-evictable Key: CALCITE-3932 URL: https://issues.apache.org/jira/browse/CALCITE-3932 Project: Calcite Issue Type: Improvement Components: core Reporter: Haisheng Yuan Data type can be evicted out of cache, which is global, thread-safe. `=` based type comparison may suffer from cache eviction. It seems not necessary to cache them globally, because most of them are RelRecordType, which is query dependent, not sharable between different queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3149) CACHE in RelDataTypeFactoryImpl can't be garbage collected
[ https://issues.apache.org/jira/browse/CALCITE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085043#comment-17085043 ] Haisheng Yuan commented on CALCITE-3149: Created JIRA: https://issues.apache.org/jira/browse/CALCITE-3932 > CACHE in RelDataTypeFactoryImpl can't be garbage collected > --- > > Key: CALCITE-3149 > URL: https://issues.apache.org/jira/browse/CALCITE-3149 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Assignee: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Key and Value are pointing to the same object, even with soft value > references, the objects in CACHE will not be garbage collected. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3149) CACHE in RelDataTypeFactoryImpl can't be garbage collected
[ https://issues.apache.org/jira/browse/CALCITE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085083#comment-17085083 ] Haisheng Yuan commented on CALCITE-3149: Correct, If the data type is in use, it should not be garbage collected. We haven't seen any issue in production. But one thing is confirmed that the global cache seems not necessary. > CACHE in RelDataTypeFactoryImpl can't be garbage collected > --- > > Key: CALCITE-3149 > URL: https://issues.apache.org/jira/browse/CALCITE-3149 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Assignee: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Key and Value are pointing to the same object, even with soft value > references, the objects in CACHE will not be garbage collected. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3932: --- Description: Data type can be evicted out of cache, which is global, thread-safe. It seems not necessary to cache them globally, because most of them are RelRecordType, which is query dependent, not sharable between different queries. was: Data type can be evicted out of cache, which is global, thread-safe. `=` based type comparison may suffer from cache eviction. It seems not necessary to cache them globally, because most of them are RelRecordType, which is query dependent, not sharable between different queries. > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085290#comment-17085290 ] Haisheng Yuan commented on CALCITE-3932: Most of the data type among different queries are not reusable except basic sql type. > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-2970) Performance issue when enabling abstract converter for EnumerableConvention
[
https://issues.apache.org/jira/browse/CALCITE-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085296#comment-17085296
]
Haisheng Yuan commented on CALCITE-2970:
Getting back to the physical RelBuilder pull request.
- RelBuilderFactory and RelFactories.LOGICAL_BUILDER should remain unchanged,
otherwise it is breaking.
- Convention can have a `getRelBuilder()` interface, in EnumerableConvention,
the change is as simple as:
{code:java}
private static RelBuilderFactory factory = RelBuilder.proto(
(RelFactories.SortFactory) (input, collation, offset, fetch)
-> EnumerableSort.create(input, collation, offset, fetch), ..., ...);
public RelBuilder getRelBuilder(RelOptCluster cluster) {
return factory.create(cluster, null);
}
{code}
- RelTraitDef.convert and RelCollationTraitDef.convert shouldn't require
toConvention, it can get convention from either rel (for non-convention trait)
or toTrait (for convention trait).
> Performance issue when enabling abstract converter for EnumerableConvention
> ---
>
> Key: CALCITE-2970
> URL: https://issues.apache.org/jira/browse/CALCITE-2970
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 13h 10m
> Remaining Estimate: 0h
>
> If we enable the use of abstract converter for {{EnumerableConvention}}, by
> making {{useAbstractConvertersForConversion}} return true,
> {{JDBCTest.testJoinManyWay}} will not complete.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085322#comment-17085322 ] Haisheng Yuan commented on CALCITE-3932: RelDataTypeFactoryImpl KEY2TYPE_CACHE and DATATYPE_CACHE. [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/type/RelDataTypeFactoryImpl.java#L53-L65] > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-2970) Performance issue when enabling abstract converter for EnumerableConvention
[
https://issues.apache.org/jira/browse/CALCITE-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085348#comment-17085348
]
Haisheng Yuan commented on CALCITE-2970:
[~julianhyde] Will {{UnaryOperator getRelBuilderTransform()}} work
with {{RelBuilder}}'s sub-classes? I am afraid people might still want to
sub-class RelBuilder given that the default RelBuilder won't satisfy all the
needs and it can't deal with traits correctly yet, like CALCITE-3865.
> Performance issue when enabling abstract converter for EnumerableConvention
> ---
>
> Key: CALCITE-2970
> URL: https://issues.apache.org/jira/browse/CALCITE-2970
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 13h 10m
> Remaining Estimate: 0h
>
> If we enable the use of abstract converter for {{EnumerableConvention}}, by
> making {{useAbstractConvertersForConversion}} return true,
> {{JDBCTest.testJoinManyWay}} will not complete.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085406#comment-17085406 ] Haisheng Yuan commented on CALCITE-3932: How can it be garbage collected while still being referenced? Even for weak interner? In this issue, I want it non-evictable because guava strong interner is better performanced given that we want the cache short lived (suppose there is no thread-pool). > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086015#comment-17086015
]
Haisheng Yuan commented on CALCITE-3896:
{quote}
I hope this "won't and shouldn't" can be enforced by the interface, not just
noted in javadoc.
{quote}
That is impossible. Interface is just a contract, can't control what is done in
the implementation. Car makers won't recommend people drunk-drive, sleep and
drive when selling cars, but some still do. The only thing framework can do is
throw exception when new RelSet is generated.
Here is an example:
https://docs.atlassian.com/software/jira/docs/api/8.4.2/com/atlassian/jira/util/cache/WeakInterner.html
{noformat}
Note: This assumes that your interned objects are sane for interning purposes,
meaning that they are immutable objects with stable hash codes that are
consistent with equals and have a reasonably efficient implementation for
equals. Violate these assumptions at your own risk.
{noformat}
In other words, if you want get things done, do as I tell you.
Regarding your example about plan, as Danny said, it must be generated by your
logical or physical rules, before VolcanoPlanner can pass down required traits.
The framework will do it in a top-down way, from root to leaf. But the
interface still applies.
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086242#comment-17086242 ] Haisheng Yuan commented on CALCITE-3932: [~julianhyde] What do you think? Are the two caches good candidates to use ThreadLocal instead of global? > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3937) Only fire rule for RelSubset when it is derived
Haisheng Yuan created CALCITE-3937: -- Summary: Only fire rule for RelSubset when it is derived Key: CALCITE-3937 URL: https://issues.apache.org/jira/browse/CALCITE-3937 Project: Calcite Issue Type: Improvement Components: core Reporter: Haisheng Yuan It is meaningless to fire rule for RelSubset when it is generated by parent's trait requirement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3937) Fire rule for RelSubset only when it is derived
[ https://issues.apache.org/jira/browse/CALCITE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3937: --- Summary: Fire rule for RelSubset only when it is derived (was: Only fire rule for RelSubset when it is derived ) > Fire rule for RelSubset only when it is derived > > > Key: CALCITE-3937 > URL: https://issues.apache.org/jira/browse/CALCITE-3937 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > > It is meaningless to fire rule for RelSubset when it is generated by parent's > trait requirement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3918) SubQueryFilterRemoveRule failed to decorrelate subquery for TPCH q17
[
https://issues.apache.org/jira/browse/CALCITE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087315#comment-17087315
]
Haisheng Yuan commented on CALCITE-3918:
Modify the code and return the relnode directly without decorrelating in
RelDecorrelator.
> SubQueryFilterRemoveRule failed to decorrelate subquery for TPCH q17
>
>
> Key: CALCITE-3918
> URL: https://issues.apache.org/jira/browse/CALCITE-3918
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Disable RelDecorrelator and run TpchTest.testQuery17(),
> SubQueryFilterRemoveRule generates plan with Correlate, which is not expected.
> {code:java}
> EnumerableProject(AVG_YEARLY=[/($0, 7.0:DECIMAL(2, 1))])
> EnumerableAggregate(group=[{}], agg#0=[SUM($2)])
> EnumerableFilter(condition=[AND(=($3, $0), =(CAST($4):VARCHAR,
> 'Brand#13'), =(CAST($5):VARCHAR, 'JUMBO CAN'), <($1, $6))])
> EnumerableCorrelate(correlation=[$cor0], joinType=[left],
> requiredColumns=[{3}])
> EnumerableNestedLoopJoin(condition=[true], joinType=[inner])
> EnumerableProject(L_PARTKEY=[$1], L_QUANTITY=[$4],
> L_EXTENDEDPRICE=[$5])
> EnumerableTableScan(table=[[TPCH_01, LINEITEM]])
> EnumerableProject(P_PARTKEY=[$0], P_BRAND=[$3], P_CONTAINER=[$6])
> EnumerableTableScan(table=[[TPCH_01, PART]])
> EnumerableProject($f0=[*(0.2:DECIMAL(2, 1), CAST(/(CASE(=($1, 0),
> null:JavaType(class java.lang.Long), $0), $1)):JavaType(class
> java.lang.Long))])
> EnumerableAggregate(group=[{}], agg#0=[$SUM0($4)],
> agg#1=[COUNT($4)])
> EnumerableFilter(condition=[=($1, $cor0.P_PARTKEY)])
> EnumerableTableScan(table=[[TPCH_01, LINEITEM]])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3918) SubQueryFilterRemoveRule failed to decorrelate subquery for TPCH q17
[
https://issues.apache.org/jira/browse/CALCITE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087318#comment-17087318
]
Haisheng Yuan commented on CALCITE-3918:
I am expecting SubQueryFilterRemoveRule can decorrelate the plan. Am I getting
it wrong?
> SubQueryFilterRemoveRule failed to decorrelate subquery for TPCH q17
>
>
> Key: CALCITE-3918
> URL: https://issues.apache.org/jira/browse/CALCITE-3918
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Disable RelDecorrelator and run TpchTest.testQuery17(),
> SubQueryFilterRemoveRule generates plan with Correlate, which is not expected.
> {code:java}
> EnumerableProject(AVG_YEARLY=[/($0, 7.0:DECIMAL(2, 1))])
> EnumerableAggregate(group=[{}], agg#0=[SUM($2)])
> EnumerableFilter(condition=[AND(=($3, $0), =(CAST($4):VARCHAR,
> 'Brand#13'), =(CAST($5):VARCHAR, 'JUMBO CAN'), <($1, $6))])
> EnumerableCorrelate(correlation=[$cor0], joinType=[left],
> requiredColumns=[{3}])
> EnumerableNestedLoopJoin(condition=[true], joinType=[inner])
> EnumerableProject(L_PARTKEY=[$1], L_QUANTITY=[$4],
> L_EXTENDEDPRICE=[$5])
> EnumerableTableScan(table=[[TPCH_01, LINEITEM]])
> EnumerableProject(P_PARTKEY=[$0], P_BRAND=[$3], P_CONTAINER=[$6])
> EnumerableTableScan(table=[[TPCH_01, PART]])
> EnumerableProject($f0=[*(0.2:DECIMAL(2, 1), CAST(/(CASE(=($1, 0),
> null:JavaType(class java.lang.Long), $0), $1)):JavaType(class
> java.lang.Long))])
> EnumerableAggregate(group=[{}], agg#0=[$SUM0($4)],
> agg#1=[COUNT($4)])
> EnumerableFilter(condition=[=($1, $cor0.P_PARTKEY)])
> EnumerableTableScan(table=[[TPCH_01, LINEITEM]])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3918) SubQueryFilterRemoveRule failed to decorrelate subquery for TPCH q17
[
https://issues.apache.org/jira/browse/CALCITE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087324#comment-17087324
]
Haisheng Yuan commented on CALCITE-3918:
Sorry, my fault. Haven't read this piece of code for a long time. Will close
this issue.
> SubQueryFilterRemoveRule failed to decorrelate subquery for TPCH q17
>
>
> Key: CALCITE-3918
> URL: https://issues.apache.org/jira/browse/CALCITE-3918
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Disable RelDecorrelator and run TpchTest.testQuery17(),
> SubQueryFilterRemoveRule generates plan with Correlate, which is not expected.
> {code:java}
> EnumerableProject(AVG_YEARLY=[/($0, 7.0:DECIMAL(2, 1))])
> EnumerableAggregate(group=[{}], agg#0=[SUM($2)])
> EnumerableFilter(condition=[AND(=($3, $0), =(CAST($4):VARCHAR,
> 'Brand#13'), =(CAST($5):VARCHAR, 'JUMBO CAN'), <($1, $6))])
> EnumerableCorrelate(correlation=[$cor0], joinType=[left],
> requiredColumns=[{3}])
> EnumerableNestedLoopJoin(condition=[true], joinType=[inner])
> EnumerableProject(L_PARTKEY=[$1], L_QUANTITY=[$4],
> L_EXTENDEDPRICE=[$5])
> EnumerableTableScan(table=[[TPCH_01, LINEITEM]])
> EnumerableProject(P_PARTKEY=[$0], P_BRAND=[$3], P_CONTAINER=[$6])
> EnumerableTableScan(table=[[TPCH_01, PART]])
> EnumerableProject($f0=[*(0.2:DECIMAL(2, 1), CAST(/(CASE(=($1, 0),
> null:JavaType(class java.lang.Long), $0), $1)):JavaType(class
> java.lang.Long))])
> EnumerableAggregate(group=[{}], agg#0=[$SUM0($4)],
> agg#1=[COUNT($4)])
> EnumerableFilter(condition=[=($1, $cor0.P_PARTKEY)])
> EnumerableTableScan(table=[[TPCH_01, LINEITEM]])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3918) SubQueryFilterRemoveRule failed to decorrelate subquery for TPCH q17
[
https://issues.apache.org/jira/browse/CALCITE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-3918.
Resolution: Not A Problem
> SubQueryFilterRemoveRule failed to decorrelate subquery for TPCH q17
>
>
> Key: CALCITE-3918
> URL: https://issues.apache.org/jira/browse/CALCITE-3918
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Disable RelDecorrelator and run TpchTest.testQuery17(),
> SubQueryFilterRemoveRule generates plan with Correlate, which is not expected.
> {code:java}
> EnumerableProject(AVG_YEARLY=[/($0, 7.0:DECIMAL(2, 1))])
> EnumerableAggregate(group=[{}], agg#0=[SUM($2)])
> EnumerableFilter(condition=[AND(=($3, $0), =(CAST($4):VARCHAR,
> 'Brand#13'), =(CAST($5):VARCHAR, 'JUMBO CAN'), <($1, $6))])
> EnumerableCorrelate(correlation=[$cor0], joinType=[left],
> requiredColumns=[{3}])
> EnumerableNestedLoopJoin(condition=[true], joinType=[inner])
> EnumerableProject(L_PARTKEY=[$1], L_QUANTITY=[$4],
> L_EXTENDEDPRICE=[$5])
> EnumerableTableScan(table=[[TPCH_01, LINEITEM]])
> EnumerableProject(P_PARTKEY=[$0], P_BRAND=[$3], P_CONTAINER=[$6])
> EnumerableTableScan(table=[[TPCH_01, PART]])
> EnumerableProject($f0=[*(0.2:DECIMAL(2, 1), CAST(/(CASE(=($1, 0),
> null:JavaType(class java.lang.Long), $0), $1)):JavaType(class
> java.lang.Long))])
> EnumerableAggregate(group=[{}], agg#0=[$SUM0($4)],
> agg#1=[COUNT($4)])
> EnumerableFilter(condition=[=($1, $cor0.P_PARTKEY)])
> EnumerableTableScan(table=[[TPCH_01, LINEITEM]])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-2057) StackOverflowError when running a JDBC query
[
https://issues.apache.org/jira/browse/CALCITE-2057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17088293#comment-17088293
]
Haisheng Yuan commented on CALCITE-2057:
The query posted by Julian doesn't trigger stackoverflow anymore on 1.23:
{code:java}
select * from (
select a."id" as a,b."id" as b,c."id" as c
from "nullables" a
join "nullables" b on (a."id" = b."id")
join "nullables" c on (a."id" = b."id")
where b."id" <= 1 and c."id" <= 1) z
where a > 1 or b > 1;
!plan
{code}
This is the plan:
{code:java}
> EnumerableNestedLoopJoin(condition=[true], joinType=[inner])
> EnumerableMergeJoin(condition=[AND(=($0, $1), OR(>($0, 1), >($1, 1)))],
> joinType=[inner])
> EnumerableSort(sort0=[$0], dir0=[ASC])
> EnumerableCalc(expr#0..1=[{inputs}], id=[$t0])
> EnumerableTableScan(table=[[s, nullables]])
> EnumerableSort(sort0=[$0], dir0=[ASC])
> EnumerableCalc(expr#0..1=[{inputs}], expr#2=[1], expr#3=[<=($t0, $t2)],
> id=[$t0], $condition=[$t3])
> EnumerableTableScan(table=[[s, nullables]])
> EnumerableCalc(expr#0..1=[{inputs}], expr#2=[1], expr#3=[<=($t0, $t2)],
> id=[$t0], $condition=[$t3])
> EnumerableTableScan(table=[[s, nullables]])
{code}
> StackOverflowError when running a JDBC query
>
>
> Key: CALCITE-2057
> URL: https://issues.apache.org/jira/browse/CALCITE-2057
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Eyal Segal
>Priority: Minor
>
> I've ran the following code:
> {code:java}
> try (ResultSet rs = statement.executeQuery(
> "SELECT p.name, r.views " +
> "FROM reports.a as r " +
> "INNER JOIN main.b as p ON p.id = CAST(r.p_id AS INT) AND
> p.id = 196 " +
> "WHERE p.id = 196 " +
> "AND r.p_id = 196 " +
> "AND r.data_timestamp = TIMESTAMP '2017-11-14 00:00:00'
> " )) {
> {code}
> And I'm getting the following StackOverflowError:
> {code:java}
> Exception in thread "main" java.lang.StackOverflowError
> at
> com.google.common.collect.ImmutableList.indexOf(ImmutableList.java:339)
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source)
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source)
> at
> org.apache.calcite.rel.metadata.RelMetadataQuery.getRowCount(RelMetadataQuery.java:235)
> at
> org.apache.calcite.rel.metadata.RelMdRowCount.getRowCount(RelMdRowCount.java:71)
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source)
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source)
> at
> org.apache.calcite.rel.metadata.RelMetadataQuery.getRowCount(RelMetadataQuery.java:235)
> at
> org.apache.calcite.rel.metadata.RelMdUtil.estimateFilteredRows(RelMdUtil.java:718)
> at
> org.apache.calcite.rel.metadata.RelMdRowCount.getRowCount(RelMdRowCount.java:123)
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source)
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source)
> at
> org.apache.calcite.rel.metadata.RelMetadataQuery.getRowCount(RelMetadataQuery.java:235)
> at
> org.apache.calcite.rel.metadata.RelMdRowCount.getRowCount(RelMdRowCount.java:71)
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source)
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source)
> at
> org.apache.calcite.rel.metadata.RelMetadataQuery.getRowCount(RelMetadataQuery.java:235)
> at
> org.apache.calcite.rel.metadata.RelMdUtil.estimateFilteredRows(RelMdUtil.java:718)
> at
> org.apache.calcite.rel.metadata.RelMdRowCount.getRowCount(RelMdRowCount.java:123)
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source)
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source)
> ...
> {code}
> It's solved by removing "AND p.id = 196" from the INNER JOIN, but it's still
> worth fixing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3868) Remove redundant ruleSet and ruleNames in VolcanoPlanner
[ https://issues.apache.org/jira/browse/CALCITE-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3868. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/90f3e98bb7b2de44a7ac0e2bf1d16cfb09888be1]. > Remove redundant ruleSet and ruleNames in VolcanoPlanner > > > Key: CALCITE-3868 > URL: https://issues.apache.org/jira/browse/CALCITE-3868 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Fix For: 1.23.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > ruleSet and ruleNames in VolcanoPlanner are redundant, remove them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3944) Move dumpSets and dumpGraphviz out of VolcanoPlanner
Haisheng Yuan created CALCITE-3944: -- Summary: Move dumpSets and dumpGraphviz out of VolcanoPlanner Key: CALCITE-3944 URL: https://issues.apache.org/jira/browse/CALCITE-3944 Project: Calcite Issue Type: Improvement Components: core Reporter: Haisheng Yuan They are helper methods, will move to Dumper.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3944) Move dumpSets and dumpGraphviz out of VolcanoPlanner
[ https://issues.apache.org/jira/browse/CALCITE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17088918#comment-17088918 ] Haisheng Yuan commented on CALCITE-3944: Yes, it is named Dumpers.java > Move dumpSets and dumpGraphviz out of VolcanoPlanner > > > Key: CALCITE-3944 > URL: https://issues.apache.org/jira/browse/CALCITE-3944 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > They are helper methods, will move to Dumper.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3927) RelSubset is not fired for rule when set gets merged
[ https://issues.apache.org/jira/browse/CALCITE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3927. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/ff71266352df23208a75b9076045e89f30b9bb0b], thanks for the PR, [~botong]! > RelSubset is not fired for rule when set gets merged > > > Key: CALCITE-3927 > URL: https://issues.apache.org/jira/browse/CALCITE-3927 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Fix For: 1.23.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > In VolcanoPlanner, when set gets merged, planner fires rules again for > RelNodes in both sets, but not for RelSubset. We might miss something because > of this. > If all the logical transformation rules and physical implementation rules are > separated out in different stage and physical rules don't do logical work, we > might be OK. But the reality is that all the things are mixed together at the > moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3937) Fire rule for RelSubset only when it is derived
[ https://issues.apache.org/jira/browse/CALCITE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3937. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/6da208625bb691fa67bfc7cea86cce4ce10657fd]. > Fire rule for RelSubset only when it is derived > > > Key: CALCITE-3937 > URL: https://issues.apache.org/jira/browse/CALCITE-3937 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > It is meaningless to fire rule for RelSubset when it is generated by parent's > trait requirement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3947) AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs
[ https://issues.apache.org/jira/browse/CALCITE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089082#comment-17089082 ] Haisheng Yuan commented on CALCITE-3947: Will the plan be different between different runs? > AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match > order is deterministic across runs > --- > > Key: CALCITE-3947 > URL: https://issues.apache.org/jira/browse/CALCITE-3947 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > AbstractRelOptPlanner.classes is used by subClasses() to determine things to > put into VolcanoPlanner.classOperands, which is then used in > VolcanoPlanner.fireRules(). Since AbstractRelOptPlanner.classes is now a > HashSet, its iteration order is not deterministic across runs, making > debugging hard. It should be LinkedHashSet just like many other fields in the > planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3947) AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs
[ https://issues.apache.org/jira/browse/CALCITE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3947. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/53bbd2daff8104f3a1113eaaa01bf58ca3361beb], thanks for the PR, [~botong]! > AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match > order is deterministic across runs > --- > > Key: CALCITE-3947 > URL: https://issues.apache.org/jira/browse/CALCITE-3947 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Minor > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > AbstractRelOptPlanner.classes is used by subClasses() to determine things to > put into VolcanoPlanner.classOperands, which is then used in > VolcanoPlanner.fireRules(). Since AbstractRelOptPlanner.classes is now a > HashSet, its iteration order is not deterministic across runs, making > debugging hard. It should be LinkedHashSet just like many other fields in the > planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3944) Move dumpSets and dumpGraphviz out of VolcanoPlanner
[ https://issues.apache.org/jira/browse/CALCITE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3944. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/21777f13d39b112cf226a75334ca9f5bae9303c5.|https://github.com/apache/calcite/commit/21777f13d39b112cf226a75334ca9f5bae9303c5] > Move dumpSets and dumpGraphviz out of VolcanoPlanner > > > Key: CALCITE-3944 > URL: https://issues.apache.org/jira/browse/CALCITE-3944 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > They are helper methods, will move to Dumper.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3939) Change UnionEliminatorRule and ProjectRemoveRule to auto pruning SubstitutionRule
[ https://issues.apache.org/jira/browse/CALCITE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3939. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/a136dc4884677bca7496a783c854220834440058], thanks for the PR, [~botong]! > Change UnionEliminatorRule and ProjectRemoveRule to auto pruning > SubstitutionRule > - > > Key: CALCITE-3939 > URL: https://issues.apache.org/jira/browse/CALCITE-3939 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Major > Fix For: 1.23.0 > > Time Spent: 50m > Remaining Estimate: 0h > > UnionEliminatorRule and ProjectRemoveRule are both pruning rules for a > RelNode. They can also become SubstitutionRule with autoprune enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3878) Make ArrayList creation with initial capacity when size is fixed
[ https://issues.apache.org/jira/browse/CALCITE-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089237#comment-17089237 ] Haisheng Yuan commented on CALCITE-3878: Hi [~julianhyde], how do you like the latest changes? > Make ArrayList creation with initial capacity when size is fixed > > > Key: CALCITE-3878 > URL: https://issues.apache.org/jira/browse/CALCITE-3878 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.22.0 >Reporter: neoremind >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > I find many places in Calcite where _new ArrayList<>()_ is used, if the list > is expected to be immutable or not resizing, it is always a good manner to > create with initial capacity, better for memory usage and performance. > I search all occurrences, focus on the core module, to make it safe, I only > update local variables with fixed size and not working in recursive method. > If the local variable reference goes out of scope, if resizing is needed, > things will work normally as well, so no side effect, but for the "escaping" > case, I am very conservative and do not change them. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3422) reOpen CALCITE-1751 tests in PigRelBuilderStyleTest.
[ https://issues.apache.org/jira/browse/CALCITE-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3422. Resolution: Won't Fix > reOpen CALCITE-1751 tests in PigRelBuilderStyleTest. > > > Key: CALCITE-3422 > URL: https://issues.apache.org/jira/browse/CALCITE-3422 > Project: Calcite > Issue Type: Wish >Reporter: xzh_dz >Priority: Minor > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > reOpen CALCITE-1751 tests in PigRelBuilderStyleTest after disable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3949) RelDistributions.of() and RelCollations.of() should canonize trait instance
Haisheng Yuan created CALCITE-3949: -- Summary: RelDistributions.of() and RelCollations.of() should canonize trait instance Key: CALCITE-3949 URL: https://issues.apache.org/jira/browse/CALCITE-3949 Project: Calcite Issue Type: Bug Components: core Reporter: Haisheng Yuan -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3949) RelDistributions.of() and RelCollations.of() should canonize trait instance
[ https://issues.apache.org/jira/browse/CALCITE-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3949: --- Description: Otherwise, in Sort constructor, the assertion might fail. assert traits.containsIfApplicable(collation) > RelDistributions.of() and RelCollations.of() should canonize trait instance > --- > > Key: CALCITE-3949 > URL: https://issues.apache.org/jira/browse/CALCITE-3949 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > > Otherwise, in Sort constructor, the assertion might fail. > assert traits.containsIfApplicable(collation) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089712#comment-17089712 ] Haisheng Yuan commented on CALCITE-3932: If we consider the use of ThreadPool, we probably still need evictable cache, e.g. weak interner. > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3758) FilterTableScanRule generate wrong mapping for filter condition when underlying is BindableTableScan
[ https://issues.apache.org/jira/browse/CALCITE-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3758. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/55c58538256119abbac0a260b8ea63f107f2407c, thanks for the PR, [~jinxing6...@126.com]! > FilterTableScanRule generate wrong mapping for filter condition when > underlying is BindableTableScan > > > Key: CALCITE-3758 > URL: https://issues.apache.org/jira/browse/CALCITE-3758 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Jin Xing >Assignee: Jin Xing >Priority: Major > Labels: pull-request-available > Fix For: 1.23.0 > > Time Spent: 40m > Remaining Estimate: 0h > > When FilterTableScanRule push filter condition into > ProjectableFilterableTable, the filter condition should be adjusted to be > based on the internal table of BindableTableScan. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3948) Improve operand's RelSubset matching handling in VolcanoRuleCall
[ https://issues.apache.org/jira/browse/CALCITE-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3948. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/d3286ad4725fa97aae35c70f73ab768fabd409d8]. > Improve operand's RelSubset matching handling in VolcanoRuleCall > > > Key: CALCITE-3948 > URL: https://issues.apache.org/jira/browse/CALCITE-3948 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Major > Fix For: 1.23.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > For operands matching for a RelSubset, more handling under various cases are > needed to be consistent in VolcanoRuleCall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3915) Add rule listener to report rule attempts and time at DEBUG log level
[
https://issues.apache.org/jira/browse/CALCITE-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-3915.
Fix Version/s: 1.23.0
Resolution: Fixed
Fixed in
[https://github.com/apache/calcite/commit/789a2ae7f4de582c6de16c2bcec747372f941b44],
thanks for the PR, [~xndai]!
> Add rule listener to report rule attempts and time at DEBUG log level
> -
>
> Key: CALCITE-3915
> URL: https://issues.apache.org/jira/browse/CALCITE-3915
> Project: Calcite
> Issue Type: Bug
>Reporter: Xiening Dai
>Assignee: Xiening Dai
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.23.0
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> We can add a rule listener to the VolcanoPlanner for recording rule attempts
> and the accumulative time for the rule firing. This would be very helpful to
> debug planner performance issue (such as CALCITE-2970). The listener will
> only be added under DEBUG log level.
> It will produce a rule attempt summary like below -
> {noformat}
> 2020-04-10 18:03:24,621 [main] DEBUG -
> Rules
> Attempts Time (us)
> ProjectMergeRule:force_mode
> 1,203 1,124,242
> EnumerableProjectRule(in:NONE,out:ENUMERABLE)
> 129 226,475
> AggregatePullUpConstantsRule
> 84 1,949
> AggregateProjectMergeRule
> 81 128,575
> ProjectFilterTransposeRule
> 76 8,734
> JoinPushExpressionsRule
> 71 10,314
> FilterJoinRule:FilterJoinRule:no-filter
> 71 1,306
> JoinPushThroughJoinRule:right
> 47 676
> JoinPushThroughJoinRule:left
> 45 175,696
> FilterJoinRule:FilterJoinRule:filter
> 41 89,381
> ReduceExpressionsRule(Filter)
> 24 38,128
> EnumerableFilterRule(in:NONE,out:ENUMERABLE)
> 24 52,457
> EnumerableJoinRule(in:NONE,out:ENUMERABLE)
> 24 71,752
> EnumerableMergeJoinRule(in:NONE,out:ENUMERABLE)
> 24 47,987
> FilterProjectTransposeRule
> 22 55,177
> JoinCommuteRule
> 20 70,240
> EnumerableAggregateRule(in:NONE,out:ENUMERABLE)
> 10 26,522
> AggregateExpandDistinctAggregatesRule
> 10 37
> EnumerableTableScanRule(in:NONE,out:ENUMERABLE)
> 1 527
> EnumerableInterpreterRule(in:BINDABLE,out:ENUMERABLE)
> 1 764
> BindableTableScanRule
> 1 1,710
> ExpandConversionRule
> 1 175
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090133#comment-17090133
]
Haisheng Yuan commented on CALCITE-2223:
[~rubenql] I think it is resolved by other commits.
> ProjectMergeRule is infinitely matched when is applied after
> ProjectReduceExpressionsRule
> -
>
> Key: CALCITE-2223
> URL: https://issues.apache.org/jira/browse/CALCITE-2223
> Project: Calcite
> Issue Type: Bug
>Reporter: Vova Vysotskyi
>Priority: Critical
> Labels: pull-request-available
> Attachments:
> TestLimitWithExchanges_testPushLimitPastUnionExchange.png, heap_overview.png,
> provenance_contents.png
>
>
> For queries like this:
> {code:sql}
> select t1.f from (select cast(f as int) f, f from (select cast(f as int) f
> from (values('1')) t(f))) as t1
> {code}
> OOM is thrown when {{ProjectMergeRule}} is applied before applying
> {{ProjectReduceExpressionsRule}} in VolcanoPlanner.
> A simple test to reproduce this issue (in {{RelOptRulesTest}}):
> {code:java}
> @Test public void testOomProjectMergeRule() {
> RelBuilder relBuilder =
> RelBuilder.create(RelBuilderTest.config().build());
> RelNode relNode = relBuilder
> .values(new String[]{"f"}, "1")
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f"))
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f0"),
> relBuilder.alias(relBuilder.field(0), "f"))
> .project(
> relBuilder.alias(relBuilder.field(0), "f"))
> .build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> RuleSet ruleSet =
> RuleSets.ofList(
> ReduceExpressionsRule.PROJECT_INSTANCE,
> new ProjectMergeRuleWithLongerName(),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_VALUES_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> relNode.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(),
> ImmutableList.of());
> // check for output
> }
> /**
>* ProjectMergeRule inheritor which has
>* class name greater than ProjectReduceExpressionsRule class name
> (String.compareTo()).
>*
>* It is needed for RuleQueue.popMatch() method
>* to apply this rule before ProjectReduceExpressionsRule.
>*/
> private static class ProjectMergeRuleWithLongerName extends
> ProjectMergeRule {
> public ProjectMergeRuleWithLongerName() {
> super(true, RelFactories.LOGICAL_BUILDER);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3878) Make ArrayList creation with initial capacity when size is fixed
[ https://issues.apache.org/jira/browse/CALCITE-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090154#comment-17090154 ] Haisheng Yuan commented on CALCITE-3878: I will commit this patch if there are no more objections from anyone on the latest change in 72 hours, don't want our contributors wait too long. > Make ArrayList creation with initial capacity when size is fixed > > > Key: CALCITE-3878 > URL: https://issues.apache.org/jira/browse/CALCITE-3878 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.22.0 >Reporter: neoremind >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > I find many places in Calcite where _new ArrayList<>()_ is used, if the list > is expected to be immutable or not resizing, it is always a good manner to > create with initial capacity, better for memory usage and performance. > I search all occurrences, focus on the core module, to make it safe, I only > update local variables with fixed size and not working in recursive method. > If the local variable reference goes out of scope, if resizing is needed, > things will work normally as well, so no side effect, but for the "escaping" > case, I am very conservative and do not change them. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090165#comment-17090165 ] Haisheng Yuan commented on CALCITE-3932: Thanks for your advice. [~Junzheng Zheng] I opened a PR. [~julianhyde] [~danny0405] [~vladimirsitnikov] Can you take a look? > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3932: --- External issue URL: https://github.com/apache/calcite/pull/1939 > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3932: --- External issue URL: (was: https://github.com/apache/calcite/pull/1939) > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090165#comment-17090165 ] Haisheng Yuan edited comment on CALCITE-3932 at 4/23/20, 2:14 AM: -- Thanks for your advice. [~Junzheng Zheng] I opened a [PR|[https://github.com/apache/calcite/pull/1939]]. [~julianhyde] [~danny0405] [~vladimirsitnikov] Can you take a look? was (Author: hyuan): Thanks for your advice. [~Junzheng Zheng] I opened a PR. [~julianhyde] [~danny0405] [~vladimirsitnikov] Can you take a look? > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3932: --- Comment: was deleted (was: Thanks for your advice. [~Junzheng Zheng] I opened a [PR|[https://github.com/apache/calcite/pull/1939]]. [~julianhyde] [~danny0405] [~vladimirsitnikov] Can you take a look?) > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3952) Improve SortRemoveRule to remove Sort based on rowcount
[
https://issues.apache.org/jira/browse/CALCITE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090212#comment-17090212
]
Haisheng Yuan commented on CALCITE-3952:
What if we have limit 0?
> Improve SortRemoveRule to remove Sort based on rowcount
> ---
>
> Key: CALCITE-3952
> URL: https://issues.apache.org/jira/browse/CALCITE-3952
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> If a query is guaranteed to produce maximum one row it is safe to remove Sort
> (along with limit).
> Example:
> {code:sql}
> select count(*) cs from store_sales where ss_ext_sales_price > 100.00 order
> by cs limit 100
> {code}
> Although logically equivalent this can greatly benefit physical plans by
> removing extra operator and avoiding unnecessary data transfer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3952) Improve SortRemoveRule to remove Sort based on rowcount
[
https://issues.apache.org/jira/browse/CALCITE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090221#comment-17090221
]
Haisheng Yuan commented on CALCITE-3952:
OK, got it. thanks.
> Improve SortRemoveRule to remove Sort based on rowcount
> ---
>
> Key: CALCITE-3952
> URL: https://issues.apache.org/jira/browse/CALCITE-3952
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> If a query is guaranteed to produce maximum one row it is safe to remove Sort
> (along with limit).
> Example:
> {code:sql}
> select count(*) cs from store_sales where ss_ext_sales_price > 100.00 order
> by cs limit 100
> {code}
> Although logically equivalent this can greatly benefit physical plans by
> removing extra operator and avoiding unnecessary data transfer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3952) Improve SortRemoveRule to remove Sort based on rowcount
[
https://issues.apache.org/jira/browse/CALCITE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090225#comment-17090225
]
Haisheng Yuan commented on CALCITE-3952:
Oh, I even didn't know we can generate baseline xml file, I always manually
modify the xml file... like an idiot.
> Improve SortRemoveRule to remove Sort based on rowcount
> ---
>
> Key: CALCITE-3952
> URL: https://issues.apache.org/jira/browse/CALCITE-3952
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> If a query is guaranteed to produce maximum one row it is safe to remove Sort
> (along with limit).
> Example:
> {code:sql}
> select count(*) cs from store_sales where ss_ext_sales_price > 100.00 order
> by cs limit 100
> {code}
> Although logically equivalent this can greatly benefit physical plans by
> removing extra operator and avoiding unnecessary data transfer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3952) Improve SortRemoveRule to remove Sort based on rowcount
[
https://issues.apache.org/jira/browse/CALCITE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090240#comment-17090240
]
Haisheng Yuan commented on CALCITE-3952:
That makes sense. deptno is not the output column of Aggregate.
> Improve SortRemoveRule to remove Sort based on rowcount
> ---
>
> Key: CALCITE-3952
> URL: https://issues.apache.org/jira/browse/CALCITE-3952
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> If a query is guaranteed to produce maximum one row it is safe to remove Sort
> (along with limit).
> Example:
> {code:sql}
> select count(*) cs from store_sales where ss_ext_sales_price > 100.00 order
> by cs limit 100
> {code}
> Although logically equivalent this can greatly benefit physical plans by
> removing extra operator and avoiding unnecessary data transfer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090927#comment-17090927 ] Haisheng Yuan commented on CALCITE-3932: For thread local, even it is long-lived, as long as the cache is evictable, we are fine. But for the case of materialized views shared among connections, that's indeed a problem. I will close it with won't fix. > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3932. Resolution: Won't Fix > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3952) Improve SortRemoveRule to remove Sort based on rowcount
[
https://issues.apache.org/jira/browse/CALCITE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091027#comment-17091027
]
Haisheng Yuan commented on CALCITE-3952:
[~vgarg] select count(*) as c from foo order by c limit 100 offset 10;
It should return empty.
> Improve SortRemoveRule to remove Sort based on rowcount
> ---
>
> Key: CALCITE-3952
> URL: https://issues.apache.org/jira/browse/CALCITE-3952
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If a query is guaranteed to produce maximum one row it is safe to remove Sort
> (along with limit).
> Example:
> {code:sql}
> select count(*) cs from store_sales where ss_ext_sales_price > 100.00 order
> by cs limit 100
> {code}
> Although logically equivalent this can greatly benefit physical plans by
> removing extra operator and avoiding unnecessary data transfer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3952) Improve SortRemoveRule to remove Sort based on rowcount
[
https://issues.apache.org/jira/browse/CALCITE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091029#comment-17091029
]
Haisheng Yuan commented on CALCITE-3952:
[~vgarg] Why do we need a configuration to turn it on/off? Isn't it always good?
> Improve SortRemoveRule to remove Sort based on rowcount
> ---
>
> Key: CALCITE-3952
> URL: https://issues.apache.org/jira/browse/CALCITE-3952
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If a query is guaranteed to produce maximum one row it is safe to remove Sort
> (along with limit).
> Example:
> {code:sql}
> select count(*) cs from store_sales where ss_ext_sales_price > 100.00 order
> by cs limit 100
> {code}
> Although logically equivalent this can greatly benefit physical plans by
> removing extra operator and avoiding unnecessary data transfer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3932) Make data type cache thread local, non-evictable
[
https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091078#comment-17091078
]
Haisheng Yuan commented on CALCITE-3932:
Yes, that is feasible. But what if down stream projects already use \{{==}} for
data type comparison? I don't know if there are.
> Make data type cache thread local, non-evictable
>
>
> Key: CALCITE-3932
> URL: https://issues.apache.org/jira/browse/CALCITE-3932
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Fix For: 1.23.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Data type can be evicted out of cache, which is global, thread-safe.
> It seems not necessary to cache them globally, because most of them are
> RelRecordType, which is query dependent, not sharable between different
> queries.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3932) Make data type cache thread local, non-evictable
[ https://issues.apache.org/jira/browse/CALCITE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3932: --- Fix Version/s: (was: 1.23.0) > Make data type cache thread local, non-evictable > > > Key: CALCITE-3932 > URL: https://issues.apache.org/jira/browse/CALCITE-3932 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Data type can be evicted out of cache, which is global, thread-safe. > It seems not necessary to cache them globally, because most of them are > RelRecordType, which is query dependent, not sharable between different > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3957) AggregateMergeRule should be able to merge SUM0 into COUNT even with empty group by
[ https://issues.apache.org/jira/browse/CALCITE-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091907#comment-17091907 ] Haisheng Yuan commented on CALCITE-3957: Agree. Here is the context: [https://github.com/apache/calcite/pull/1087#issuecomment-480001492] Can someone create a patch for this? Thanks a lot. > AggregateMergeRule should be able to merge SUM0 into COUNT even with empty > group by > --- > > Key: CALCITE-3957 > URL: https://issues.apache.org/jira/browse/CALCITE-3957 > Project: Calcite > Issue Type: Bug >Reporter: Steven Talbot >Priority: Major > > Basically, the comment at > [https://github.com/apache/calcite/blame/571731b80a58eb095ebac7123285c375e7afff90/core/src/main/java/org/apache/calcite/rel/rules/AggregateMergeRule.java#L117-L120] > does not apply if the top aggregate is a SUM0, I believe. Without a fix for > this, there's this one case where this rule falls down and isn't able to > merge something that just got rolled up: otherwise it's a pretty nice > invariant that if one of our standard rollup rules for aggregates (say, in > MaterializedViewRule) just split the aggregate, this rule should be able to > merge it back. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3947) AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs
[ https://issues.apache.org/jira/browse/CALCITE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17092004#comment-17092004 ] Haisheng Yuan commented on CALCITE-3947: Oops, I misinterpreted it. Will revert the commit. > AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match > order is deterministic across runs > --- > > Key: CALCITE-3947 > URL: https://issues.apache.org/jira/browse/CALCITE-3947 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Minor > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > AbstractRelOptPlanner.classes is used by subClasses() to determine things to > put into VolcanoPlanner.classOperands, which is then used in > VolcanoPlanner.fireRules(). Since AbstractRelOptPlanner.classes is now a > HashSet, its iteration order is not deterministic across runs, making > debugging hard. It should be LinkedHashSet just like many other fields in the > planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3947) AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs
[ https://issues.apache.org/jira/browse/CALCITE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17092005#comment-17092005 ] Haisheng Yuan commented on CALCITE-3947: I guess [~botong] is trying to solve the same issue as CALCITE-3836, in a different way. > AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match > order is deterministic across runs > --- > > Key: CALCITE-3947 > URL: https://issues.apache.org/jira/browse/CALCITE-3947 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Minor > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > AbstractRelOptPlanner.classes is used by subClasses() to determine things to > put into VolcanoPlanner.classOperands, which is then used in > VolcanoPlanner.fireRules(). Since AbstractRelOptPlanner.classes is now a > HashSet, its iteration order is not deterministic across runs, making > debugging hard. It should be LinkedHashSet just like many other fields in the > planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3947) AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs
[ https://issues.apache.org/jira/browse/CALCITE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3947: --- Fix Version/s: (was: 1.23.0) > AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match > order is deterministic across runs > --- > > Key: CALCITE-3947 > URL: https://issues.apache.org/jira/browse/CALCITE-3947 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > AbstractRelOptPlanner.classes is used by subClasses() to determine things to > put into VolcanoPlanner.classOperands, which is then used in > VolcanoPlanner.fireRules(). Since AbstractRelOptPlanner.classes is now a > HashSet, its iteration order is not deterministic across runs, making > debugging hard. It should be LinkedHashSet just like many other fields in the > planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (CALCITE-3947) AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs
[ https://issues.apache.org/jira/browse/CALCITE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan reopened CALCITE-3947: Reverted in https://github.com/apache/calcite/commit/af8c8efdc465435fa095197ff8365d527147ee7a. > AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match > order is deterministic across runs > --- > > Key: CALCITE-3947 > URL: https://issues.apache.org/jira/browse/CALCITE-3947 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Minor > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > AbstractRelOptPlanner.classes is used by subClasses() to determine things to > put into VolcanoPlanner.classOperands, which is then used in > VolcanoPlanner.fireRules(). Since AbstractRelOptPlanner.classes is now a > HashSet, its iteration order is not deterministic across runs, making > debugging hard. It should be LinkedHashSet just like many other fields in the > planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3949) RelDistributions.of() and RelCollations.of() should canonize trait instance
[ https://issues.apache.org/jira/browse/CALCITE-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094731#comment-17094731 ] Haisheng Yuan commented on CALCITE-3949: PR: https://github.com/apache/calcite/pull/1952 > RelDistributions.of() and RelCollations.of() should canonize trait instance > --- > > Key: CALCITE-3949 > URL: https://issues.apache.org/jira/browse/CALCITE-3949 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Otherwise, in Sort constructor, the assertion might fail. > assert traits.containsIfApplicable(collation) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (CALCITE-3937) Fire rule for RelSubset only when it is derived
[ https://issues.apache.org/jira/browse/CALCITE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan reopened CALCITE-3937: > Fire rule for RelSubset only when it is derived > > > Key: CALCITE-3937 > URL: https://issues.apache.org/jira/browse/CALCITE-3937 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > It is meaningless to fire rule for RelSubset when it is generated by parent's > trait requirement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3937) Fire rule for RelSubset only when it is derived
[ https://issues.apache.org/jira/browse/CALCITE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3937: --- Fix Version/s: (was: 1.23.0) > Fire rule for RelSubset only when it is derived > > > Key: CALCITE-3937 > URL: https://issues.apache.org/jira/browse/CALCITE-3937 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > It is meaningless to fire rule for RelSubset when it is generated by parent's > trait requirement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3937) Fire rule for RelSubset only when it is derived
[ https://issues.apache.org/jira/browse/CALCITE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3937. Resolution: Not A Problem Reverted in https://github.com/apache/calcite/commit/914b6c5d0d58bfdf17d71b6fa37de469e424d969, in case someone just wants to match RelSubset. > Fire rule for RelSubset only when it is derived > > > Key: CALCITE-3937 > URL: https://issues.apache.org/jira/browse/CALCITE-3937 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > It is meaningless to fire rule for RelSubset when it is generated by parent's > trait requirement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3949) RelDistributions.of() and RelCollations.of() should canonize trait instance
[ https://issues.apache.org/jira/browse/CALCITE-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3949. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/11c8868d6a0924c12a0d5c0df0ec4db4dadc3e0a. > RelDistributions.of() and RelCollations.of() should canonize trait instance > --- > > Key: CALCITE-3949 > URL: https://issues.apache.org/jira/browse/CALCITE-3949 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Otherwise, in Sort constructor, the assertion might fail. > assert traits.containsIfApplicable(collation) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3878) Make ArrayList creation with initial capacity when size is fixed
[ https://issues.apache.org/jira/browse/CALCITE-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3878. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/650ce466abf63309dcc0452d35a71dc497fb5f3c, thanks for the PR [~neoremind]! > Make ArrayList creation with initial capacity when size is fixed > > > Key: CALCITE-3878 > URL: https://issues.apache.org/jira/browse/CALCITE-3878 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.22.0 >Reporter: neoremind >Priority: Minor > Labels: pull-request-available > Fix For: 1.23.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > I find many places in Calcite where _new ArrayList<>()_ is used, if the list > is expected to be immutable or not resizing, it is always a good manner to > create with initial capacity, better for memory usage and performance. > I search all occurrences, focus on the core module, to make it safe, I only > update local variables with fixed size and not working in recursive method. > If the local variable reference goes out of scope, if resizing is needed, > things will work normally as well, so no side effect, but for the "escaping" > case, I am very conservative and do not change them. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-2970) Performance issue when enabling abstract converter for EnumerableConvention
[
https://issues.apache.org/jira/browse/CALCITE-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094955#comment-17094955
]
Haisheng Yuan commented on CALCITE-2970:
I am good the current change. [~julianhyde] Can you take a look?
> Performance issue when enabling abstract converter for EnumerableConvention
> ---
>
> Key: CALCITE-2970
> URL: https://issues.apache.org/jira/browse/CALCITE-2970
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 16h 40m
> Remaining Estimate: 0h
>
> If we enable the use of abstract converter for {{EnumerableConvention}}, by
> making {{useAbstractConvertersForConversion}} return true,
> {{JDBCTest.testJoinManyWay}} will not complete.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3916) Support cascades style top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3916: --- Summary: Support cascades style top-down driven rule apply (was: Apply rules bottom up by RelSet) > Support cascades style top-down driven rule apply > - > > Key: CALCITE-3916 > URL: https://issues.apache.org/jira/browse/CALCITE-3916 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Assignee: Haisheng Yuan >Priority: Major > > Apply rules by leaf RelSet -> root RelSet order. For every RelNode in a > RelSet, rule is matched and applied sequentially. No RuleQueue and > DeferringRuleCall is needed anymore. This will make space pruning and rule > mutual exclusivity check possible. > Rule that use AbstractConverter as operand is an exception, to keep backward > compatibility, this kind of rule still needs top-down apply. > This should be done after CALCITE-3896. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-2970) Performance issue when enabling abstract converter for EnumerableConvention
[
https://issues.apache.org/jira/browse/CALCITE-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095582#comment-17095582
]
Haisheng Yuan commented on CALCITE-2970:
LOL. [~zabetak], we had the same thought:
https://github.com/apache/calcite/pull/1884#discussion_r399588178
> Performance issue when enabling abstract converter for EnumerableConvention
> ---
>
> Key: CALCITE-2970
> URL: https://issues.apache.org/jira/browse/CALCITE-2970
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 17h 10m
> Remaining Estimate: 0h
>
> If we enable the use of abstract converter for {{EnumerableConvention}}, by
> making {{useAbstractConvertersForConversion}} return true,
> {{JDBCTest.testJoinManyWay}} will not complete.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-2970) Performance issue when enabling abstract converter for EnumerableConvention
[
https://issues.apache.org/jira/browse/CALCITE-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095582#comment-17095582
]
Haisheng Yuan edited comment on CALCITE-2970 at 4/29/20, 4:04 PM:
--
LOL. [~zabetak], we had the same thought:
https://github.com/apache/calcite/pull/1884#discussion_r399588178
and this:
https://issues.apache.org/jira/browse/CALCITE-2970?focusedCommentId=17085296&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17085296
was (Author: hyuan):
LOL. [~zabetak], we had the same thought:
https://github.com/apache/calcite/pull/1884#discussion_r399588178
> Performance issue when enabling abstract converter for EnumerableConvention
> ---
>
> Key: CALCITE-2970
> URL: https://issues.apache.org/jira/browse/CALCITE-2970
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 17h 10m
> Remaining Estimate: 0h
>
> If we enable the use of abstract converter for {{EnumerableConvention}}, by
> making {{useAbstractConvertersForConversion}} return true,
> {{JDBCTest.testJoinManyWay}} will not complete.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3963) Maintains logical properties at RelSet (equivalent group) instead of RelNode
[
https://issues.apache.org/jira/browse/CALCITE-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17096985#comment-17096985
]
Haisheng Yuan commented on CALCITE-3963:
As long as all the alternatives in a RelSet share the same logical properties,
we don't care where the logical properties are stored.
I am afraid the 'fold' operator will make things complicated. What about
cardinality and selectivity? We may just end up with choosing one blindly. It
doesn't seem right that we use alternative 1's cardinality info, use
alternative 2's selectivity info, and use all the alternatives' unique keys ...
Admittedly, each alternative's stats may vary a lot, one of the reason is that
Calcite believes all the simplification should be done in VolcanoPlanner and
selected based on cost, while other systems like Sql Server and Greenplum do
all the simplification like constant folding, join simplification, predicate
push-down before the logical plan goes into the MEMO.
One of the reason to share logical properties between alternatives in a group
is that it becomes possible (in the future) to do early decision to stop
exploring this group. If we use the 'fold' operator to decide the group's
logical properties, when is it good time to decide?
Option 1: whenever there is a new alternative, recomputing the logical
properties. That may be not better than just storing logical properties for
each relnode.
Option 2: roll it up after all the logical alternatives are generated. But
there is no logical / physical difference, we don't know it is logical operator
or not. Judging by convention is not perfect, because systems like Flink,
Drill, Ignite define their own logical convention. There is no logical rule and
physical rule difference either, they are matched and applied at the same
stage. Physical rules can even generate logical operators, like
ProjectMergeRule, will these generated logical operators be counted?
Another reason to share logical properties is to avoid redundant computation.
For example,
{code:java}
SELECT a,b,c,max(d) FROM foo GROUP BY a,b,c;
HashAggregate
+-- TableScan
{code}
In distributed system, suppose we generate HashAgg with distribution
alternatives of all the 8 key combinations. In SQL Server, there is only 1
physical operator HashAgg, but in Calcite, there are 8 HashAgg operators, the
same HashAgg with different traitset. We will get another 8 exchange operators
(in Calcite 1.22 and before, there were more than 50 exchange operators), we
need to compute the logical properties for all the HashAgg and Exchange
operators, even the result is cached in metadata system, but these operators
are just throwing money that are left on the table by LogicalAggregate operator.
> Maintains logical properties at RelSet (equivalent group) instead of RelNode
>
>
> Key: CALCITE-3963
> URL: https://issues.apache.org/jira/browse/CALCITE-3963
> Project: Calcite
> Issue Type: Bug
>Reporter: Xiening Dai
>Assignee: Xiening Dai
>Priority: Major
>
> Currently the logical properties (such as row count, distinct row count, etc)
> are maintained at RelNode level. This creates a number of meta data
> consistency problems, e.g. CALCITE-1048, CALCITE-2166.
> In theory, all RelNodes in a RelSet should share the same logical properties
> per definition of relational equivalence. So it makes more sense to keep
> logical properties at RelSet level, rather than the RelNode. And such
> properties shouldn't change when new sub set is created or subset's best is
> changed.
> Specifically I think below build in metadata should fall into the logical
> properties category -
> Selectivity
> UniqueKeys
> ColumnUniqueness
> RowCount
> MaxRowCount
> MinRowCount
> DistinctRowCount
> Size (averageRowSize, averageColumnSize)
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3965) Excessive time waiting on DiffRepository lock
[
https://issues.apache.org/jira/browse/CALCITE-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17097072#comment-17097072
]
Haisheng Yuan commented on CALCITE-3965:
Is it duplicate with CALCITE-3517?
> Excessive time waiting on DiffRepository lock
> -
>
> Key: CALCITE-3965
> URL: https://issues.apache.org/jira/browse/CALCITE-3965
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Laurent Goujon
>Assignee: Laurent Goujon
>Priority: Major
>
> When running the whole test suite from commandline, tests are parallelized
> and gradle/junit tries to use as many cores as possible (16 on my machine).
> But the tests take a very long time, approximatevely 90minutes on my machine,
> and several of them failed because they took too long to complete.
> Using jstack to look at the threads state while tests are running show that
> most of them are waiting on {{DiffRepository}} methods
> ({{DiffRepository#expand}} in most cases) while one of the thread obtained
> the lock (and is usually flushing data on disk).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3923) Refactor how planner rules are parameterized
[
https://issues.apache.org/jira/browse/CALCITE-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17097081#comment-17097081
]
Haisheng Yuan commented on CALCITE-3923:
topProject convention should be always same with botttomProject convention. So
the check is not needed.
> Refactor how planner rules are parameterized
>
>
> Key: CALCITE-3923
> URL: https://issues.apache.org/jira/browse/CALCITE-3923
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
>Priority: Major
>
> People often want different variants of planner rules. An example is
> {{FilterJoinRule}}, which has a 'boolean smart’ parameter, a predicate (which
> returns whether to pull up filter conditions), operands (which determine the
> precise sub-classes of {{RelNode}} that the rule should match) and a
> {{RelBuilderFactory}} (which controls the type of {{RelNode}} created by this
> rule).
> Suppose you have an instance of {{FilterJoinRule}} and you want to change
> {{smart}} from true to false. The {{smart}} parameter is immutable (good!)
> but you can’t easily create a clone of the rule because you don’t know the
> values of the other parameters. Your instance might even be (unbeknownst to
> you) a sub-class with extra parameters and a private constructor.
> So, my proposal is to put all of the config information of a {{RelOptRule}}
> into a single {{config}} parameter that contains all relevant properties.
> Each sub-class of {{RelOptRule}} would have one constructor with just a
> ‘config’ parameter. Each config knows which sub-class of {{RelOptRule}} to
> create. Therefore it is easy to copy a config, change one or more properties,
> and create a new rule instance.
> Adding a property to a rule’s config does not require us to add or deprecate
> any constructors.
> The operands are part of the config, so if you have a rule that matches a
> {{EnumerableFilter}} on an {{EnumerableJoin}} and you want to make it match
> an {{EnumerableFilter}} on an {{EnumerableNestedLoopJoin}}, you can easily
> create one with one changed operand.
> The config is immutable and self-describing, so we can use it to
> automatically generate a unique description for each rule instance.
> (See the email thread [[DISCUSS] Refactor how planner rules are
> parameterized|https://lists.apache.org/thread.html/rfdf6f9b7821988bdd92b0377e3d293443a6376f4773c4c658c891cf9%40%3Cdev.calcite.apache.org%3E].)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3928) Canonicalization doesn't do field trimming before materialized view matching
[
https://issues.apache.org/jira/browse/CALCITE-3928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-3928.
Fix Version/s: 1.23.0
Resolution: Fixed
Fixed in
[https://github.com/apache/calcite/commit/24dd26640db01114c6931d6615b90a63969ffc42],
thanks for the PR, [~xzh_dz]!
> Canonicalization doesn't do field trimming before materialized view matching
>
>
> Key: CALCITE-3928
> URL: https://issues.apache.org/jira/browse/CALCITE-3928
> Project: Calcite
> Issue Type: Bug
>Reporter: Jin Xing
>Priority: Major
> Labels: materializedviews
> Fix For: 1.23.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> If we have query and materialized view as below:
> {code:java}
> query:
> LogicalAggregate(group=[{4}], agg#0=[COUNT($3)])
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], SAL=[$5], DEPTNO=[$7])
> LogicalTableScan(table=[[scott, EMP]])
> mv logic:
> LogicalAggregate(group=[{1}], agg#0=[COUNT($0)])
> LogicalProject(SAL=[$5], DEPTNO=[$7])
> LogicalTableScan(table=[[scott, EMP]])
> {code}
> The semantics of query and mv logic are the same. Materialized view matching
> failed, because field trimming is not done when canonicalizing the plans.
> Currently Calcite does field trimming when convert sql to rel. But my
> company's internal system does materialization detection – – generates &
> transforms & stores the RelNode.
> Shall we add the field trimming when canonicalizing materialized view logic?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3966) Trigger rules for existing RelSubset when it becomes delivered
Haisheng Yuan created CALCITE-3966: -- Summary: Trigger rules for existing RelSubset when it becomes delivered Key: CALCITE-3966 URL: https://issues.apache.org/jira/browse/CALCITE-3966 Project: Calcite Issue Type: Improvement Components: core Reporter: Haisheng Yuan Fix For: 1.23.0 Existing RelSubset may be first created by parent trait request, so its state is required. But when there is a new RelNode in the same RelSet can deliver the same traitset, we mark the subset as derived/delivered too, in which case, we should fire rule for the subset too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-2970) Performance issue when enabling abstract converter for EnumerableConvention
[
https://issues.apache.org/jira/browse/CALCITE-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17097745#comment-17097745
]
Haisheng Yuan commented on CALCITE-2970:
Make sense. I am good with current change. It has been dragging so long, can we
make it as soon as possible? [~julianhyde] [~zabetak] [~danny0405]
> Performance issue when enabling abstract converter for EnumerableConvention
> ---
>
> Key: CALCITE-2970
> URL: https://issues.apache.org/jira/browse/CALCITE-2970
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 18h 20m
> Remaining Estimate: 0h
>
> If we enable the use of abstract converter for {{EnumerableConvention}}, by
> making {{useAbstractConvertersForConversion}} return true,
> {{JDBCTest.testJoinManyWay}} will not complete.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3889) Add apply(Mappings.Mapping) to RelTrait and RelTraitSet
[
https://issues.apache.org/jira/browse/CALCITE-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-3889.
Fix Version/s: 1.23.0
Resolution: Fixed
Fixed in
[https://github.com/apache/calcite/commit/4787cba48eb00190f4477b92386db06cd2c7fb63].
> Add apply(Mappings.Mapping) to RelTrait and RelTraitSet
> ---
>
> Key: CALCITE-3889
> URL: https://issues.apache.org/jira/browse/CALCITE-3889
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.23.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> RelTrait Collation, Distribution have key indices, when we pass down the
> traitset to child or propagate to parent operator, we have to remap these
> keys. It would be nice to have {{apply(Mappings.Mapping)}} on RelTrait and
> RelTraitSet. RelDistribution already has the method, but we may want it on
> every RelTrait except Convention.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3968) Disable JoinPushThroughJoinRule.left by default
Haisheng Yuan created CALCITE-3968: -- Summary: Disable JoinPushThroughJoinRule.left by default Key: CALCITE-3968 URL: https://issues.apache.org/jira/browse/CALCITE-3968 Project: Calcite Issue Type: Improvement Components: core Reporter: Haisheng Yuan JoinPushThroughJoinRule.right does (RS)T -> (RT)S JoinPushThroughJoinRule.left does (RS)T -> (TS)R If JoinCommuteRule is enabled, only enabling JoinPushThroughJoinRule.right can achieve the same alternative with JoinPushThroughJoinRule.left, vice versa (suppose they are connected). So there is no need to enable left and right instances at the same time, which will slow down the optimization dramatically, e.g TPCH q05, q07 in TpchTest.java never finish. There is also the same bug report from [1]. Meanwhile, JoinPushThroughJoinRule matches RelNode.class, which is unnecessary. It should be just a group, or RelSet / RelSubset, as a place holder, because we don't care about what exactly the top join's right child is. But since the rule is designed to work with both HepPlanner and VolcanoPlanner, so just bear with the slowness. [1] https://lists.apache.org/thread.html/r195c267ef15f50aa21bbcefd7bf523f8bf2495b2345fd163e91e3c36%40%3Cdev.calcite.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3968) Disable JoinPushThroughJoinRule.left by default
[ https://issues.apache.org/jira/browse/CALCITE-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098001#comment-17098001 ] Haisheng Yuan commented on CALCITE-3968: ||Left-Hand Bracketed||Right-Hand Bracketed||Equivalence|| |(A inner B) inner C|A inner (B inner C)|Equivalent| |(A left B) inner C|A left (B inner C)|*Not equivalent*| |(A right B) inner C|A right (B inner C)|Equivalent| |(A full B) inner C|A full (B inner C)|*Not equivalent*| |(A inner B) left C|A inner (B left C)|Equivalent| |(A left B) left C|A left (B left C)|Equivalent| |(A right B) left C|A right (B left C)|Equivalent| |(A full B) left C|A full (B left C)|Equivalent| |(A inner B) right C|A inner (B right C)|*Not equivalent*| |(A left B) right C|A left (B right C)|*Not equivalent*| |(A right B) right C|A right (B right C)|Equivalent| |(A full B) right C|A full (B right C)|*Not equivalent*| |(A inner B) full C|A inner (B full C)|*Not equivalent*| |(A left B) full C|A left (B full C)|*Not equivalent*| |(A right B) full C|A right (B full C)|Equivalent| |(A full B) full C|A full (B full C)|Equivalent| The above table is from: [http://blog.ylett.com/2011/09/non-associativity-of-sql-table-joins.html] > Disable JoinPushThroughJoinRule.left by default > --- > > Key: CALCITE-3968 > URL: https://issues.apache.org/jira/browse/CALCITE-3968 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > JoinPushThroughJoinRule.right does > (RS)T -> (RT)S > JoinPushThroughJoinRule.left does > (RS)T -> (TS)R > If JoinCommuteRule is enabled, only enabling JoinPushThroughJoinRule.right > can achieve the same alternative with JoinPushThroughJoinRule.left, vice > versa (suppose they are connected). So there is no need to enable left and > right instances at the same time, which will slow down the optimization > dramatically, e.g TPCH q05, q07 in TpchTest.java never finish. There is also > the same bug report from [1]. > Meanwhile, JoinPushThroughJoinRule matches RelNode.class, which is > unnecessary. It should be just a group, or RelSet / RelSubset, as a place > holder, because we don't care about what exactly the top join's right child > is. But since the rule is designed to work with both HepPlanner and > VolcanoPlanner, so just bear with the slowness. > [1] > https://lists.apache.org/thread.html/r195c267ef15f50aa21bbcefd7bf523f8bf2495b2345fd163e91e3c36%40%3Cdev.calcite.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3968) Disable JoinPushThroughJoinRule.right by default
[ https://issues.apache.org/jira/browse/CALCITE-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3968: --- Summary: Disable JoinPushThroughJoinRule.right by default (was: Disable JoinPushThroughJoinRule.left by default) > Disable JoinPushThroughJoinRule.right by default > > > Key: CALCITE-3968 > URL: https://issues.apache.org/jira/browse/CALCITE-3968 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > JoinPushThroughJoinRule.right does > (RS)T -> (RT)S > JoinPushThroughJoinRule.left does > (RS)T -> (TS)R > If JoinCommuteRule is enabled, only enabling JoinPushThroughJoinRule.right > can achieve the same alternative with JoinPushThroughJoinRule.left, vice > versa (suppose they are connected). So there is no need to enable left and > right instances at the same time, which will slow down the optimization > dramatically, e.g TPCH q05, q07 in TpchTest.java never finish. There is also > the same bug report from [1]. > Meanwhile, JoinPushThroughJoinRule matches RelNode.class, which is > unnecessary. It should be just a group, or RelSet / RelSubset, as a place > holder, because we don't care about what exactly the top join's right child > is. But since the rule is designed to work with both HepPlanner and > VolcanoPlanner, so just bear with the slowness. > [1] > https://lists.apache.org/thread.html/r195c267ef15f50aa21bbcefd7bf523f8bf2495b2345fd163e91e3c36%40%3Cdev.calcite.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-2970) Performance issue when enabling abstract converter for EnumerableConvention
[
https://issues.apache.org/jira/browse/CALCITE-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098106#comment-17098106
]
Haisheng Yuan commented on CALCITE-2970:
{quote}
I was thinking that even the RelBuilderFactory that is part of the rule could
be possibly replaced with Convention#getRelFactory at some point.
{quote}
[~zabetak], that is indeed true.
> Performance issue when enabling abstract converter for EnumerableConvention
> ---
>
> Key: CALCITE-2970
> URL: https://issues.apache.org/jira/browse/CALCITE-2970
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 18h 20m
> Remaining Estimate: 0h
>
> If we enable the use of abstract converter for {{EnumerableConvention}}, by
> making {{useAbstractConvertersForConversion}} return true,
> {{JDBCTest.testJoinManyWay}} will not complete.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3919) Upgrade ProjectJoinTransposeRule to allow user choose whether to keep join condition
[
https://issues.apache.org/jira/browse/CALCITE-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098114#comment-17098114
]
Haisheng Yuan commented on CALCITE-3919:
That is your special business logic, not generally applicable. You should
create your own rule instead of asking the built-in rule to provide the ability
you want.
> Upgrade ProjectJoinTransposeRule to allow user choose whether to keep join
> condition
>
>
> Key: CALCITE-3919
> URL: https://issues.apache.org/jira/browse/CALCITE-3919
> Project: Calcite
> Issue Type: Improvement
>Reporter: Wang Yanlin
>Assignee: Wang Yanlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, when apply ProjectJoinTransposeRule, the join condition may be
> also pushed, for example
> for sql
> {noformat}
> select emp.ename, sum(bonus.sal) from emp left outer join bonus on emp.ename
> = bonus.ename and floor(emp.sal) > 10 group by emp.ename
> {noformat}
> the algebra is
> {noformat}
> LogicalAggregate(group=[{0}], EXPR$1=[SUM($1)])
> LogicalProject(ENAME=[$1], SAL0=[$11])
> LogicalJoin(condition=[AND(=($1, $9), >(FLOOR($5), 10))], joinType=[left])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalTableScan(table=[[CATALOG, SALES, BONUS]])
> {noformat}
> After applying *ProjectJoinTransposeRule*, the join condition 'floor(emp.sal)
> > 10' is also pushed down,
> {noformat}
> LogicalAggregate(group=[{0}], EXPR$1=[SUM($1)])
> LogicalProject(ENAME=[$0], SAL0=[$3])
> LogicalJoin(condition=[AND(=($0, $2), $1)], joinType=[left])
> LogicalProject(ENAME=[$1], >=[>(FLOOR($5), 10)])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(ENAME=[$0], SAL=[$2])
> LogicalTableScan(table=[[CATALOG, SALES, BONUS]])
> {noformat}
> In some cases, users may want to push Project past Join, but keep the join
> condition unchanged, so we can upgrade *ProjectJoinTransposeRule* to allow
> user choose whether to keep join condition
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3919) Upgrade ProjectJoinTransposeRule to allow user choose whether to keep join condition
[
https://issues.apache.org/jira/browse/CALCITE-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098117#comment-17098117
]
Haisheng Yuan commented on CALCITE-3919:
But for the left outer join, if the join condition column is on the left side,
we don't need to push them down. I doubt how much improvement will be achieved,
unless the condition is a heavy operation, or the output tuples of outer join
bloat a lot. If you are targeting outer join, and the condition on the non-null
generating side, we may consider just disabling push it down. But for the null
generating side, it still doesn't make sense to keep the filter in the join
condition.
> Upgrade ProjectJoinTransposeRule to allow user choose whether to keep join
> condition
>
>
> Key: CALCITE-3919
> URL: https://issues.apache.org/jira/browse/CALCITE-3919
> Project: Calcite
> Issue Type: Improvement
>Reporter: Wang Yanlin
>Assignee: Wang Yanlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, when apply ProjectJoinTransposeRule, the join condition may be
> also pushed, for example
> for sql
> {noformat}
> select emp.ename, sum(bonus.sal) from emp left outer join bonus on emp.ename
> = bonus.ename and floor(emp.sal) > 10 group by emp.ename
> {noformat}
> the algebra is
> {noformat}
> LogicalAggregate(group=[{0}], EXPR$1=[SUM($1)])
> LogicalProject(ENAME=[$1], SAL0=[$11])
> LogicalJoin(condition=[AND(=($1, $9), >(FLOOR($5), 10))], joinType=[left])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalTableScan(table=[[CATALOG, SALES, BONUS]])
> {noformat}
> After applying *ProjectJoinTransposeRule*, the join condition 'floor(emp.sal)
> > 10' is also pushed down,
> {noformat}
> LogicalAggregate(group=[{0}], EXPR$1=[SUM($1)])
> LogicalProject(ENAME=[$0], SAL0=[$3])
> LogicalJoin(condition=[AND(=($0, $2), $1)], joinType=[left])
> LogicalProject(ENAME=[$1], >=[>(FLOOR($5), 10)])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(ENAME=[$0], SAL=[$2])
> LogicalTableScan(table=[[CATALOG, SALES, BONUS]])
> {noformat}
> In some cases, users may want to push Project past Join, but keep the join
> condition unchanged, so we can upgrade *ProjectJoinTransposeRule* to allow
> user choose whether to keep join condition
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3969) Method RelTrait.apply(Mappings.Mapping) throws exception when mapping doesn't cover collation or distribution keys
[
https://issues.apache.org/jira/browse/CALCITE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098485#comment-17098485
]
Haisheng Yuan commented on CALCITE-3969:
Agree. In RexUtil.applyFields(), if the apply() returns null, should just break
the for loop, instead of adding null.
> Method RelTrait.apply(Mappings.Mapping) throws exception when mapping doesn't
> cover collation or distribution keys
> --
>
> Key: CALCITE-3969
> URL: https://issues.apache.org/jira/browse/CALCITE-3969
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Roman Kondakov
>Assignee: Roman Kondakov
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Let's consider we have an input {{(id, name)}} ordered by {{id}} (i.e.
> collation == {{[0]}}). If we have a {{Project("name")}} on the top of this
> input and we apply project's mapping on the collation, we'll end up with
> exception:
> {noformat}
> java.lang.NullPointerException: at index 0
> at
> com.google.common.collect.ObjectArrays.checkElementNotNull(ObjectArrays.java:239)
> at
> com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:230)
> at
> com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:225)
> at
> com.google.common.collect.ImmutableList.construct(ImmutableList.java:281)
> at
> com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:239)
> at org.apache.calcite.rel.RelCollations.of(RelCollations.java:69)
> at org.apache.calcite.rex.RexUtil.apply(RexUtil.java:1271)
> at
> org.apache.calcite.rel.RelCollationImpl.apply(RelCollationImpl.java:122)
> at
> org.apache.calcite.rel.RelCollationImpl.apply(RelCollationImpl.java:40)
> {noformat}
> This happens because the collation field {{id}} is not a part of the mapping.
> The same problem is with distribution trait when distribution keys are not
> covered by the mapping.
> Calcite should handle such situations gracefully. If it is not possible to
> deduce the collation/distribution after the mapping application, we should
> return:
> * {{EMPTY}} collation for {{RelCollation}} trait.
> * {{RANDOM_DISTRIBUTED}} distribution for {{RelDistribution}} trait.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3968) TPCH queries take forever for planning
[ https://issues.apache.org/jira/browse/CALCITE-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3968: --- Summary: TPCH queries take forever for planning (was: Disable JoinPushThroughJoinRule.right by default) > TPCH queries take forever for planning > -- > > Key: CALCITE-3968 > URL: https://issues.apache.org/jira/browse/CALCITE-3968 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > JoinPushThroughJoinRule.right does > (RS)T -> (RT)S > JoinPushThroughJoinRule.left does > (RS)T -> (TS)R > If JoinCommuteRule is enabled, only enabling JoinPushThroughJoinRule.right > can achieve the same alternative with JoinPushThroughJoinRule.left, vice > versa (suppose they are connected). So there is no need to enable left and > right instances at the same time, which will slow down the optimization > dramatically, e.g TPCH q05, q07 in TpchTest.java never finish. There is also > the same bug report from [1]. > Meanwhile, JoinPushThroughJoinRule matches RelNode.class, which is > unnecessary. It should be just a group, or RelSet / RelSubset, as a place > holder, because we don't care about what exactly the top join's right child > is. But since the rule is designed to work with both HepPlanner and > VolcanoPlanner, so just bear with the slowness. > [1] > https://lists.apache.org/thread.html/r195c267ef15f50aa21bbcefd7bf523f8bf2495b2345fd163e91e3c36%40%3Cdev.calcite.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3921) Support TableModify json serialization and deserialization
[
https://issues.apache.org/jira/browse/CALCITE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-3921:
---
Summary: Support TableModify json serialization and deserialization (was:
Support serialize TableModify to json string and deserialize from it)
> Support TableModify json serialization and deserialization
> --
>
> Key: CALCITE-3921
> URL: https://issues.apache.org/jira/browse/CALCITE-3921
> Project: Calcite
> Issue Type: Bug
>Reporter: Wang Yanlin
>Assignee: Wang Yanlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> When trying to serialize TableModify to json string, got
> {noformat}
> java.lang.UnsupportedOperationException: type not serializable: INSERT (type
> org.apache.calcite.rel.core.TableModify.Operation)
> at org.apache.calcite.rel.externalize.RelJson.toJson(RelJson.java:322)
> at
> org.apache.calcite.rel.externalize.RelJsonWriter.put(RelJsonWriter.java:84)
> at
> org.apache.calcite.rel.externalize.RelJsonWriter.explain_(RelJsonWriter.java:67)
> at
> org.apache.calcite.rel.externalize.RelJsonWriter.done(RelJsonWriter.java:129)
> at
> org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:303)
> at org.apache.calcite.plan.RelOptUtil.dumpPlan(RelOptUtil.java:2110)
> at
> org.apache.calcite.plan.RelWriterTest.testTableModify(RelWriterTest.java:922)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:139)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:81)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
> at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchica
[jira] [Resolved] (CALCITE-3921) Support TableModify json serialization and deserialization
[
https://issues.apache.org/jira/browse/CALCITE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-3921.
Fix Version/s: 1.23.0
Resolution: Fixed
Fixed in
[https://github.com/apache/calcite/commit/ac7d2b6b98334b80e2a43ffd9a79f36ea547e410].
> Support TableModify json serialization and deserialization
> --
>
> Key: CALCITE-3921
> URL: https://issues.apache.org/jira/browse/CALCITE-3921
> Project: Calcite
> Issue Type: Bug
>Reporter: Wang Yanlin
>Assignee: Wang Yanlin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.23.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> When trying to serialize TableModify to json string, got
> {noformat}
> java.lang.UnsupportedOperationException: type not serializable: INSERT (type
> org.apache.calcite.rel.core.TableModify.Operation)
> at org.apache.calcite.rel.externalize.RelJson.toJson(RelJson.java:322)
> at
> org.apache.calcite.rel.externalize.RelJsonWriter.put(RelJsonWriter.java:84)
> at
> org.apache.calcite.rel.externalize.RelJsonWriter.explain_(RelJsonWriter.java:67)
> at
> org.apache.calcite.rel.externalize.RelJsonWriter.done(RelJsonWriter.java:129)
> at
> org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:303)
> at org.apache.calcite.plan.RelOptUtil.dumpPlan(RelOptUtil.java:2110)
> at
> org.apache.calcite.plan.RelWriterTest.testTableModify(RelWriterTest.java:922)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:139)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:81)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
> at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.plat
[jira] [Resolved] (CALCITE-3958) Revise documentation of gradle.properties in Cassandra/Piglet and SubstitutionVisitor
[
https://issues.apache.org/jira/browse/CALCITE-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-3958.
Fix Version/s: 1.23.0
Resolution: Fixed
Fixed in
[https://github.com/apache/calcite/commit/eb57665ee1dfff6c1f9529223bafc9e74f54ff42],
thanks for the PR, [~xzh_dz]!
> Revise documentation of gradle.properties in Cassandra/Piglet and
> SubstitutionVisitor
> -
>
> Key: CALCITE-3958
> URL: https://issues.apache.org/jira/browse/CALCITE-3958
> Project: Calcite
> Issue Type: Wish
>Reporter: xzh_dz
>Priority: Major
> Fix For: 1.23.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> 1. Revise artifact.name and description in Cassandra and Piglet
> gradle.properties.
> Cassandra gradle.properties :
> {code:java}
> description=A library designed to abstract away any required dependency on a
> metrics library
> artifact.name=Apache Calcite Avatica Metrics
> {code}
> we should revise artifact.name and description.
> 2. The document of SubstitutionVisitor should contain Calc and Intersect
> operators.
> Add the Calc and Intersect operators to the SubstituionVisitor document.[1]
> [1][SubstituionVisitor|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L116]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3969) Method RelTrait.apply(Mappings.Mapping) throws exception when mapping doesn't cover collation or distribution keys
[
https://issues.apache.org/jira/browse/CALCITE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-3969:
---
Fix Version/s: 1.23.0
> Method RelTrait.apply(Mappings.Mapping) throws exception when mapping doesn't
> cover collation or distribution keys
> --
>
> Key: CALCITE-3969
> URL: https://issues.apache.org/jira/browse/CALCITE-3969
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Roman Kondakov
>Assignee: Roman Kondakov
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.23.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Let's consider we have an input {{(id, name)}} ordered by {{id}} (i.e.
> collation == {{[0]}}). If we have a {{Project("name")}} on the top of this
> input and we apply project's mapping on the collation, we'll end up with
> exception:
> {noformat}
> java.lang.NullPointerException: at index 0
> at
> com.google.common.collect.ObjectArrays.checkElementNotNull(ObjectArrays.java:239)
> at
> com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:230)
> at
> com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:225)
> at
> com.google.common.collect.ImmutableList.construct(ImmutableList.java:281)
> at
> com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:239)
> at org.apache.calcite.rel.RelCollations.of(RelCollations.java:69)
> at org.apache.calcite.rex.RexUtil.apply(RexUtil.java:1271)
> at
> org.apache.calcite.rel.RelCollationImpl.apply(RelCollationImpl.java:122)
> at
> org.apache.calcite.rel.RelCollationImpl.apply(RelCollationImpl.java:40)
> {noformat}
> This happens because the collation field {{id}} is not a part of the mapping.
> The same problem is with distribution trait when distribution keys are not
> covered by the mapping.
> Calcite should handle such situations gracefully. If it is not possible to
> deduce the collation/distribution after the mapping application, we should
> return:
> * {{EMPTY}} collation for {{RelCollation}} trait.
> * {{RANDOM_DISTRIBUTED}} distribution for {{RelDistribution}} trait.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3961) VolcanoPlanner.prunedNodes information is lost when duplicate relNode is discarded
[ https://issues.apache.org/jira/browse/CALCITE-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3961: --- Fix Version/s: 1.23.0 > VolcanoPlanner.prunedNodes information is lost when duplicate relNode is > discarded > -- > > Key: CALCITE-3961 > URL: https://issues.apache.org/jira/browse/CALCITE-3961 > Project: Calcite > Issue Type: Bug >Reporter: Botong Huang >Priority: Major > Fix For: 1.23.0 > > Time Spent: 50m > Remaining Estimate: 0h > > VolcanoPlanner.prunedNodes stores the list of relNodes that are marked > useless. Whenever the planner see two identical relNode (e.g. when Relsets > are merged), one of them are discarded. However, when the preserved node is > not in the pruned list while the discarded one is, this pruned information is > lost. In general, we should preserve this info whenever duplicate relNodes > are discarded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3926) CannotPlanException when an empty LogicalValues requires a certain collation
[
https://issues.apache.org/jira/browse/CALCITE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-3926:
---
Fix Version/s: 1.23.0
> CannotPlanException when an empty LogicalValues requires a certain collation
>
>
> Key: CALCITE-3926
> URL: https://issues.apache.org/jira/browse/CALCITE-3926
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Ruben Q L
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.23.0
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> The issue can be reproduced with the following test (in RelBuilderTest.java):
> {code:java}
> @Test void testEmptyValues() throws Exception {
> final RelBuilder builder = RelBuilder.create(config().build());
> final RelNode root =
> builder
> .scan("DEPT")
> .filter(builder.literal(false)) // generates an empty
> LogicalValues
> .sort(
> builder.field("DNAME"),
> builder.field("DEPTNO"))
> .build();
> try (PreparedStatement preparedStatement = RelRunners.run(root)) {
> final String s =
> CalciteAssert.toString(preparedStatement.executeQuery());
> final String result = "";
> assertThat(s, is(result));
> }
> }
> {code}
> The test fails with the following message:
> {code:java}
> Caused by: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There
> are not enough rules to produce a node with desired properties:
> convention=ENUMERABLE, sort=[1, 0].
> Missing conversion is LogicalValues[convention: NONE -> ENUMERABLE, sort:
> [[0, 1, 2], [1, 2], [2]] -> [1, 0]]
> There is 1 empty subset: rel#14:Subset#0.ENUMERABLE.[1, 0], the relevant part
> of the original plan is as follows
> 1:LogicalValues(tuples=[[]])
> {code}
> The issue seems to be caused by the empty values: Scan + Filter(false) =>
> LogicalValues(tuples=[[]]).
> This empty LogicalValues has as collation (in case of 3 fields): [[0, 1, 2],
> [1, 2], [2]]; and in the plan we are requesting as sort collation [1, 0].
> This could be achieved by: EnumerableSort(sort=[1,0]) +
> EnumerableValues(tuples=[[]]) ; but for some reason this plan is not
> generated.
> It seems the issue was somehow introduced by CALCITE-3886
>
> [https://github.com/apache/calcite/commit/08f4a9889728f94903ff424ed9c20d940b821af5]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3866) ReturnTypes.AGG_SUM may cause "numeric field overflow" on PostgreSQL when generate the sql after using the rule AggregateJoinTransposeRule.EXTENDED.
[
https://issues.apache.org/jira/browse/CALCITE-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-3866:
---
Fix Version/s: 1.23.0
> ReturnTypes.AGG_SUM may cause "numeric field overflow" on PostgreSQL when
> generate the sql after using the rule AggregateJoinTransposeRule.EXTENDED.
> -
>
> Key: CALCITE-3866
> URL: https://issues.apache.org/jira/browse/CALCITE-3866
> Project: Calcite
> Issue Type: Bug
>Reporter: TANG Wen-hui
>Assignee: TANG Wen-hui
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.23.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When we try to generate a sql after applying
> AggregateJoinTransposeRule.EXTENDED, the result sql can not run on
> PostgreSQL, and throws the following exception:
> {code:java}
> PSQLException: ERROR: numeric field overflow Detail: A field with precision
> 7, scale 2 must round to an absolute value less than 10^5.
> {code}
> I found that the main reason is that :
> the return types of sum may have wrong precision when the type of
> its operand is the decimal with precision, for example:
> {code:java}
> @Test public void testSum() {
> String query =
> "select sum(e1.\"store_sales\"), sum(e2.\"store_sales\") from
> \"sales_fact_dec_1998\" as "
> + "e1 , \"sales_fact_dec_1998\" as e2 where e1.\"product_id\" =
> e2.\"product_id\"";
> String expect = "";
> HepProgramBuilder builder = new HepProgramBuilder();
> builder.addRuleClass(FilterJoinRule.class);
> builder.addRuleClass(AggregateProjectMergeRule.class);
> builder.addRuleClass(AggregateJoinTransposeRule.class);
> HepPlanner hepPlanner = new HepPlanner(builder.build());
> RuleSet rules = RuleSets.ofList(FilterJoinRule.FILTER_ON_JOIN,
> FilterJoinRule.JOIN,
> AggregateProjectMergeRule.INSTANCE,
> AggregateJoinTransposeRule.EXTENDED);
> sql(query).withPostgresql().optimize(rules, hepPlanner).ok(expect);
> }
> {code}
> the result generated sql of the query is :
> {code:java}
> SELECT SUM(CAST(\"t\".\"EXPR$0\" * \"t0\".\"$f1\" AS DECIMAL(10, 4))),
> SUM(CAST(\"t\".\"$f2\" * \"t0\".\"EXPR$1\" AS DECIMAL(10, 4)))
> FROM (SELECT \"product_id\", SUM(\"store_sales\") AS \"EXPR$0\", COUNT(*) AS
> \"$f2\"
> FROM \"foodmart\".\"sales_fact_dec_1998\"
> GROUP BY \"product_id\") AS \"t\"
> INNER JOIN (SELECT \"product_id\", COUNT(*) AS \"$f1\", SUM(\"store_sales\")
> AS \"EXPR$1\"
> FROM \"foodmart\".\"sales_fact_dec_1998\"
> GROUP BY \"product_id\") AS \"t0\" ON \"t\".\"product_id\" =
> \"t0\".\"product_id\"
> {code}
> AggregateJoinTransposeRule.EXTENDED generates a Aggregate to sum up the
> sub-totals:
> {code:java}
> // Aggregate above to sum up the sub-totals
> final List newAggCalls = new ArrayList<>();
> final int groupCount = aggregate.getGroupCount();
> final int newLeftWidth = sides.get(0).newInput.getRowType().getFieldCount();
> final List projects =
> new ArrayList<>(
> rexBuilder.identityProjects(relBuilder.peek().getRowType()));
> for (Ord aggCall : Ord.zip(aggregate.getAggCallList())) {
> final SqlAggFunction aggregation = aggCall.e.getAggregation();
> final SqlSplittableAggFunction splitter =
> Objects.requireNonNull(
> aggregation.unwrap(SqlSplittableAggFunction.class));
> final Integer leftSubTotal = sides.get(0).split.get(aggCall.i);
> final Integer rightSubTotal = sides.get(1).split.get(aggCall.i);
> newAggCalls.add(
> splitter.topSplit(rexBuilder, registry(projects),
> groupCount, relBuilder.peek().getRowType(), aggCall.e,
> leftSubTotal == null ? -1 : leftSubTotal,
> rightSubTotal == null ? -1 : rightSubTotal + newLeftWidth));
> }
> public AggregateCall topSplit(RexBuilder rexBuilder,
> Registry extra, int offset, RelDataType inputRowType,
> AggregateCall aggregateCall, int leftSubTotal, int rightSubTotal) {
> final List merges = new ArrayList<>();
> final List fieldList = inputRowType.getFieldList();
> if (leftSubTotal >= 0) {
> final RelDataType type = fieldList.get(leftSubTotal).getType();
> merges.add(rexBuilder.makeInputRef(type, leftSubTotal));
> }
> if (rightSubTotal >= 0) {
> final RelDataType type = fieldList.get(rightSubTotal).getType();
> merges.add(rexBuilder.makeInputRef(type, rightSubTotal));
> }
> RexNode node;
> switch (merges.size()) {
> case 1:
> node = merges.get(0);
> break;
> case 2:
> node = rexBuilder.makeCall(SqlStdOperatorTable.MULTIPLY, merges);
> node = rexBuilder.makeAbstractCast(aggregateCall.type, node);
> break;
> default:
> throw new AssertionError("unexpected count " + merges);
> }
> int o
[jira] [Commented] (CALCITE-3969) Method RelTrait.apply(Mappings.Mapping) throws exception when mapping doesn't cover collation or distribution keys
[
https://issues.apache.org/jira/browse/CALCITE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17100281#comment-17100281
]
Haisheng Yuan commented on CALCITE-3969:
I don't think there is a need for \{{RelTrait.canApply(TargetMapping)}}. e.g.
RelCollation, the result is either another collation or not sorted anymore.
> Method RelTrait.apply(Mappings.Mapping) throws exception when mapping doesn't
> cover collation or distribution keys
> --
>
> Key: CALCITE-3969
> URL: https://issues.apache.org/jira/browse/CALCITE-3969
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Roman Kondakov
>Assignee: Roman Kondakov
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.23.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Let's consider we have an input {{(id, name)}} ordered by {{id}} (i.e.
> collation == {{[0]}}). If we have a {{Project("name")}} on the top of this
> input and we apply project's mapping on the collation, we'll end up with
> exception:
> {noformat}
> java.lang.NullPointerException: at index 0
> at
> com.google.common.collect.ObjectArrays.checkElementNotNull(ObjectArrays.java:239)
> at
> com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:230)
> at
> com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:225)
> at
> com.google.common.collect.ImmutableList.construct(ImmutableList.java:281)
> at
> com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:239)
> at org.apache.calcite.rel.RelCollations.of(RelCollations.java:69)
> at org.apache.calcite.rex.RexUtil.apply(RexUtil.java:1271)
> at
> org.apache.calcite.rel.RelCollationImpl.apply(RelCollationImpl.java:122)
> at
> org.apache.calcite.rel.RelCollationImpl.apply(RelCollationImpl.java:40)
> {noformat}
> This happens because the collation field {{id}} is not a part of the mapping.
> The same problem is with distribution trait when distribution keys are not
> covered by the mapping.
> Calcite should handle such situations gracefully. If it is not possible to
> deduce the collation/distribution after the mapping application, we should
> return:
> * {{EMPTY}} collation for {{RelCollation}} trait.
> * {{RANDOM_DISTRIBUTED}} distribution for {{RelDistribution}} trait.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3718) Support Intersect and Minus in Bindables
[ https://issues.apache.org/jira/browse/CALCITE-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3718. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/0cdb7a8a520eabdf103519a399c148b0df3ba20b]. > Support Intersect and Minus in Bindables > > > Key: CALCITE-3718 > URL: https://issues.apache.org/jira/browse/CALCITE-3718 > Project: Calcite > Issue Type: Wish >Reporter: xzh_dz >Priority: Major > Labels: pull-request-available > Fix For: 1.23.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Intersect and Minus are not supported in Bindables, this pr try support > Intersect and Minus in Bindables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3966) Trigger rules for existing RelSubset when it becomes delivered
[ https://issues.apache.org/jira/browse/CALCITE-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3966. Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/e7224147916d5a9bf43c322894c04b302b3a545d]. > Trigger rules for existing RelSubset when it becomes delivered > -- > > Key: CALCITE-3966 > URL: https://issues.apache.org/jira/browse/CALCITE-3966 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Existing RelSubset may be first created by parent trait request, so its state > is required. But when there is a new RelNode in the same RelSet can deliver > the same traitset, we mark the subset as derived/delivered too, in which > case, we should fire rule for the subset too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
