[jira] [Commented] (CALCITE-3928) Canonicalization doesn't do field trimming before materialized view matching
[
https://issues.apache.org/jira/browse/CALCITE-3928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089277#comment-17089277
]
Jin Xing commented on CALCITE-3928:
---
[~xzh_dz]
Thanks for PR ~ comments left.
> Canonicalization doesn't do field trimming before materialized view matching
>
>
> Key: CALCITE-3928
> URL: https://issues.apache.org/jira/browse/CALCITE-3928
> Project: Calcite
> Issue Type: Bug
>Reporter: Jin Xing
>Priority: Major
> Labels: materializedviews
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> If we have query and materialized view as below:
> {code:java}
> query:
> LogicalAggregate(group=[{4}], agg#0=[COUNT($3)])
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], SAL=[$5], DEPTNO=[$7])
> LogicalTableScan(table=[[scott, EMP]])
> mv logic:
> LogicalAggregate(group=[{1}], agg#0=[COUNT($0)])
> LogicalProject(SAL=[$5], DEPTNO=[$7])
> LogicalTableScan(table=[[scott, EMP]])
> {code}
> The semantics of query and mv logic are the same. Materialized view matching
> failed, because field trimming is not done when canonicalizing the plans.
> Currently Calcite does field trimming when convert sql to rel. But my
> company's internal system does materialization detection – – generates &
> transforms & stores the RelNode.
> Shall we add the field trimming when canonicalizing materialized view logic?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3949) RelDistributions.of() and RelCollations.of() should canonize trait instance
[ https://issues.apache.org/jira/browse/CALCITE-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-3949: --- Description: Otherwise, in Sort constructor, the assertion might fail. assert traits.containsIfApplicable(collation) > RelDistributions.of() and RelCollations.of() should canonize trait instance > --- > > Key: CALCITE-3949 > URL: https://issues.apache.org/jira/browse/CALCITE-3949 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > > Otherwise, in Sort constructor, the assertion might fail. > assert traits.containsIfApplicable(collation) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3949) RelDistributions.of() and RelCollations.of() should canonize trait instance
Haisheng Yuan created CALCITE-3949: -- Summary: RelDistributions.of() and RelCollations.of() should canonize trait instance Key: CALCITE-3949 URL: https://issues.apache.org/jira/browse/CALCITE-3949 Project: Calcite Issue Type: Bug Components: core Reporter: Haisheng Yuan -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3942) Move type-coercion configurations into SqlValidator.Config
[ https://issues.apache.org/jira/browse/CALCITE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved CALCITE-3942. - Resolution: Fixed Fixed in [3807696|https://github.com/apache/calcite/commit/38076961698c5354375be7856944809e71cd5a21] ! > Move type-coercion configurations into SqlValidator.Config > -- > > Key: CALCITE-3942 > URL: https://issues.apache.org/jira/browse/CALCITE-3942 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.22.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > SqlValidator.Config is the new role to config all kinds of configurations of > SqlValidator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3422) reOpen CALCITE-1751 tests in PigRelBuilderStyleTest.
[ https://issues.apache.org/jira/browse/CALCITE-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3422. Resolution: Won't Fix > reOpen CALCITE-1751 tests in PigRelBuilderStyleTest. > > > Key: CALCITE-3422 > URL: https://issues.apache.org/jira/browse/CALCITE-3422 > Project: Calcite > Issue Type: Wish >Reporter: xzh_dz >Priority: Minor > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > reOpen CALCITE-1751 tests in PigRelBuilderStyleTest after disable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3928) Canonicalization doesn't do field trimming before materialized view matching
[
https://issues.apache.org/jira/browse/CALCITE-3928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jin Xing updated CALCITE-3928:
--
Description:
If we have query and materialized view as below:
{code:java}
query:
LogicalAggregate(group=[{4}], agg#0=[COUNT($3)])
LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], SAL=[$5], DEPTNO=[$7])
LogicalTableScan(table=[[scott, EMP]])
mv logic:
LogicalAggregate(group=[{1}], agg#0=[COUNT($0)])
LogicalProject(SAL=[$5], DEPTNO=[$7])
LogicalTableScan(table=[[scott, EMP]])
{code}
The semantics of query and mv logic are the same. Materialized view matching
failed, because field trimming is not done when canonicalizing the plans.
Currently Calcite does field trimming when convert sql to rel. But my
company's internal system does materialization detection – – generates &
transforms & stores the RelNode.
Shall we add the field trimming when canonicalizing materialized view logic?
was:
If we have query and materialized view as below:
{code:java}
query:
LogicalAggregate(group=[{0}], EXPR$1=[afunc($1, $1)])
LogicalProject(a=$0, b=[bfunc($1)])
LogicalTableScan(table=[[default, user_table]])
mv:
LogicalAggregate(group=[{0}], EXPR$1=[afunc($1, $2)])
LogicalProject(a=$0, b=[bfunc($1)], c=[bfunc($1)])
LogicalTableScan(table=[[default, user_table]])
{code}
The semantics of query and mv logic are the same. Materialized view matching
failed, because field trimming is not done when canonicalizing the plans.
Currently Calcite does field trimming when convert sql to rel. But my
company's internal system does materialization detection – – generates &
transforms & stores the RelNode.
Shall we add the field trimming when canonicalizing materialized view logic?
> Canonicalization doesn't do field trimming before materialized view matching
>
>
> Key: CALCITE-3928
> URL: https://issues.apache.org/jira/browse/CALCITE-3928
> Project: Calcite
> Issue Type: Bug
>Reporter: Jin Xing
>Priority: Major
> Labels: materializedviews
>
> If we have query and materialized view as below:
> {code:java}
> query:
> LogicalAggregate(group=[{4}], agg#0=[COUNT($3)])
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], SAL=[$5], DEPTNO=[$7])
> LogicalTableScan(table=[[scott, EMP]])
> mv logic:
> LogicalAggregate(group=[{1}], agg#0=[COUNT($0)])
> LogicalProject(SAL=[$5], DEPTNO=[$7])
> LogicalTableScan(table=[[scott, EMP]])
> {code}
> The semantics of query and mv logic are the same. Materialized view matching
> failed, because field trimming is not done when canonicalizing the plans.
> Currently Calcite does field trimming when convert sql to rel. But my
> company's internal system does materialization detection – – generates &
> transforms & stores the RelNode.
> Shall we add the field trimming when canonicalizing materialized view logic?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3928) Canonicalization doesn't do field trimming before materialized view matching
[
https://issues.apache.org/jira/browse/CALCITE-3928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089249#comment-17089249
]
Jin Xing commented on CALCITE-3928:
---
The case I mentioned is not accurate, [~xzh_dz] I updated your case into Jira
description.
> Canonicalization doesn't do field trimming before materialized view matching
>
>
> Key: CALCITE-3928
> URL: https://issues.apache.org/jira/browse/CALCITE-3928
> Project: Calcite
> Issue Type: Bug
>Reporter: Jin Xing
>Priority: Major
> Labels: materializedviews
>
> If we have query and materialized view as below:
> {code:java}
> query:
> LogicalAggregate(group=[{0}], EXPR$1=[afunc($1, $1)])
> LogicalProject(a=$0, b=[bfunc($1)])
> LogicalTableScan(table=[[default, user_table]])
> mv:
> LogicalAggregate(group=[{0}], EXPR$1=[afunc($1, $2)])
> LogicalProject(a=$0, b=[bfunc($1)], c=[bfunc($1)])
> LogicalTableScan(table=[[default, user_table]])
> {code}
> The semantics of query and mv logic are the same. Materialized view matching
> failed, because field trimming is not done when canonicalizing the plans.
> Currently Calcite does field trimming when convert sql to rel. But my
> company's internal system does materialization detection – – generates &
> transforms & stores the RelNode.
> Shall we add the field trimming when canonicalizing materialized view logic?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3878) Make ArrayList creation with initial capacity when size is fixed
[ https://issues.apache.org/jira/browse/CALCITE-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089237#comment-17089237 ] Haisheng Yuan commented on CALCITE-3878: Hi [~julianhyde], how do you like the latest changes? > Make ArrayList creation with initial capacity when size is fixed > > > Key: CALCITE-3878 > URL: https://issues.apache.org/jira/browse/CALCITE-3878 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.22.0 >Reporter: neoremind >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > I find many places in Calcite where _new ArrayList<>()_ is used, if the list > is expected to be immutable or not resizing, it is always a good manner to > create with initial capacity, better for memory usage and performance. > I search all occurrences, focus on the core module, to make it safe, I only > update local variables with fixed size and not working in recursive method. > If the local variable reference goes out of scope, if resizing is needed, > things will work normally as well, so no side effect, but for the "escaping" > case, I am very conservative and do not change them. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3928) Canonicalization doesn't do field trimming before materialized view matching
[
https://issues.apache.org/jira/browse/CALCITE-3928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089233#comment-17089233
]
xzh_dz commented on CALCITE-3928:
-
Follow [~jinxing6...@126.com] illustrated, A case as below:
{code:java}
query:
LogicalAggregate(group=[{4}], agg#0=[COUNT($3)])
LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], SAL=[$5], DEPTNO=[$7])
LogicalTableScan(table=[[scott, EMP]])
mv logic:
LogicalAggregate(group=[{1}], agg#0=[COUNT($0)])
LogicalProject(SAL=[$5], DEPTNO=[$7])
LogicalTableScan(table=[[scott, EMP]])
{code}
query has the same semantics with mv logic. Materialization reconginze will be
failed because the mv logic unable to express query.
So we should try to add the field trimming for canonicalizing.
> Canonicalization doesn't do field trimming before materialized view matching
>
>
> Key: CALCITE-3928
> URL: https://issues.apache.org/jira/browse/CALCITE-3928
> Project: Calcite
> Issue Type: Bug
>Reporter: Jin Xing
>Priority: Major
> Labels: materializedviews
>
> If we have query and materialized view as below:
> {code:java}
> query:
> LogicalAggregate(group=[{0}], EXPR$1=[afunc($1, $1)])
> LogicalProject(a=$0, b=[bfunc($1)])
> LogicalTableScan(table=[[default, user_table]])
> mv:
> LogicalAggregate(group=[{0}], EXPR$1=[afunc($1, $2)])
> LogicalProject(a=$0, b=[bfunc($1)], c=[bfunc($1)])
> LogicalTableScan(table=[[default, user_table]])
> {code}
> The semantics of query and mv logic are the same. Materialized view matching
> failed, because field trimming is not done when canonicalizing the plans.
> Currently Calcite does field trimming when convert sql to rel. But my
> company's internal system does materialization detection – – generates &
> transforms & stores the RelNode.
> Shall we add the field trimming when canonicalizing materialized view logic?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3939) Change UnionEliminatorRule and ProjectRemoveRule to auto pruning SubstitutionRule
[ https://issues.apache.org/jira/browse/CALCITE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3939. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/a136dc4884677bca7496a783c854220834440058], thanks for the PR, [~botong]! > Change UnionEliminatorRule and ProjectRemoveRule to auto pruning > SubstitutionRule > - > > Key: CALCITE-3939 > URL: https://issues.apache.org/jira/browse/CALCITE-3939 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Major > Fix For: 1.23.0 > > Time Spent: 50m > Remaining Estimate: 0h > > UnionEliminatorRule and ProjectRemoveRule are both pruning rules for a > RelNode. They can also become SubstitutionRule with autoprune enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3944) Move dumpSets and dumpGraphviz out of VolcanoPlanner
[ https://issues.apache.org/jira/browse/CALCITE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3944. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/21777f13d39b112cf226a75334ca9f5bae9303c5.|https://github.com/apache/calcite/commit/21777f13d39b112cf226a75334ca9f5bae9303c5] > Move dumpSets and dumpGraphviz out of VolcanoPlanner > > > Key: CALCITE-3944 > URL: https://issues.apache.org/jira/browse/CALCITE-3944 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > They are helper methods, will move to Dumper.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-2970) Performance issue when enabling abstract converter for EnumerableConvention
[
https://issues.apache.org/jira/browse/CALCITE-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089171#comment-17089171
]
Xiening Dai commented on CALCITE-2970:
--
Hi [~julianhyde], I am not sure I understand. Do you mean
getRelBuilderTransform() is a lambda provided by conventions? How can we have
multiple ones co-existed? Also sub-classing RelBuilder may not be a good idea,
but it's been there for some time. MaxCompute, Flink and other adapters are
already doing that...
> Performance issue when enabling abstract converter for EnumerableConvention
> ---
>
> Key: CALCITE-2970
> URL: https://issues.apache.org/jira/browse/CALCITE-2970
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 13h 10m
> Remaining Estimate: 0h
>
> If we enable the use of abstract converter for {{EnumerableConvention}}, by
> making {{useAbstractConvertersForConversion}} return true,
> {{JDBCTest.testJoinManyWay}} will not complete.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3947) AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs
[ https://issues.apache.org/jira/browse/CALCITE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089093#comment-17089093 ] Botong Huang commented on CALCITE-3947: --- Thanks [~hyuan] and [~julianhyde] for the quick review! > AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match > order is deterministic across runs > --- > > Key: CALCITE-3947 > URL: https://issues.apache.org/jira/browse/CALCITE-3947 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Minor > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > AbstractRelOptPlanner.classes is used by subClasses() to determine things to > put into VolcanoPlanner.classOperands, which is then used in > VolcanoPlanner.fireRules(). Since AbstractRelOptPlanner.classes is now a > HashSet, its iteration order is not deterministic across runs, making > debugging hard. It should be LinkedHashSet just like many other fields in the > planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3947) AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs
[ https://issues.apache.org/jira/browse/CALCITE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3947. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/53bbd2daff8104f3a1113eaaa01bf58ca3361beb], thanks for the PR, [~botong]! > AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match > order is deterministic across runs > --- > > Key: CALCITE-3947 > URL: https://issues.apache.org/jira/browse/CALCITE-3947 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Minor > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > AbstractRelOptPlanner.classes is used by subClasses() to determine things to > put into VolcanoPlanner.classOperands, which is then used in > VolcanoPlanner.fireRules(). Since AbstractRelOptPlanner.classes is now a > HashSet, its iteration order is not deterministic across runs, making > debugging hard. It should be LinkedHashSet just like many other fields in the > planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3947) AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs
[ https://issues.apache.org/jira/browse/CALCITE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089083#comment-17089083 ] Botong Huang commented on CALCITE-3947: --- Yes it will. Since the iteration order changes across runs, the order of rule matched and thus fired are different. > AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match > order is deterministic across runs > --- > > Key: CALCITE-3947 > URL: https://issues.apache.org/jira/browse/CALCITE-3947 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > AbstractRelOptPlanner.classes is used by subClasses() to determine things to > put into VolcanoPlanner.classOperands, which is then used in > VolcanoPlanner.fireRules(). Since AbstractRelOptPlanner.classes is now a > HashSet, its iteration order is not deterministic across runs, making > debugging hard. It should be LinkedHashSet just like many other fields in the > planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3947) AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs
[ https://issues.apache.org/jira/browse/CALCITE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089082#comment-17089082 ] Haisheng Yuan commented on CALCITE-3947: Will the plan be different between different runs? > AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match > order is deterministic across runs > --- > > Key: CALCITE-3947 > URL: https://issues.apache.org/jira/browse/CALCITE-3947 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > AbstractRelOptPlanner.classes is used by subClasses() to determine things to > put into VolcanoPlanner.classOperands, which is then used in > VolcanoPlanner.fireRules(). Since AbstractRelOptPlanner.classes is now a > HashSet, its iteration order is not deterministic across runs, making > debugging hard. It should be LinkedHashSet just like many other fields in the > planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3948) Improve operand's RelSubset matching handling in VolcanoRuleCall
Botong Huang created CALCITE-3948: - Summary: Improve operand's RelSubset matching handling in VolcanoRuleCall Key: CALCITE-3948 URL: https://issues.apache.org/jira/browse/CALCITE-3948 Project: Calcite Issue Type: Improvement Reporter: Botong Huang For operands matching for a RelSubset, more handling under various cases are needed to be consistent in VolcanoRuleCall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3947) AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs
[ https://issues.apache.org/jira/browse/CALCITE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089072#comment-17089072 ] Julian Hyde commented on CALCITE-3947: -- Maybe. > AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match > order is deterministic across runs > --- > > Key: CALCITE-3947 > URL: https://issues.apache.org/jira/browse/CALCITE-3947 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > AbstractRelOptPlanner.classes is used by subClasses() to determine things to > put into VolcanoPlanner.classOperands, which is then used in > VolcanoPlanner.fireRules(). Since AbstractRelOptPlanner.classes is now a > HashSet, its iteration order is not deterministic across runs, making > debugging hard. It should be LinkedHashSet just like many other fields in the > planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3937) Fire rule for RelSubset only when it is derived
[ https://issues.apache.org/jira/browse/CALCITE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3937. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/6da208625bb691fa67bfc7cea86cce4ce10657fd]. > Fire rule for RelSubset only when it is derived > > > Key: CALCITE-3937 > URL: https://issues.apache.org/jira/browse/CALCITE-3937 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > It is meaningless to fire rule for RelSubset when it is generated by parent's > trait requirement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3947) AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs
Botong Huang created CALCITE-3947: - Summary: AbstractRelOptPlanner.classes should be LinkedHashSet so that rule match order is deterministic across runs Key: CALCITE-3947 URL: https://issues.apache.org/jira/browse/CALCITE-3947 Project: Calcite Issue Type: Improvement Reporter: Botong Huang AbstractRelOptPlanner.classes is used by subClasses() to determine things to put into VolcanoPlanner.classOperands, which is then used in VolcanoPlanner.fireRules(). Since AbstractRelOptPlanner.classes is now a HashSet, its iteration order is not deterministic across runs, making debugging hard. It should be LinkedHashSet just like many other fields in the planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3946) Add parser support for MULTISET/SET and VOLATILE modifiers in CREATE TABLE statements
[
https://issues.apache.org/jira/browse/CALCITE-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089027#comment-17089027
]
Julian Hyde commented on CALCITE-3946:
--
Currently Babel does not support {{CREATE TABLE}} (or indeed any DDL). I have
no objection to it moving in that direction.
There is a more divergence between databases in DDL syntax than DML and query
syntax, so we'll have to be, well, careful if DDL statements turn out to be
ambiguous.
I notice that the {{SET}}, {{MULTISET}}, {{VOLATILE}} keywords seem to be
Teradata-specific. I have no problem with that.
The equivalent to VOLATILE in other DBs might be TEMPORARY or TRANSIENT (see
e.g.
[Snowflake|https://docs.snowflake.com/en/user-guide/tables-temp-transient.html]).
> Add parser support for MULTISET/SET and VOLATILE modifiers in CREATE TABLE
> statements
> -
>
> Key: CALCITE-3946
> URL: https://issues.apache.org/jira/browse/CALCITE-3946
> Project: Calcite
> Issue Type: Improvement
> Components: babel
>Affects Versions: 1.22.0
>Reporter: dasch
>Priority: Major
>
> Add support to Calcite's Babel parser for MULTISET/SET and VOLATILE modifiers
> in CREATE TABLE statements.
> The syntax for these statements is:
> CREATE TABLE [SET|MULTISET] [VOLATILE] [IF NOT EXISTS]
> ( , ...);
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3946) Add parser support for MULTISET/SET and VOLATILE modifiers in CREATE TABLE statements
[ https://issues.apache.org/jira/browse/CALCITE-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde updated CALCITE-3946: - Affects Version/s: 1.22.0 > Add parser support for MULTISET/SET and VOLATILE modifiers in CREATE TABLE > statements > - > > Key: CALCITE-3946 > URL: https://issues.apache.org/jira/browse/CALCITE-3946 > Project: Calcite > Issue Type: Improvement > Components: babel >Affects Versions: 1.22.0 >Reporter: dasch >Priority: Major > > Add support to Calcite's Babel parser for MULTISET/SET and VOLATILE modifiers > in CREATE TABLE statements. > The syntax for these statements is: > CREATE TABLE [SET|MULTISET] [VOLATILE] [IF NOT EXISTS] > ( , ...); -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3927) RelSubset is not fired for rule when set gets merged
[ https://issues.apache.org/jira/browse/CALCITE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088982#comment-17088982 ] Botong Huang commented on CALCITE-3927: --- Thanks [~hyuan] and everyone for the review and comments! > RelSubset is not fired for rule when set gets merged > > > Key: CALCITE-3927 > URL: https://issues.apache.org/jira/browse/CALCITE-3927 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Fix For: 1.23.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > In VolcanoPlanner, when set gets merged, planner fires rules again for > RelNodes in both sets, but not for RelSubset. We might miss something because > of this. > If all the logical transformation rules and physical implementation rules are > separated out in different stage and physical rules don't do logical work, we > might be OK. But the reality is that all the things are mixed together at the > moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3927) RelSubset is not fired for rule when set gets merged
[ https://issues.apache.org/jira/browse/CALCITE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3927. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/ff71266352df23208a75b9076045e89f30b9bb0b], thanks for the PR, [~botong]! > RelSubset is not fired for rule when set gets merged > > > Key: CALCITE-3927 > URL: https://issues.apache.org/jira/browse/CALCITE-3927 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Fix For: 1.23.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > In VolcanoPlanner, when set gets merged, planner fires rules again for > RelNodes in both sets, but not for RelSubset. We might miss something because > of this. > If all the logical transformation rules and physical implementation rules are > separated out in different stage and physical rules don't do logical work, we > might be OK. But the reality is that all the things are mixed together at the > moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3946) Add parser support for MULTISET/SET and VOLATILE modifiers in CREATE TABLE statements
dasch created CALCITE-3946: -- Summary: Add parser support for MULTISET/SET and VOLATILE modifiers in CREATE TABLE statements Key: CALCITE-3946 URL: https://issues.apache.org/jira/browse/CALCITE-3946 Project: Calcite Issue Type: Improvement Components: babel Reporter: dasch Add support to Calcite's Babel parser for MULTISET/SET and VOLATILE modifiers in CREATE TABLE statements. The syntax for these statements is: CREATE TABLE [SET|MULTISET] [VOLATILE] [IF NOT EXISTS] ( , ...); -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3944) Move dumpSets and dumpGraphviz out of VolcanoPlanner
[ https://issues.apache.org/jira/browse/CALCITE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088918#comment-17088918 ] Haisheng Yuan commented on CALCITE-3944: Yes, it is named Dumpers.java > Move dumpSets and dumpGraphviz out of VolcanoPlanner > > > Key: CALCITE-3944 > URL: https://issues.apache.org/jira/browse/CALCITE-3944 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > They are helper methods, will move to Dumper.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3927) RelSubset is not fired for rule when set gets merged
[ https://issues.apache.org/jira/browse/CALCITE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088895#comment-17088895 ] Botong Huang edited comment on CALCITE-3927 at 4/21/20, 5:35 PM: - The unit test gives an example of what can happen in real case, where PhysSingleSubsetRule is a physical rule that has an operand matching for RelSubset. When two RelSets having none identical physical traits get merged (e.g. PHYS_CALLING_CONVENTION3 that is in the new RelSet but not in the original RelSet), the RelSubset with the new physical trait that gets merged in will never trigger the rule match again, but it should. In reality, the two physical trait PHYS_CALLING_CONVENTION and PHYS_CALLING_CONVENTION3 with the satisfy relationship can be something like "any" and "hash[0]" was (Author: botong): The unit test gives an example of what can happen in real case, where PhysSingleSubsetRule is a physical rule that has an operand matching for RelSubset. When two RelSets having none identical physical traits get merged (e.g. PHYS_CALLING_CONVENTION3 that is in the new RelSet but not in the original RelSet), the RelSubset with the new physical trait that gets merged in will never trigger the rule match again, but it should. > RelSubset is not fired for rule when set gets merged > > > Key: CALCITE-3927 > URL: https://issues.apache.org/jira/browse/CALCITE-3927 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > In VolcanoPlanner, when set gets merged, planner fires rules again for > RelNodes in both sets, but not for RelSubset. We might miss something because > of this. > If all the logical transformation rules and physical implementation rules are > separated out in different stage and physical rules don't do logical work, we > might be OK. But the reality is that all the things are mixed together at the > moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3944) Move dumpSets and dumpGraphviz out of VolcanoPlanner
[ https://issues.apache.org/jira/browse/CALCITE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088898#comment-17088898 ] Julian Hyde commented on CALCITE-3944: -- If they’re static - and no one besides them needs to create a visitor object or whatever - put them in a util class with a plural name. Easier to invoke from a debugger, etc. > Move dumpSets and dumpGraphviz out of VolcanoPlanner > > > Key: CALCITE-3944 > URL: https://issues.apache.org/jira/browse/CALCITE-3944 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > They are helper methods, will move to Dumper.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3927) RelSubset is not fired for rule when set gets merged
[ https://issues.apache.org/jira/browse/CALCITE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088895#comment-17088895 ] Botong Huang commented on CALCITE-3927: --- The unit test gives an example of what can happen in real case, where PhysSingleSubsetRule is a physical rule that has an operand matching for RelSubset. When two RelSets having none identical physical traits get merged (e.g. PHYS_CALLING_CONVENTION3 that is in the new RelSet but not in the original RelSet), the RelSubset with the new physical trait that gets merged in will never trigger the rule match again, but it should. > RelSubset is not fired for rule when set gets merged > > > Key: CALCITE-3927 > URL: https://issues.apache.org/jira/browse/CALCITE-3927 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > In VolcanoPlanner, when set gets merged, planner fires rules again for > RelNodes in both sets, but not for RelSubset. We might miss something because > of this. > If all the logical transformation rules and physical implementation rules are > separated out in different stage and physical rules don't do logical work, we > might be OK. But the reality is that all the things are mixed together at the > moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3945) Update Streaming documentation
Rui Wang created CALCITE-3945: - Summary: Update Streaming documentation Key: CALCITE-3945 URL: https://issues.apache.org/jira/browse/CALCITE-3945 Project: Calcite Issue Type: Sub-task Reporter: Rui Wang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3927) RelSubset is not fired for rule when set gets merged
[ https://issues.apache.org/jira/browse/CALCITE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088864#comment-17088864 ] Xiening Dai commented on CALCITE-3927: -- Creating test cases by analyzing the code is totally legit. That's so called white box testing. > RelSubset is not fired for rule when set gets merged > > > Key: CALCITE-3927 > URL: https://issues.apache.org/jira/browse/CALCITE-3927 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > In VolcanoPlanner, when set gets merged, planner fires rules again for > RelNodes in both sets, but not for RelSubset. We might miss something because > of this. > If all the logical transformation rules and physical implementation rules are > separated out in different stage and physical rules don't do logical work, we > might be OK. But the reality is that all the things are mixed together at the > moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3944) Move dumpSets and dumpGraphviz out of VolcanoPlanner
Haisheng Yuan created CALCITE-3944: -- Summary: Move dumpSets and dumpGraphviz out of VolcanoPlanner Key: CALCITE-3944 URL: https://issues.apache.org/jira/browse/CALCITE-3944 Project: Calcite Issue Type: Improvement Components: core Reporter: Haisheng Yuan They are helper methods, will move to Dumper.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3925) Support materialized view matching for LogicalCorrelate in SubsitutionVisitor
[ https://issues.apache.org/jira/browse/CALCITE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088631#comment-17088631 ] Jin Xing commented on CALCITE-3925: --- [~xzh_dz] Thanks a lot for PR, comments left ~ > Support materialized view matching for LogicalCorrelate in SubsitutionVisitor > - > > Key: CALCITE-3925 > URL: https://issues.apache.org/jira/browse/CALCITE-3925 > Project: Calcite > Issue Type: Bug >Reporter: Jin Xing >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > LogicalCorrelate is widely used in user SQLs to performs nested-loop join. > We need to support substitution based materialized view matching for it. The > matching algorithm should be able to do Project & Filter compensation when > matching query with materialized view. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3868) Remove redundant ruleSet and ruleNames in VolcanoPlanner
[ https://issues.apache.org/jira/browse/CALCITE-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3868. Fix Version/s: 1.23.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/90f3e98bb7b2de44a7ac0e2bf1d16cfb09888be1]. > Remove redundant ruleSet and ruleNames in VolcanoPlanner > > > Key: CALCITE-3868 > URL: https://issues.apache.org/jira/browse/CALCITE-3868 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Fix For: 1.23.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > ruleSet and ruleNames in VolcanoPlanner are redundant, remove them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3943) Remove the JSON functions keyword from parser
[ https://issues.apache.org/jira/browse/CALCITE-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088412#comment-17088412 ] Jark Wu commented on CALCITE-3943: -- big +1 for this > Remove the JSON functions keyword from parser > - > > Key: CALCITE-3943 > URL: https://issues.apache.org/jira/browse/CALCITE-3943 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.22.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Fix For: 1.23.0 > > > The JSON functions was introduced in CALCITE-2266, we coded the JSON keywords > into the parser and generates the operator directly, so there is no chance > for downstream projects to change the builtin operator and do some override. > This issue tries to remove these keywords from the parser, we should always > generates SqlUnresolvedFunction instead specific operators in the SQL parser. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3943) Remove the JSON functions keyword from parser
Danny Chen created CALCITE-3943: --- Summary: Remove the JSON functions keyword from parser Key: CALCITE-3943 URL: https://issues.apache.org/jira/browse/CALCITE-3943 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.22.0 Reporter: Danny Chen Assignee: Danny Chen Fix For: 1.23.0 The JSON functions was introduced in CALCITE-2266, we coded the JSON keywords into the parser and generates the operator directly, so there is no chance for downstream projects to change the builtin operator and do some override. This issue tries to remove these keywords from the parser, we should always generates SqlUnresolvedFunction instead specific operators in the SQL parser. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3789) Support validation of UNNEST multiple array columns like Presto
[
https://issues.apache.org/jira/browse/CALCITE-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088398#comment-17088398
]
Danny Chen commented on CALCITE-3789:
-
I have no other objections, go ahead with you fix.
> Support validation of UNNEST multiple array columns like Presto
> ---
>
> Key: CALCITE-3789
> URL: https://issues.apache.org/jira/browse/CALCITE-3789
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.21.0
>Reporter: Will Yu
>Assignee: Will Yu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 13h 40m
> Remaining Estimate: 0h
>
> In Presto, users are able to UNNEST multiple array columns and CROSS JOIN
> with the original table . As shown in the [Presto
> doc|https://prestodb.io/docs/current/sql/select.html]:
> {code:sql}
> SELECT numbers, animals, n, a
> FROM (
> VALUES
> (ARRAY[2, 5], ARRAY['dog', 'cat', 'bird']),
> (ARRAY[7, 8, 9], ARRAY['cow', 'pig'])
> ) AS x (numbers, animals)
> CROSS JOIN UNNEST(numbers, animals) AS t (n, a)
> {code}
> yields:
> numbers | animals | n | a
> ---+--+--+--
> [2, 5]| [dog, cat, bird] |2 | dog
> [2, 5]| [dog, cat, bird] |5 | cat
> [2, 5]| [dog, cat, bird] | NULL | bird
> [7, 8, 9] | [cow, pig] |7 | cow
> [7, 8, 9] | [cow, pig] |8 | pig
> [7, 8, 9] | [cow, pig] |9 | NULL
> It seems Calcite does not have such a feature to support this semantics. In
> Calcite and for above SQL, _n_ and _a_ will be identified as alias of
> subfields of numbers.
> The plan will be to introduce a new Presto conformance and enable validation
> of such SQLs
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3942) Move type-coercion configurations into SqlValidator.Config
Danny Chen created CALCITE-3942: --- Summary: Move type-coercion configurations into SqlValidator.Config Key: CALCITE-3942 URL: https://issues.apache.org/jira/browse/CALCITE-3942 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.22.0 Reporter: Danny Chen Assignee: Danny Chen Fix For: 1.23.0 SqlValidator.Config is the new role to config all kinds of configurations of SqlValidator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3927) RelSubset is not fired for rule when set gets merged
[ https://issues.apache.org/jira/browse/CALCITE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088339#comment-17088339 ] Danny Chen commented on CALCITE-3927: - Can we give a real failing case there ? We do not expect to create cases just by analyzing the code. > RelSubset is not fired for rule when set gets merged > > > Key: CALCITE-3927 > URL: https://issues.apache.org/jira/browse/CALCITE-3927 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > In VolcanoPlanner, when set gets merged, planner fires rules again for > RelNodes in both sets, but not for RelSubset. We might miss something because > of this. > If all the logical transformation rules and physical implementation rules are > separated out in different stage and physical rules don't do logical work, we > might be OK. But the reality is that all the things are mixed together at the > moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
