[jira] [Commented] (CALCITE-4486) UNIQUE predicate
[
https://issues.apache.org/jira/browse/CALCITE-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417888#comment-17417888
]
duan xiong commented on CALCITE-4486:
-
[~julianhyde] Hi. This PR has been review for a month. I think it is a good
shape. So If you have free time, Please do the final review. Thanks again.
> UNIQUE predicate
>
>
> Key: CALCITE-4486
> URL: https://issues.apache.org/jira/browse/CALCITE-4486
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Julian Hyde
>Assignee: duan xiong
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Implement the UNIQUE predicate, as specified by the SQL standard.
> Example:
> {code:java}
> UNIQUE (SELECT publishedIn FROM Book
> WHERE authorId = 3)
> NOT UNIQUE (SELECT publishedIn FROM BOOK
> WHERE authorId = 3)
> {code}
> {{UNIQUE}} returns {{FALSE}} if the query returns two or more rows that do
> not include NULL values and are equal. Like {{EXISTS}}, it never returns
> {{UNKNOWN}}.
> More details at
> [JOOQ|https://www.jooq.org/doc/3.0/manual/sql-building/conditional-expressions/unique-predicate/].
> This Issue support the UNIQUE predicate by set {{isExpand}} to return false,
> so sub-queries are converted to {{RexSubQuery}} expressions and handled later
> by {{SubQueryRemoveRule}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4757) In Avatica, support columns of type "NULL" in query results
[
https://issues.apache.org/jira/browse/CALCITE-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417883#comment-17417883
]
duan xiong commented on CALCITE-4757:
-
[~julianhyde] Hi. If you have free time, please review this. Thanks.
> In Avatica, support columns of type "NULL" in query results
> ---
>
> Key: CALCITE-4757
> URL: https://issues.apache.org/jira/browse/CALCITE-4757
> Project: Calcite
> Issue Type: Bug
> Components: avatica
>Affects Versions: avatica-1.18.0
>Reporter: duan xiong
>Assignee: duan xiong
>Priority: Major
> Fix For: avatica-1.19.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When I run:
> {code:java}
> select * from (values(null,true))
> {code}
> Caused by: java.lang.RuntimeException: unknown type 0.
> Looks like AvaticaSite can't handle Types.NULL.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4786) Facilitate use of graalvm native-image compilation
[ https://issues.apache.org/jira/browse/CALCITE-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417879#comment-17417879 ] Julian Hyde commented on CALCITE-4786: -- Sounds like an exciting project. > Facilitate use of graalvm native-image compilation > -- > > Key: CALCITE-4786 > URL: https://issues.apache.org/jira/browse/CALCITE-4786 > Project: Calcite > Issue Type: Improvement > Components: build >Reporter: Jacques Nadeau >Priority: Major > > Right now, there are number of things that make it difficult to use Calcite > with GraalVM native compilation. > There are several reasons that supporting this kind of compilation could be > beneficial: > - Enable use of Calcite as a Lambda with minimal startup-time > - Create a Calcite shared library that can be easily embedded in other > languages > Initially, I would focus this work on core parsing and query planning. > This work was inspired by work on https://substrait.io > Let's use this ticket to track improvements that can be done to enable this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4177) RelJson should throw if asked to deserialize a call to an unknown operator
[
https://issues.apache.org/jira/browse/CALCITE-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde updated CALCITE-4177:
-
Summary: RelJson should throw if asked to deserialize a call to an unknown
operator (was: Throw exception when deserialize SqlOperator fails, do not
return null)
> RelJson should throw if asked to deserialize a call to an unknown operator
> --
>
> Key: CALCITE-4177
> URL: https://issues.apache.org/jira/browse/CALCITE-4177
> Project: Calcite
> Issue Type: Improvement
>Reporter: Wang Yanlin
>Assignee: Wang Yanlin
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Currently, when deserialize a RexNode fails, calcite returns a null value,
> causing NPE in somewhere else. So we cannot get any information about the
> SqlOperator in the stacktrace. This makes it hard to determine the reason
> when deserialize a very long json string.
> {noformat}
> Caused by: java.lang.NullPointerException
> at java.util.Objects.requireNonNull(Objects.java:203)
> at
> org.apache.calcite.rel.core.AggregateCall.(AggregateCall.java:98)
> at
> org.apache.calcite.rel.core.AggregateCall.create(AggregateCall.java:198)
> at
> org.apache.calcite.rel.externalize.RelJsonReader.toAggCall(RelJsonReader.java:289)
> at
> org.apache.calcite.rel.externalize.RelJsonReader.access$500(RelJsonReader.java:59)
> at
> org.apache.calcite.rel.externalize.RelJsonReader$2.getAggregateCalls(RelJsonReader.java:172)
> at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:220)
> at
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:105)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:264)
> at
> org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91)
> at
> org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85)
> at
> org.apache.calcite.plan.RelWriterTest.lambda$deserializeAndDumpToTextFormat$6(RelWriterTest.java:894)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
> ... 67 more
> {noformat}
> Maybe it's better to throw exception instead of return null, adding the name,
> kind, and syntax in the error message
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417875#comment-17417875
]
Julian Hyde commented on CALCITE-4773:
--
In the comments, don't call it {{SqlSingleValueAggFunction}}, call it
{{SINGLE_VALUE}}. Also justify why the simplification is valid; maybe give an
example in the form of a simple SQL query. From that example, people will
understand why the row type is the same (including nullability).
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
> relBuilder.getTypeFactory()
> .createTypeWithNullability(projExprs.get(0).getType(), true);
> final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
> projExprs.get(0));
> relBuilder.push(aggregate).project(cast);
> call.transformTo(relBuilder.build());
> {code}
> Note that the comment assumes that _"singleAggRel produces a nullable type"_,
> but in this particular case, it seems to produce a non-nullable type, so
> probably this piece of code needs to be adapted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3583) Support serialized to json and deserialized from json for Exchange relation operator
[
https://issues.apache.org/jira/browse/CALCITE-3583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417862#comment-17417862
]
Julian Hyde commented on CALCITE-3583:
--
Can you change the subject to "Externalize Exchange operator to and from JSON"?
"Serialize" misleadingly implies that we are using Java serialization
({{interface Serializable}}).
> Support serialized to json and deserialized from json for Exchange relation
> operator
>
>
> Key: CALCITE-3583
> URL: https://issues.apache.org/jira/browse/CALCITE-3583
> Project: Calcite
> Issue Type: Improvement
>Reporter: Wang Yanlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently, serialize Exchange relnode to json will cause exception
> {code:java}
> // RelWriterTest
> @Test public void testExchange() {
> final FrameworkConfig config = RelBuilderTest.config().build();
> final RelBuilder builder = RelBuilder.create(config);
> final RelNode rel = builder
> .scan("EMP")
> .exchange(RelDistributions.hash(ImmutableList.of(0, 1)))
> .build();
> String relJson = RelOptUtil.dumpPlan("", rel,
> SqlExplainFormat.JSON, SqlExplainLevel.EXPPLAN_ATTRIBUTES);
> String s = deserializeAndDumpToTextFormat(getSchema(rel), relJson);
> final String expected = ""
> + "LogicalExchange(distribution=[hash[0, 1]])\n"
> + " LogicalTableScan(table=[[scott, EMP]])\n";
> assertThat(s, isLinux(expected));
> }
> {code}
> got
> {code:java}
> java.lang.UnsupportedOperationException: type not serializable: hash[0, 1]
> (type org.apache.calcite.rel.RelDistributions.RelDistributionImpl)
> at org.apache.calcite.rel.externalize.RelJson.toJson(RelJson.java:290)
> at
> org.apache.calcite.rel.externalize.RelJsonWriter.put(RelJsonWriter.java:83)
> at
> org.apache.calcite.rel.externalize.RelJsonWriter.explain_(RelJsonWriter.java:66)
> at
> org.apache.calcite.rel.externalize.RelJsonWriter.done(RelJsonWriter.java:128)
> at
> org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:299)
> at org.apache.calcite.plan.RelOptUtil.dumpPlan(RelOptUtil.java:1981)
> at
> org.apache.calcite.plan.RelWriterTest.testExchange(RelWriterTest.java:772)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3583) Support serialized to json and deserialized from json for Exchange relation operator
[
https://issues.apache.org/jira/browse/CALCITE-3583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417861#comment-17417861
]
Julian Hyde commented on CALCITE-3583:
--
Is this a duplicate of CALCITE-4778?
> Support serialized to json and deserialized from json for Exchange relation
> operator
>
>
> Key: CALCITE-3583
> URL: https://issues.apache.org/jira/browse/CALCITE-3583
> Project: Calcite
> Issue Type: Improvement
>Reporter: Wang Yanlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently, serialize Exchange relnode to json will cause exception
> {code:java}
> // RelWriterTest
> @Test public void testExchange() {
> final FrameworkConfig config = RelBuilderTest.config().build();
> final RelBuilder builder = RelBuilder.create(config);
> final RelNode rel = builder
> .scan("EMP")
> .exchange(RelDistributions.hash(ImmutableList.of(0, 1)))
> .build();
> String relJson = RelOptUtil.dumpPlan("", rel,
> SqlExplainFormat.JSON, SqlExplainLevel.EXPPLAN_ATTRIBUTES);
> String s = deserializeAndDumpToTextFormat(getSchema(rel), relJson);
> final String expected = ""
> + "LogicalExchange(distribution=[hash[0, 1]])\n"
> + " LogicalTableScan(table=[[scott, EMP]])\n";
> assertThat(s, isLinux(expected));
> }
> {code}
> got
> {code:java}
> java.lang.UnsupportedOperationException: type not serializable: hash[0, 1]
> (type org.apache.calcite.rel.RelDistributions.RelDistributionImpl)
> at org.apache.calcite.rel.externalize.RelJson.toJson(RelJson.java:290)
> at
> org.apache.calcite.rel.externalize.RelJsonWriter.put(RelJsonWriter.java:83)
> at
> org.apache.calcite.rel.externalize.RelJsonWriter.explain_(RelJsonWriter.java:66)
> at
> org.apache.calcite.rel.externalize.RelJsonWriter.done(RelJsonWriter.java:128)
> at
> org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:299)
> at org.apache.calcite.plan.RelOptUtil.dumpPlan(RelOptUtil.java:1981)
> at
> org.apache.calcite.plan.RelWriterTest.testExchange(RelWriterTest.java:772)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4702) Error when executing query with GROUP BY constant via JDBC adapter
[
https://issues.apache.org/jira/browse/CALCITE-4702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417859#comment-17417859
]
Julian Hyde commented on CALCITE-4702:
--
Yes, my suggestion is to always use the rewriting with the dummy table when a
database (e.g., Redshift) does not support GROUP BY TRUE.
I recommend that we never generate "GROUP BY ()" for these or other queries
(even if the original Calcite query has "GROUP BY()") because databases
implement it so inconsistently.
> Error when executing query with GROUP BY constant via JDBC adapter
> --
>
> Key: CALCITE-4702
> URL: https://issues.apache.org/jira/browse/CALCITE-4702
> Project: Calcite
> Issue Type: Bug
> Components: core, jdbc-adapter
>Affects Versions: 1.27.0
>Reporter: Stamatis Zampetakis
>Assignee: Soumyakanti Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following functionally equivalent SQL queries are accepted by Calcite and
> produce a valid plan
> {noformat}
> select avg(salary) from employee group by true
> select avg(salary) from employee group by 'a'
> {noformat}
> but they may fail if they are executed via the JDBC adapter since not all
> DBMS allow grouping by constants expressions. Moreover, what works for one
> may not work for the other.
> +Examples+
> The {{GROUP BY TRUE}} query works in Postgres, and MySQL but fails in
> Redshift with the following exception:
> {noformat}
> com.amazon.redshift.util.RedshiftException: ERROR: non-integer constant in
> GROUP BY
> {noformat}
> The {{GROUP BY 'a'}} query works in MySQL but fails in Postgres with the
> following exception:
> {noformat}
> ERROR: non-integer constant in GROUP BY
> {noformat}
> +Edit:+
> The {{GROUP BY}} constant is similar to {{GROUP BY ()}} "nothing" but as
> shown in the discussion below they are not equivalent. There is a nice [blog
> post|https://blog.jooq.org/2018/05/25/how-to-group-by-nothing-in-sql/]
> listing some limitations of various DBMS when it comes to {{GROUP BY ()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4544) Deprecate Metadata API backed by Java Reflection
[ https://issues.apache.org/jira/browse/CALCITE-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417827#comment-17417827 ] Jesus Camacho Rodriguez commented on CALCITE-4544: -- [~jamesstarr], I'll help reviewing this patch and others related to CALCITE-4539. I took a look at your PR and left a few minor comments. > Deprecate Metadata API backed by Java Reflection > > > Key: CALCITE-4544 > URL: https://issues.apache.org/jira/browse/CALCITE-4544 > Project: Calcite > Issue Type: Improvement >Reporter: James Starr >Assignee: James Starr >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > A large portion of the metadata api is functionally deprecated, we should > make it explicit. > Apache Calcite currently has 2 ways deriving metadata about rel node, one > through runtime generated handler classes that are compiled using Janino, and > the other using java reflection. The former will be referred to as Janino > and the latter as java reflection for brevity. Java reflection is not used > due to performance reason in calcite main, nor is it tested, and using it can > lead to degraded performance due to interactions with caching of the janino > handlers. Both share a common way of registering classes with implementation > by using ReflectiveRelMetadataProvider and ChainedRelMetadataProvider. There > are also 2 different subsystems for supporting metadata caching, one where a > reference to a cache is stored in the metadata query and the other where > caching is a wrapper pattern around the providers. There are also 2 separate > ways for invalidating the caching, one with a counter stored in the planner > used by java reflection api, and another where volcano planner tracks the > parents of all rel nodes that it mutates and then explicitly invalidates > those parents used by the janino api. > So the following could be removed without the lose of any functionality: > * CachingRelMetadataProvider - used for java reflection base metadata caching > * RelOptPlanner.getRelMetadataTimestamp - used by java reflection to > invalidate the metadata cache > * RelOptPlanner.registerMetadataProviders(List list) - > used by java reflection API to support of Hep and Volcano nodes. > * MetadataFactory - used to caching an internal artifact of the > implementation of java reflection API > * HepRelMetadataProvider - used by java reflection to support Hep node > delegation to a child node. > * VolcanoRelMetadataProvider - used by java reflection to support RelSubset > node delegation to a child node. > * AbstractRelNode.metadata - i think was an even older api that now bridges > to the java reflection api > * RelNode.metadata - see above > * MetadataDef.metadtaClass - used by java reflection and for a unique > descriptive name of the janino generated handler. Janino dependency on it is > easily broken. > * ReflectiveRelMetadataProvider. map and metadataClass0 - used by java > reflection implementation > * RelMetadataProvider.apply() - most of the actual implementation java > reflection API > Metadata and MetadataDef could also be removed, however removing it would > require reworking portions of janino code do so. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4702) Error when executing query with GROUP BY constant via JDBC adapter
[
https://issues.apache.org/jira/browse/CALCITE-4702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417819#comment-17417819
]
Stamatis Zampetakis commented on CALCITE-4702:
--
The table below summarizes what happens when we run some variant of a {{GROUP
BY constant}} query in the databases discussed so far in this JIRA.
||Query variant||Redshift||Postgres||MySQL||Informix||
|GROUP BY TRUE|KO|OK|OK|KO|
|GROUP BY 'a'|KO|KO|OK|KO|
|GROUP BY (SELECT TRUE)|OK|OK|OK|KO|
|FROM ... (SELECT true x) dummy GROUP BY dummy.x|OK|OK|OK|OK|
[~jhyde] If I understand well your last comment your suggestion is to always
use the rewriting with the dummy table when a database (e.g., Redshift) does
not support {{GROUP BY TRUE}}. Is that right?
> Error when executing query with GROUP BY constant via JDBC adapter
> --
>
> Key: CALCITE-4702
> URL: https://issues.apache.org/jira/browse/CALCITE-4702
> Project: Calcite
> Issue Type: Bug
> Components: core, jdbc-adapter
>Affects Versions: 1.27.0
>Reporter: Stamatis Zampetakis
>Assignee: Soumyakanti Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following functionally equivalent SQL queries are accepted by Calcite and
> produce a valid plan
> {noformat}
> select avg(salary) from employee group by true
> select avg(salary) from employee group by 'a'
> {noformat}
> but they may fail if they are executed via the JDBC adapter since not all
> DBMS allow grouping by constants expressions. Moreover, what works for one
> may not work for the other.
> +Examples+
> The {{GROUP BY TRUE}} query works in Postgres, and MySQL but fails in
> Redshift with the following exception:
> {noformat}
> com.amazon.redshift.util.RedshiftException: ERROR: non-integer constant in
> GROUP BY
> {noformat}
> The {{GROUP BY 'a'}} query works in MySQL but fails in Postgres with the
> following exception:
> {noformat}
> ERROR: non-integer constant in GROUP BY
> {noformat}
> +Edit:+
> The {{GROUP BY}} constant is similar to {{GROUP BY ()}} "nothing" but as
> shown in the discussion below they are not equivalent. There is a nice [blog
> post|https://blog.jooq.org/2018/05/25/how-to-group-by-nothing-in-sql/]
> listing some limitations of various DBMS when it comes to {{GROUP BY ()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4786) Facilitate use of graalvm native-image compilation
[ https://issues.apache.org/jira/browse/CALCITE-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417801#comment-17417801 ] Jacques Nadeau commented on CALCITE-4786: - For those that are interested, tracking the work to compile within the context of Substrait here: [https://github.com/substrait-io/substrait/issues/21] To give people some sense of the differences using GraalVM vs JVM, I did a non-scientific test: simple cli program that parses a trivial query, converts it to rel and then converts it to a substrait plan (using linux time command). || ||JVM||Native Image|| |Query Parsing & Substrait Conversion Time|1.5s|<0.01s| As mentioned, this would enable use cases such as FaaS/AWS Lambda where startup time are critical. As part of this work I also hope to introduce a C API & shared library for this type of functionality so Calcite's richness can be used from more languages. > Facilitate use of graalvm native-image compilation > -- > > Key: CALCITE-4786 > URL: https://issues.apache.org/jira/browse/CALCITE-4786 > Project: Calcite > Issue Type: Improvement > Components: build >Reporter: Jacques Nadeau >Priority: Major > > Right now, there are number of things that make it difficult to use Calcite > with GraalVM native compilation. > There are several reasons that supporting this kind of compilation could be > beneficial: > - Enable use of Calcite as a Lambda with minimal startup-time > - Create a Calcite shared library that can be easily embedded in other > languages > Initially, I would focus this work on core parsing and query planning. > This work was inspired by work on https://substrait.io > Let's use this ticket to track improvements that can be done to enable this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4787) Evaluate use of Immutables instead of ImmutableBeans
[ https://issues.apache.org/jira/browse/CALCITE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417799#comment-17417799 ] Jacques Nadeau commented on CALCITE-4787: - Posted an initial PR for feedback. > Evaluate use of Immutables instead of ImmutableBeans > > > Key: CALCITE-4787 > URL: https://issues.apache.org/jira/browse/CALCITE-4787 > Project: Calcite > Issue Type: Improvement >Reporter: Jacques Nadeau >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In the creation of CALCITE-3328, [Immutables|https://immutables.github.io/] > was discussed as an alternative to a custom implementation. This ticket is to > evaluate the impact to the codebase of changing. Ideally, introduction of > immutables would both add flexibility and reduce the amount of code > associated with these classes. > Immutables works via annotation processor which means that it is should be > relatively seamless to build systems and IDEs. > The switch would also make it easier to work with these objects types in the > context of aot compilation tools like GraalVM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-4787) Evaluate use of Immutables instead of ImmutableBeans
[ https://issues.apache.org/jira/browse/CALCITE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacques Nadeau reassigned CALCITE-4787: --- Assignee: Jacques Nadeau > Evaluate use of Immutables instead of ImmutableBeans > > > Key: CALCITE-4787 > URL: https://issues.apache.org/jira/browse/CALCITE-4787 > Project: Calcite > Issue Type: Improvement >Reporter: Jacques Nadeau >Assignee: Jacques Nadeau >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In the creation of CALCITE-3328, [Immutables|https://immutables.github.io/] > was discussed as an alternative to a custom implementation. This ticket is to > evaluate the impact to the codebase of changing. Ideally, introduction of > immutables would both add flexibility and reduce the amount of code > associated with these classes. > Immutables works via annotation processor which means that it is should be > relatively seamless to build systems and IDEs. > The switch would also make it easier to work with these objects types in the > context of aot compilation tools like GraalVM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4787) Evaluate use of Immutables instead of ImmutableBeans
[ https://issues.apache.org/jira/browse/CALCITE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-4787: Labels: pull-request-available (was: ) > Evaluate use of Immutables instead of ImmutableBeans > > > Key: CALCITE-4787 > URL: https://issues.apache.org/jira/browse/CALCITE-4787 > Project: Calcite > Issue Type: Improvement >Reporter: Jacques Nadeau >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In the creation of CALCITE-3328, [Immutables|https://immutables.github.io/] > was discussed as an alternative to a custom implementation. This ticket is to > evaluate the impact to the codebase of changing. Ideally, introduction of > immutables would both add flexibility and reduce the amount of code > associated with these classes. > Immutables works via annotation processor which means that it is should be > relatively seamless to build systems and IDEs. > The switch would also make it easier to work with these objects types in the > context of aot compilation tools like GraalVM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3409) Add an interface in MaterializedViewSubstitutionVisitor to allow registering UnifyRule
[
https://issues.apache.org/jira/browse/CALCITE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417703#comment-17417703
]
Jesus Camacho Rodriguez commented on CALCITE-3409:
--
[~xzh_dz], following up on [~zabetak]'s feedback above. Until now, we had not
exposed the {{UnifyRules}} in the planner and they were rather restricted to
the {{SubstitutionVisitor}}. It seems you do not need them to be in the planner
either, instead it would be sufficient for you to have a
{{SubstitutionVisitor}} that is extensible. Maybe we could limit this patch to
the changes in the visitor, and avoid changing the {{RelOptPlanner}} APIs for
the time being. If it turns out that extensibility from the planner is needed,
we can revisit the idea later on. Thoughts?
> Add an interface in MaterializedViewSubstitutionVisitor to allow registering
> UnifyRule
> --
>
> Key: CALCITE-3409
> URL: https://issues.apache.org/jira/browse/CALCITE-3409
> Project: Calcite
> Issue Type: Improvement
>Reporter: Jin Xing
>Assignee: Jin Xing
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> In current code of MaterializedViewSubstitutionVisitor, all matching rules
> are internal defined. The existing rules support the most popular scenarios.
> But my customers sometimes ask for the ability to self define some matching
> rules, thus to support some special scenarios.
> I take below example as an illustration:
> {code:java}
> Query:
> select * from table
> where from_unixtime(_EVENT_TIME_, "mmdd hh") >= "20190909 00"
> and from_unixtime(_EVENT_TIME_, "mmdd hh") <= "20190909 23" ;
> Materialized View:
> select * from table
> where from_unixtime(_EVENT_TIME_, "mmdd") = "20190909";{code}
> It's hard to enumerate the matching pattern for different functions in
> internal matching rules. We can expose a method to register new UnifyRules
> and allow user to extend the ability of MV matching
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3409) Add an interface in MaterializedViewSubstitutionVisitor to allow registering UnifyRule
[
https://issues.apache.org/jira/browse/CALCITE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417679#comment-17417679
]
Stamatis Zampetakis commented on CALCITE-3409:
--
Hey [~xzh_dz], the latest comments seem to imply that the changes in
{{RelOptPlanner}} interface are not very important and the new methods were
added just for testing purposes. If my interpretation is correct then the best
would be to avoid introducing public APIs just for testing.
However, I am not sure how somebody can take advantage of this new feature if
we cannot write a test case using public APIs.
Anyways, I don't have much time to review this properly so it is better to ask
in the dev@ list for somebody to take over this and push it forward.
> Add an interface in MaterializedViewSubstitutionVisitor to allow registering
> UnifyRule
> --
>
> Key: CALCITE-3409
> URL: https://issues.apache.org/jira/browse/CALCITE-3409
> Project: Calcite
> Issue Type: Improvement
>Reporter: Jin Xing
>Assignee: Jin Xing
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> In current code of MaterializedViewSubstitutionVisitor, all matching rules
> are internal defined. The existing rules support the most popular scenarios.
> But my customers sometimes ask for the ability to self define some matching
> rules, thus to support some special scenarios.
> I take below example as an illustration:
> {code:java}
> Query:
> select * from table
> where from_unixtime(_EVENT_TIME_, "mmdd hh") >= "20190909 00"
> and from_unixtime(_EVENT_TIME_, "mmdd hh") <= "20190909 23" ;
> Materialized View:
> select * from table
> where from_unixtime(_EVENT_TIME_, "mmdd") = "20190909";{code}
> It's hard to enumerate the matching pattern for different functions in
> internal matching rules. We can expose a method to register new UnifyRules
> and allow user to extend the ability of MV matching
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4788) Does RelOptRulesTest use foreign keys or not?
[ https://issues.apache.org/jira/browse/CALCITE-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis resolved CALCITE-4788. -- Fix Version/s: 1.28.0 Resolution: Not A Problem > Does RelOptRulesTest use foreign keys or not? > - > > Key: CALCITE-4788 > URL: https://issues.apache.org/jira/browse/CALCITE-4788 > Project: Calcite > Issue Type: Test > Components: core >Affects Versions: 1.27.0 >Reporter: Sicheng Pan >Priority: Minor > Fix For: 1.28.0 > > > It seems that the tests for rewrite rules depends on the famous SCOTT table > from Oracle, yet it also seems unclear if there exists foreign key > constraints between the EMP and DEPT tables. If there are such constraints, > are they considered in any optimization rules (CoreRules)? And if so, are > these constraints stored in RelOptTable? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4788) Does RelOptRulesTest use foreign keys or not?
[
https://issues.apache.org/jira/browse/CALCITE-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417647#comment-17417647
]
Stamatis Zampetakis commented on CALCITE-4788:
--
Hi [~Macronova], for general questions it is better to use the dev@ list
instead of creating JIRAs.
The only effort that I remember around foreign keys in Calcite was done in
CALCITE-1731 for supporting certain view rewrittings. If you want more
information I would suggest to look in the commit which introduced the
respective APIs (CALCITE-1731). You can find also some relevant discussion in
CALCITE-2967.
To answer your question, I don't know of any core rules exploiting foreign key
contraints (apart those for view based rewritting), and I doubt there is any
test in {{RelOptRulesTest}} class.
> Does RelOptRulesTest use foreign keys or not?
> -
>
> Key: CALCITE-4788
> URL: https://issues.apache.org/jira/browse/CALCITE-4788
> Project: Calcite
> Issue Type: Test
> Components: core
>Affects Versions: 1.27.0
>Reporter: Sicheng Pan
>Priority: Minor
>
> It seems that the tests for rewrite rules depends on the famous SCOTT table
> from Oracle, yet it also seems unclear if there exists foreign key
> constraints between the EMP and DEPT tables. If there are such constraints,
> are they considered in any optimization rules (CoreRules)? And if so, are
> these constraints stored in RelOptTable?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4532) Rex simplification fails with an exception on nested CASTs with ANY
[
https://issues.apache.org/jira/browse/CALCITE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-4532:
Labels: pull-request-available (was: )
> Rex simplification fails with an exception on nested CASTs with ANY
> ---
>
> Key: CALCITE-4532
> URL: https://issues.apache.org/jira/browse/CALCITE-4532
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.24.0
>Reporter: Grzegorz Gierlach
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> @Test public void testNestedBigInt() {
> CalciteAssert.AssertThat assertThat = CalciteAssert.that();
> String query = "SELECT CAST(CAST(42 AS BIGINT) AS ANY) FROM
> (VALUES(1))";
> assertThat.query(query).returns("EXPR$0=42\n");
> }
> {code}
> fails with:
> {code:java}
> Error while executing SQL "SELECT CAST(CAST(42 AS BIGINT) AS ANY)
> FROM (VALUES(1))": While compiling [CAST(42:BIGINT):ANY NOT
> NULL]Error while executing SQL "SELECT CAST(CAST(42 AS BIGINT) AS
> ANY) FROM (VALUES(1))": While compiling [CAST(42:BIGINT):ANY NOT
> NULL]java.sql.SQLException: Error while executing SQL "SELECT
> CAST(CAST(42 AS BIGINT) AS ANY) FROM (VALUES(1))": While compiling
> [CAST(42:BIGINT):ANY NOT NULL] at
> org.apache.calcite.avatica.Helper.createException(Helper.java:56) at
> org.apache.calcite.avatica.Helper.createException(Helper.java:41) at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
> at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:534)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1535)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1474)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1533)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1523)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1486)
> at org.apache.calcite.test.JdbcTest.testNestedBigInt(JdbcTest.java:7707) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564) at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
> at
> org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:139)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:81)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
> at
> org.junit.jupiter.engine.descriptor.TestMethod
[jira] [Commented] (CALCITE-4686) SubQueryRemoveRule.matchJoin should correctly rewrite all sub-queries
[
https://issues.apache.org/jira/browse/CALCITE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417586#comment-17417586
]
Stamatis Zampetakis commented on CALCITE-4686:
--
Hi [~jamesstarr],
I made another pass over your PR and did some small changes:
* rebased your changes against master;
* refactored {{RelOptUtil#transformJoinConditionToCorrelate}}, improved
javadoc, and added the {{Experimental}} annotation;
* simplified the {{testJoinOnMultipleCorrelatedSubQueries}} test to make the
plan more readable.
You can find the updates in
https://github.com/zabetak/calcite/tree/CALCITE-4686 branch. If there is a
reason to keep the more complicated query inside
{{testJoinOnMultipleCorrelatedSubQueries}} or you don't agree with the other
changes please let me know.
Unfortunately we cannot merge this PR yet. While I was checking the plans I
found some parts of the plan in {{testJoinOnMultipleCorrelatedSubQueries}} to
be wrong. Actually I created (and merged) CALCITE-4784 to guard against such
problems so if you try to run again your test you will notice that now it fails
with an {{AssertionError}}. I will also post another question around the plans
directly in the PR.
> SubQueryRemoveRule.matchJoin should correctly rewrite all sub-queries
> -
>
> Key: CALCITE-4686
> URL: https://issues.apache.org/jira/browse/CALCITE-4686
> Project: Calcite
> Issue Type: Improvement
>Reporter: James Starr
>Assignee: James Starr
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> SubQueryRemoveRule.matchJoin only rewrites the first subquery in an ON
> condition each call. It should rewrite all of them down right side of the
> join in single call. Furthermore, the filter generated is not shifted
> correctly for the scope that it is being applied to.
> RelOptRulesTest.testExpandJoinIn, which currently disabled, throw an
> exception when run because filter is shifted to be applied to in the context
> of the right side but it applied on top of the join. Which can be observed
> by commenting out the litmus check.
> {code:java}
> @Test void testExpandJoinIn() {
> final String sql = "select empno\n"
> + "from sales.emp left join sales.dept\n"
> + "on emp.deptno in (select deptno from sales.emp where empno < 20)";
> checkSubQuery(sql).check();
> }
> {code}
> {noformat}
> RexInputRef index 7 out of range 0..2
> java.lang.AssertionError: RexInputRef index 7 out of range 0..2
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:32)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:61)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:114)
> at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:144)
> at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:61)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:189)
> at org.apache.calcite.rel.core.Join.isValid(Join.java:176)
> at
> org.apache.calcite.test.SqlToRelConverterTest$RelValidityChecker.visit(SqlToRelConverterTest.java:4261)
> at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:46)
> at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:46)
> at
> org.apache.calcite.test.SqlToRelConverterTest$RelValidityChecker.visit(SqlToRelConverterTest.java:4264)
> at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
> at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:46)
> at
> org.apache.calcite.test.SqlToRelConverterTest$RelValidityChecker.visit(SqlToRelConverterTest.java:4264)
> at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
> at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:46)
> at
> org.apache.calcite.test.SqlToRelConverterTest$RelValidityChecker.visit(SqlToRelConverterTest.java:4264)
> at org.apache.calcite.rel.RelVisitor.go(RelVisitor.java:63)
> at
> org.apache.calcite.test.SqlToRelTestBase.assertValid(SqlToRelTestBase.java:154)
> at
> org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:163)
> at
> org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:109)
> at
> org.apache.calcite.test.RelOptTestBase.access$100(RelOptTestBase.java:66)
> at
> org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:340)
> at
> org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:316)
> at
> org.apache.calcite.test.RelOptRulesTest.testExpandJoinIn(RelOptRulesTest.java:5702)
>
[jira] [Resolved] (CALCITE-4784) Ensure Correlate#requiredColumns is subset of columns in left relation
[
https://issues.apache.org/jira/browse/CALCITE-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved CALCITE-4784.
--
Resolution: Fixed
Fixed in
[8ddb3f82e997d1a10619fc37ff7900823c7c0b91|https://github.com/apache/calcite/commit/8ddb3f82e997d1a10619fc37ff7900823c7c0b91].
> Ensure Correlate#requiredColumns is subset of columns in left relation
> --
>
> Key: CALCITE-4784
> URL: https://issues.apache.org/jira/browse/CALCITE-4784
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{Correlate}}
> [expression|https://github.com/apache/calcite/blob/f3baf348598fcc6bc4f97a0abee3f99309e5bf76/core/src/main/java/org/apache/calcite/rel/core/Correlate.java]
> has among others a field ({{requiredColumns}} for representing the set of
> columns that are used by the correlation.
> {code:java}
> public abstract class Correlate extends BiRel {
> protected final CorrelationId correlationId;
> protected final ImmutableBitSet requiredColumns;
> protected final JoinRelType joinType;
> /**
>* Returns the required columns in left relation required for the
> correlation
>* in the right.
>*
>* @return columns in left relation required for the correlation in the
> right
>*/
> public ImmutableBitSet getRequiredColumns() {
> return requiredColumns;
> }
> {code}
> As the javadoc indicates (implicitly) these columns must be a subset of the
> columns of the left relation. This is currently not enforced when creating a
> correlation which can lead to invalid plans and runtime failures which are
> hard to investigate.
> I propose to introduce an additional check inside {{Correlate#isValid}}
> method for enforcing that required columns are valid.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4784) Ensure Correlate#requiredColumns is subset of columns in left relation
[
https://issues.apache.org/jira/browse/CALCITE-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417576#comment-17417576
]
Stamatis Zampetakis commented on CALCITE-4784:
--
There is already an {{assert isValid(...)}} call in the constructor of the
{{Correlate}} and in the constructors of other relational expressions so I am
inclined to keep the new check inside {{isValid}} method mostly for keeping
things uniform.
> Ensure Correlate#requiredColumns is subset of columns in left relation
> --
>
> Key: CALCITE-4784
> URL: https://issues.apache.org/jira/browse/CALCITE-4784
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The {{Correlate}}
> [expression|https://github.com/apache/calcite/blob/f3baf348598fcc6bc4f97a0abee3f99309e5bf76/core/src/main/java/org/apache/calcite/rel/core/Correlate.java]
> has among others a field ({{requiredColumns}} for representing the set of
> columns that are used by the correlation.
> {code:java}
> public abstract class Correlate extends BiRel {
> protected final CorrelationId correlationId;
> protected final ImmutableBitSet requiredColumns;
> protected final JoinRelType joinType;
> /**
>* Returns the required columns in left relation required for the
> correlation
>* in the right.
>*
>* @return columns in left relation required for the correlation in the
> right
>*/
> public ImmutableBitSet getRequiredColumns() {
> return requiredColumns;
> }
> {code}
> As the javadoc indicates (implicitly) these columns must be a subset of the
> columns of the left relation. This is currently not enforced when creating a
> correlation which can lead to invalid plans and runtime failures which are
> hard to investigate.
> I propose to introduce an additional check inside {{Correlate#isValid}}
> method for enforcing that required columns are valid.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3935) Materialization-Failed, meeting to pull up bottom filter of Left/Right Join
[
https://issues.apache.org/jira/browse/CALCITE-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Yanlin resolved CALCITE-3935.
--
Resolution: Fixed
Fixed in
[https://github.com/apache/calcite/commit/a36f5a21f006ee717e7949269a99fa48a65f6abe.]
thx [~wojustme]
> Materialization-Failed, meeting to pull up bottom filter of Left/Right Join
> ---
>
> Key: CALCITE-3935
> URL: https://issues.apache.org/jira/browse/CALCITE-3935
> Project: Calcite
> Issue Type: Bug
>Reporter: Xurenhe
>Assignee: Xurenhe
>Priority: Major
> Labels: pull-request-available
> Attachments: Jietu20200417-200532.png
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Hi all.
> `SubstitutionVisitor` support more join_type in some rules of `Join`.
> Now `SubstitutionVisitor` only support join type of `INNER`, when pulling up
> bottom filters of join.
> It doesn't work as follows.
> {code:java}
> @Test public void testJoinOnLeftProjectWithFilterToJoin() {
> String mv = ""
> + "select \"emps\".\"empid\", \"depts\".\"name\",
> \"emps\".\"salary\"\n" +
> "from \"emps\"\n" +
> "left join \"depts\"\n" +
> "on \"emps\".\"deptno\" = \"depts\".\"deptno\"\n"
> + "where \"emps\".\"empid\" > 10";
> String query = ""
> + "select \"emps\".\"empid\", \"depts\".\"name\",
> \"emps\".\"salary\"\n" +
> "from \"emps\"\n" +
> "left join \"depts\"\n" +
> "on \"emps\".\"deptno\" = \"depts\".\"deptno\"\n"
> + "where \"emps\".\"empid\" > 40";
> sql(mv, query).withOnlyBySubstitution(true).ok();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4787) Evaluate use of Immutables instead of ImmutableBeans
[ https://issues.apache.org/jira/browse/CALCITE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417522#comment-17417522 ] Alessandro Solimando commented on CALCITE-4787: --- I am not familiar with GraalVM, but I am very fond of immutables library, I have used it with success in several past projects, it has always helped reducing boilerplate, made code clearer and safer (for both writing the classes and for using them). Additional good features: it also plays very well with JSON serialization out of the box (tried with Jackson but it supports GSON and others), and it's a compile time library, so no additional dependencies at runtime, and no particular setup effort to build the project with recent IDEs. So a big +1 from me on the suggested adoption. > Evaluate use of Immutables instead of ImmutableBeans > > > Key: CALCITE-4787 > URL: https://issues.apache.org/jira/browse/CALCITE-4787 > Project: Calcite > Issue Type: Improvement >Reporter: Jacques Nadeau >Priority: Major > > In the creation of CALCITE-3328, [Immutables|https://immutables.github.io/] > was discussed as an alternative to a custom implementation. This ticket is to > evaluate the impact to the codebase of changing. Ideally, introduction of > immutables would both add flexibility and reduce the amount of code > associated with these classes. > Immutables works via annotation processor which means that it is should be > relatively seamless to build systems and IDEs. > The switch would also make it easier to work with these objects types in the > context of aot compilation tools like GraalVM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417499#comment-17417499
]
Ruben Q L commented on CALCITE-4773:
[~julianhyde], do you have further comments about this ticket? Otherwise I
propose to squash commits and merge the PR in the coming days.
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
> relBuilder.getTypeFactory()
> .createTypeWithNullability(projExprs.get(0).getType(), true);
> final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
> projExprs.get(0));
> relBuilder.push(aggregate).project(cast);
> call.transformTo(relBuilder.build());
> {code}
> Note that the comment assumes that _"singleAggRel produces a nullable type"_,
> but in this particular case, it seems to produce a non-nullable type, so
> probably this piece of code needs to be adapted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
