[jira] [Commented] (CALCITE-3786) Add Digest interface to enable efficient hashCode(equals) for RexNode and RelNode
[
https://issues.apache.org/jira/browse/CALCITE-3786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133914#comment-17133914
]
Danny Chen commented on CALCITE-3786:
-
[~vladimirsitnikov] Thanks for the background sharing, if we do the
normalization during planning already, it would be better that we made the
thing more explicit, not an implicit contract there. That is to say, i prefer
to keep sync the plan structure same with the real RexTree, a normalization on
creation seems better. We can have a control flag three just like the Rex
simplification, but the default value should be true.
Personally i still question about the gains of Rex normalization because it
makes the RexNode code complex, and there is an implicit contract there, we
have already made decision that Rex normalization is the way to go, so let's
make the impl better if we can.
[~hyuan] I can thought of another benefit to use Digest, downstream projects
need only implement the #explainTerms instead of additional #hashCode and
#equals, which are both error-prone. As for the additional mem consumption for
object references, assume 100 bytes for a Digest there, and 1 rel nodes,
the total memory should be 1Mb, which i think is acceptable, so i would choose
a more concise interface, a Digest behind the #explainTerms.
> Add Digest interface to enable efficient hashCode(equals) for RexNode and
> RelNode
> -
>
> Key: CALCITE-3786
> URL: https://issues.apache.org/jira/browse/CALCITE-3786
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.21.0
>Reporter: Vladimir Sitnikov
>Assignee: Danny Chen
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Current digests for RexNode, RelNode, RelType, and similar cases use String
> concatenation.
> It is easy to implement, however, it has drawbacks:
> 1) String objects cannot be reused. For instance, RexCall has operands,
> however, the digest is duplicated. It causes extra memory use and extra CPU
> for string copying
> 2) There's no way to have multiple #toString() methods. RelType might need
> multiple digests: "including field names", "excluding field names".
> A suggested resolution might be behind the lines of
> {code:java}
> class Digest { // immutable
> final int hashCode; // speedup hashCode and equals
> final Object[] contents; // The values are either other Digest objects or
> Strings
> String toString(); // e.g. for debugging purposes
> int compareTo(Digest); // e.g. for debugging purposes.
> }
> {code}
> Note how fields in Kotlin are aligned much better, and it makes it easier to
> read:
> {code:java}
> class Digest { // immutable
> val hashCode: Int // speedup hashCode and equals
> val contents: Array // The values are either other Digest objects or
> Strings
> fun toString(): String // e.g. for debugging purposes
> fun compareTo(other: Digest): Int // e.g. for debugging purposes.
> }
> {code}
> Then the digest for RexCall could be the bits relevant to RexCall itself +
> digests of the operands (which can be reused as is)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4059) SqlTypeUtil#equalSansNullability doesn't consider Array/Map
[ https://issues.apache.org/jira/browse/CALCITE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133877#comment-17133877 ] Danny Chen commented on CALCITE-4059: - Thanks [~Aron.tao], did you try SqlTypeUtil#equalAsStructSansNullability ? > SqlTypeUtil#equalSansNullability doesn't consider Array/Map > --- > > Key: CALCITE-4059 > URL: https://issues.apache.org/jira/browse/CALCITE-4059 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4060) TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN.
[ https://issues.apache.org/jira/browse/CALCITE-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiatao Tao updated CALCITE-4060: Description: Current TypeCoercionImpl#inOperationCoercion only considers IN. !image-2020-06-12-10-43-52-584.png! was: current !image-2020-06-12-10-43-52-584.png! > TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN. > - > > Key: CALCITE-4060 > URL: https://issues.apache.org/jira/browse/CALCITE-4060 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Critical > Attachments: image-2020-06-12-10-22-49-218.png, > image-2020-06-12-10-43-52-584.png > > Time Spent: 10m > Remaining Estimate: 0h > > Current TypeCoercionImpl#inOperationCoercion only considers IN. > !image-2020-06-12-10-43-52-584.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4060) TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN.
[ https://issues.apache.org/jira/browse/CALCITE-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiatao Tao updated CALCITE-4060: Description: current !image-2020-06-12-10-43-52-584.png! > TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN. > - > > Key: CALCITE-4060 > URL: https://issues.apache.org/jira/browse/CALCITE-4060 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Critical > Attachments: image-2020-06-12-10-22-49-218.png, > image-2020-06-12-10-43-52-584.png > > Time Spent: 10m > Remaining Estimate: 0h > > current > !image-2020-06-12-10-43-52-584.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4060) TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN.
[ https://issues.apache.org/jira/browse/CALCITE-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiatao Tao updated CALCITE-4060: Attachment: image-2020-06-12-10-43-52-584.png > TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN. > - > > Key: CALCITE-4060 > URL: https://issues.apache.org/jira/browse/CALCITE-4060 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Critical > Attachments: image-2020-06-12-10-22-49-218.png, > image-2020-06-12-10-43-52-584.png > > Time Spent: 10m > Remaining Estimate: 0h > > current > !image-2020-06-12-10-43-52-584.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4060) TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN.
[ https://issues.apache.org/jira/browse/CALCITE-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133853#comment-17133853 ] Jiatao Tao commented on CALCITE-4060: - [~amaliujia] I add a ut: TypeCoercionConverterTest#testNotInOperation, imitate "TypeCoercionConverterTest#testInOperation" > TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN. > - > > Key: CALCITE-4060 > URL: https://issues.apache.org/jira/browse/CALCITE-4060 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Critical > Attachments: image-2020-06-12-10-22-49-218.png > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4060) TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN.
[ https://issues.apache.org/jira/browse/CALCITE-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133846#comment-17133846 ] Jiatao Tao edited comment on CALCITE-4060 at 6/12/20, 2:23 AM: --- Hi [~julianhyde], I add a ut, and debug find the op is NOT IN: org.apache.calcite.test.TypeCoercionConverterTest#testNotInOperation !image-2020-06-12-10-22-49-218.png! was (Author: aron.tao): Hi [~julianhyde], I add a ut, and debug we can find the op is NOT IN: org.apache.calcite.test.TypeCoercionConverterTest#testNotInOperation !image-2020-06-12-10-22-49-218.png! > TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN. > - > > Key: CALCITE-4060 > URL: https://issues.apache.org/jira/browse/CALCITE-4060 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Critical > Attachments: image-2020-06-12-10-22-49-218.png > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4060) TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN.
[ https://issues.apache.org/jira/browse/CALCITE-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133846#comment-17133846 ] Jiatao Tao commented on CALCITE-4060: - Hi [~julianhyde], I add a ut, and debug we can find the op is NOT IN: org.apache.calcite.test.TypeCoercionConverterTest#testNotInOperation !image-2020-06-12-10-22-49-218.png! > TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN. > - > > Key: CALCITE-4060 > URL: https://issues.apache.org/jira/browse/CALCITE-4060 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Critical > Attachments: image-2020-06-12-10-22-49-218.png > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4060) TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN.
[ https://issues.apache.org/jira/browse/CALCITE-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiatao Tao updated CALCITE-4060: Attachment: image-2020-06-12-10-22-49-218.png > TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN. > - > > Key: CALCITE-4060 > URL: https://issues.apache.org/jira/browse/CALCITE-4060 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Critical > Attachments: image-2020-06-12-10-22-49-218.png > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4059) SqlTypeUtil#equalSansNullability doesn't consider Array/Map
[ https://issues.apache.org/jira/browse/CALCITE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133842#comment-17133842 ] Jiatao Tao commented on CALCITE-4059: - [~amaliujia] 👌 > SqlTypeUtil#equalSansNullability doesn't consider Array/Map > --- > > Key: CALCITE-4059 > URL: https://issues.apache.org/jira/browse/CALCITE-4059 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3786) Add Digest interface to enable efficient hashCode(equals) for RexNode and RelNode
[
https://issues.apache.org/jira/browse/CALCITE-3786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133799#comment-17133799
]
Haisheng Yuan commented on CALCITE-3786:
{quote}
Even we are with these additional object references, then how much join nodes
during the planning ? They do not expect to be the cause of the OOM. What
causes the OOM is the digest strings, especially for complex RexCalls, before
this patch, their string format digest were always patched up in the digest of
the RelNode, which would be big memory comsuption.
{quote}
I am not saying this proposal causes the OOM, instead it reduces memory usage
comparing with the string digest, as long as we stop using string digest for
RexNode, OOM issue like 3784 will ease a lot. However, the so-called Digest
still waste a lot of memory, unnecessarily. If you think Join node is not that
many, how about Project? Try to compute the Digest size of Project, it may
consume much more memory.
{quote}
Why do you think we should use #hashCode and #equals to decide that two
relational expression are semantically equivalent ? Or i asked it in different
way, if two relational expression are equivalent, should they be equals to each
other ? (For example, 2 projects differs only with field aliases.)
{quote}
Why not? Although this is not mandatory, it is so intuitive and natural. Two
Project with different aliases can be equal with each other, as long as they
produce the same hashcode, we treat them as equal object, in MEMO we can use 1
instance to represent the other one.
{quote}
BTW, even if RexCall are new object, use the equals to compare would solve the
problem.
{quote}
Exactly, that is the problem. It is deep comparison, which may cause
performance issue for large complex queries. Note that different rules can
generate lots of intermediate, equal RelNode and RexNode but different
instances, especially when Project/Filter/Calc merge happens on physical
operator in VolcanoPlanner.
BTW, does anyone know that [VoltDB|https://www.voltdb.com/company/why-voltdb/],
an in-memory OLTP RDBMS, is
[experimenting|https://github.com/VoltDB/voltdb/tree/master/src/frontend/org/voltdb/plannerv2]
on integrating Calcite into its system?
If Calcite wants to be versatile for OLTP, OLAP, Batch, Stream, every bit of
memory counts, and every millisecond counts.
I am casting "-1" on this proposal, the justification has been provided in this
and previous comments. I am just expressing objection on this idea, I don't
have rights to prevent anyone from doing anything, though.
> Add Digest interface to enable efficient hashCode(equals) for RexNode and
> RelNode
> -
>
> Key: CALCITE-3786
> URL: https://issues.apache.org/jira/browse/CALCITE-3786
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.21.0
>Reporter: Vladimir Sitnikov
>Assignee: Danny Chen
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Current digests for RexNode, RelNode, RelType, and similar cases use String
> concatenation.
> It is easy to implement, however, it has drawbacks:
> 1) String objects cannot be reused. For instance, RexCall has operands,
> however, the digest is duplicated. It causes extra memory use and extra CPU
> for string copying
> 2) There's no way to have multiple #toString() methods. RelType might need
> multiple digests: "including field names", "excluding field names".
> A suggested resolution might be behind the lines of
> {code:java}
> class Digest { // immutable
> final int hashCode; // speedup hashCode and equals
> final Object[] contents; // The values are either other Digest objects or
> Strings
> String toString(); // e.g. for debugging purposes
> int compareTo(Digest); // e.g. for debugging purposes.
> }
> {code}
> Note how fields in Kotlin are aligned much better, and it makes it easier to
> read:
> {code:java}
> class Digest { // immutable
> val hashCode: Int // speedup hashCode and equals
> val contents: Array // The values are either other Digest objects or
> Strings
> fun toString(): String // e.g. for debugging purposes
> fun compareTo(other: Digest): Int // e.g. for debugging purposes.
> }
> {code}
> Then the digest for RexCall could be the bits relevant to RexCall itself +
> digests of the operands (which can be reused as is)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4060) TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN.
[ https://issues.apache.org/jira/browse/CALCITE-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133771#comment-17133771 ] Julian Hyde commented on CALCITE-4060: -- I don't think NOT_IN should exist inside RexNodes. See CALCITE-3242. > TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN. > - > > Key: CALCITE-4060 > URL: https://issues.apache.org/jira/browse/CALCITE-4060 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Critical > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4060) TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN.
[ https://issues.apache.org/jira/browse/CALCITE-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133763#comment-17133763 ] Rui Wang commented on CALCITE-4060: --- [~Aron.tao] Could you use some examples in this Jira to demonstrate your intention? E.g. use an query to demonstrate what is wrong? > TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN. > - > > Key: CALCITE-4060 > URL: https://issues.apache.org/jira/browse/CALCITE-4060 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Critical > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4059) SqlTypeUtil#equalSansNullability doesn't consider Array/Map
[ https://issues.apache.org/jira/browse/CALCITE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133759#comment-17133759 ] Rui Wang edited comment on CALCITE-4059 at 6/11/20, 10:36 PM: -- [~Aron.tao] could you improve this bug by adding some examples in the description? E.g. what is currently supported, and what is not supported? was (Author: amaliujia): [~Aron.tao] could you improve this bug by adding some examples in the description? > SqlTypeUtil#equalSansNullability doesn't consider Array/Map > --- > > Key: CALCITE-4059 > URL: https://issues.apache.org/jira/browse/CALCITE-4059 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4059) SqlTypeUtil#equalSansNullability doesn't consider Array/Map
[ https://issues.apache.org/jira/browse/CALCITE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133759#comment-17133759 ] Rui Wang commented on CALCITE-4059: --- [~Aron.tao] could you improve this bug by adding some examples in the description? > SqlTypeUtil#equalSansNullability doesn't consider Array/Map > --- > > Key: CALCITE-4059 > URL: https://issues.apache.org/jira/browse/CALCITE-4059 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-4038) Refactor RexVisitor, RexBiVisitor, RelOptUtil.InputFinder
[
https://issues.apache.org/jira/browse/CALCITE-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde resolved CALCITE-4038.
--
Fix Version/s: 1.24.0
Resolution: Fixed
Fixed in
[59d6eb85|https://github.com/apache/calcite/commit/59d6eb852676ab5d77eccd63f488c9f121804d5d].
> Refactor RexVisitor, RexBiVisitor, RelOptUtil.InputFinder
> -
>
> Key: CALCITE-4038
> URL: https://issues.apache.org/jira/browse/CALCITE-4038
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.24.0
>
>
> Various improvements to {{RexVisitor}}, {{RexBiVisitor}},
> {{RelOptUtil.InputFinder}} classes.
> 1. Make mutable field {{RelOptUtil.InputFinder.inputBitSet}} private. The
> field still exists, public and deprecated, but it shadows a new private
> field. Rather than calling {{InputFinder.inputBitSet.build()}} you should now
> call {{InputFinder.build()}}.
> 2. In {{RexVisitor}} and {{RexBiVisitor}} add methods {{visitList}} and
> {{visitEach}} (the former returns a list, and the latter returns void).
> 3. Deprecate {{RexShuttle.apply}} in favor of {{visitList}}.
> 4. Add {{RexWindowBound.accept(RexBiVisitor)}}.
> 5. For {{RexBiVisitor}} add a method {{visitEachIndexed}}, the
> index passed as the 'payload' argument.
> 6. Add abstract implementations of {{RexBiVisitor}}: {{RexUnaryBiVisitor}}
> and {{RexBiVisitorImpl}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3975) Add options to ProjectFilterTransposeRule to push down project and filter expressions whole, not just field references
[
https://issues.apache.org/jira/browse/CALCITE-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde resolved CALCITE-3975.
--
Fix Version/s: 1.24.0
Resolution: Fixed
Fixed in
[eb22c0186|https://github.com/apache/calcite/commit/eb22c01867a54cbe521a69725608351491f971ca].
> Add options to ProjectFilterTransposeRule to push down project and filter
> expressions whole, not just field references
> --
>
> Key: CALCITE-3975
> URL: https://issues.apache.org/jira/browse/CALCITE-3975
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.22.0
>Reporter: Steven Talbot
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.24.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> ProjectFilterTransposeRule should succeed for project that happens to
> reference all input columns. That is, I think if I make the trivial fix of
> just "only skip trivial projects", something like
> {noformat}
> && origProj.getProjects().stream().allMatch((proj) -> proj instanceof
> RexInputRef) {noformat}
> at
> [https://github.com/apache/calcite/blob/571731b80a58eb095ebac7123285c375e7afff90/core/src/main/java/org/apache/calcite/rel/rules/PushProjector.java#L354
>
> |https://github.com/apache/calcite/blob/571731b80a58eb095ebac7123285c375e7afff90/core/src/main/java/org/apache/calcite/rel/rules/PushProjector.java#L354]HepPlanner
> goes into infinite recursion with the rule.
> But here's the test case:
>
> {code:java}
> @Test public void testPushProjectPastFilter3() {
> final String sql = "select empno + deptno, ename, job, mgr, hiredate, sal,
> comm, slacker from emp where sal = 10 * comm\n"
> + "and upper(ename) = 'FOO'";
> sql(sql).withRule(ProjectFilterTransposeRule.INSTANCE).check();
> }
> {code}
>
>
>
> {noformat}
>
>
>
>
>
>
>
>
>
>
> {noformat}
>
> There's no reason the rule shouldn't succeed here, right? Or am I missing
> something?
> The reason this rule goes into an infinite recursion with hepplanner is
> because it sticks a project on top after transpose to handle common
> expressions extracted from the filter and the project. Ideally, it could have
> a mode where it could avoid doing that and do a true "transpose" if there was
> no need for it. For example, I don't think there is a a need for a reproject
> on top in this test case: you can just transpose and everything works as it
> should. This would be another way to avoid infinite recursion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4061) Build should fail if Calcite code uses deprecated APIs
Julian Hyde created CALCITE-4061: Summary: Build should fail if Calcite code uses deprecated APIs Key: CALCITE-4061 URL: https://issues.apache.org/jira/browse/CALCITE-4061 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Calcite build should fail if Calcite code uses deprecated APIs. This includes external APIs (e.g. Guava methods) and internal APIs (e.g. a deprecated class in the org.apache.calcite.util package). This used to occur when the build was powered by Maven. Compared to a policy where code is allowed to use deprecated APIs for a 'grace period', this policy has a number of advantages. One is that Calcite devs (or users) can upgrade dependencies at short notice, with no code changes. Another is that it forces people who are doing internal refactoring (replacing one internal Calcite API with another) to finish the job. [~vlsi] Can you take this please? I can't figure out how to set javac's '-Xlint:deprecated -Werror' from Gradle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-4061) Build should fail if Calcite code uses deprecated APIs
[ https://issues.apache.org/jira/browse/CALCITE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde reassigned CALCITE-4061: Assignee: Vladimir Sitnikov > Build should fail if Calcite code uses deprecated APIs > --- > > Key: CALCITE-4061 > URL: https://issues.apache.org/jira/browse/CALCITE-4061 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Vladimir Sitnikov >Priority: Major > > Calcite build should fail if Calcite code uses deprecated APIs. This includes > external APIs (e.g. Guava methods) and internal APIs (e.g. a deprecated class > in the org.apache.calcite.util package). > This used to occur when the build was powered by Maven. > Compared to a policy where code is allowed to use deprecated APIs for a > 'grace period', this policy has a number of advantages. One is that Calcite > devs (or users) can upgrade dependencies at short notice, with no code > changes. > Another is that it forces people who are doing internal refactoring > (replacing one internal Calcite API with another) to finish the job. > [~vlsi] Can you take this please? I can't figure out how to set javac's > '-Xlint:deprecated -Werror' from Gradle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4060) TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN.
[ https://issues.apache.org/jira/browse/CALCITE-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133280#comment-17133280 ] Jiatao Tao commented on CALCITE-4060: - MR: [https://github.com/apache/calcite/pull/2020] > TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN. > - > > Key: CALCITE-4060 > URL: https://issues.apache.org/jira/browse/CALCITE-4060 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Critical > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4060) TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN.
Jiatao Tao created CALCITE-4060: --- Summary: TypeCoercionImpl#inOperationCoercion doesn't consider NOT_IN. Key: CALCITE-4060 URL: https://issues.apache.org/jira/browse/CALCITE-4060 Project: Calcite Issue Type: Bug Reporter: Jiatao Tao -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4058) Add Limit and LogicalLimit operator and deprecate limit in Sort
[
https://issues.apache.org/jira/browse/CALCITE-4058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133249#comment-17133249
]
Haisheng Yuan commented on CALCITE-4058:
You mean objections? I would love to hear.
> Add Limit and LogicalLimit operator and deprecate limit in Sort
> ---
>
> Key: CALCITE-4058
> URL: https://issues.apache.org/jira/browse/CALCITE-4058
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Add {{Limit}} base class and {{LogicalLimit}} operator.
> Fetch and offset should not be in Sort operator, we should deprecate the
> fields while keeping the Sort work for compatibility.
> {code:java}
> select * from foo limit 5 offset 3;
> select * from foo order by a;
> {code}
> The above 2 queries are totally different query, but have the same operators,
> SortxxxTranspose sometimes means limit push down, which is counter-intuitive.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4059) SqlTypeUtil#equalSansNullability doesn't consider Array/Map
[ https://issues.apache.org/jira/browse/CALCITE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133247#comment-17133247 ] Jiatao Tao commented on CALCITE-4059: - Origin SqlTypeUtil#equalSansNullability only consider atomic type. mr: [https://github.com/apache/calcite/pull/2019] > SqlTypeUtil#equalSansNullability doesn't consider Array/Map > --- > > Key: CALCITE-4059 > URL: https://issues.apache.org/jira/browse/CALCITE-4059 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4059) SqlTypeUtil#equalSansNullability doesn't consider Array/Map
[ https://issues.apache.org/jira/browse/CALCITE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiatao Tao updated CALCITE-4059: Summary: SqlTypeUtil#equalSansNullability doesn't consider Array/Map (was: equalSansNullability doesn't consider Array/Map) > SqlTypeUtil#equalSansNullability doesn't consider Array/Map > --- > > Key: CALCITE-4059 > URL: https://issues.apache.org/jira/browse/CALCITE-4059 > Project: Calcite > Issue Type: Bug >Reporter: Jiatao Tao >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4059) equalSansNullability doesn't consider Array/Map
Jiatao Tao created CALCITE-4059: --- Summary: equalSansNullability doesn't consider Array/Map Key: CALCITE-4059 URL: https://issues.apache.org/jira/browse/CALCITE-4059 Project: Calcite Issue Type: Bug Reporter: Jiatao Tao -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3786) Add Digest interface to enable efficient hashCode(equals) for RexNode and RelNode
[
https://issues.apache.org/jira/browse/CALCITE-3786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133205#comment-17133205
]
Vladimir Sitnikov commented on CALCITE-3786:
>so that we can finally compare 2 RexCall using member equals, and it works
>well.
1) There are code parts that still use #toString() for the comparison (e.g.
RexSimplify)
2) I had exactly that suggestion (normalize RexNodes on creation), however, you
have cast -1 multiple times because that would alter string representation of
the plans with little gain.
[https://lists.apache.org/thread.html/d89103671efdd813fac768fbc2336bd125e925f0790e9137a2a16375%40%3Cdev.calcite.apache.org%3E]
My suggestion was
Vladimir: It turned out "b" (sort operands in computeDigest) is easier to
implement.
Vladimir: I've filed a PR: [https://github.com/apache/calcite/pull/1703
]
and your response was "I’m strongly -1 for this way, because it beaks the plan
test where almost all of the change are meaningless."
Note: PR1703 was updated since that discussion, and the actual implementation
normalizes nodes at the planning time only, while the user-visible plans are
displayed without normalization.
> Add Digest interface to enable efficient hashCode(equals) for RexNode and
> RelNode
> -
>
> Key: CALCITE-3786
> URL: https://issues.apache.org/jira/browse/CALCITE-3786
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.21.0
>Reporter: Vladimir Sitnikov
>Assignee: Danny Chen
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Current digests for RexNode, RelNode, RelType, and similar cases use String
> concatenation.
> It is easy to implement, however, it has drawbacks:
> 1) String objects cannot be reused. For instance, RexCall has operands,
> however, the digest is duplicated. It causes extra memory use and extra CPU
> for string copying
> 2) There's no way to have multiple #toString() methods. RelType might need
> multiple digests: "including field names", "excluding field names".
> A suggested resolution might be behind the lines of
> {code:java}
> class Digest { // immutable
> final int hashCode; // speedup hashCode and equals
> final Object[] contents; // The values are either other Digest objects or
> Strings
> String toString(); // e.g. for debugging purposes
> int compareTo(Digest); // e.g. for debugging purposes.
> }
> {code}
> Note how fields in Kotlin are aligned much better, and it makes it easier to
> read:
> {code:java}
> class Digest { // immutable
> val hashCode: Int // speedup hashCode and equals
> val contents: Array // The values are either other Digest objects or
> Strings
> fun toString(): String // e.g. for debugging purposes
> fun compareTo(other: Digest): Int // e.g. for debugging purposes.
> }
> {code}
> Then the digest for RexCall could be the bits relevant to RexCall itself +
> digests of the operands (which can be reused as is)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3786) Add Digest interface to enable efficient hashCode(equals) for RexNode and RelNode
[
https://issues.apache.org/jira/browse/CALCITE-3786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133190#comment-17133190
]
Danny Chen commented on CALCITE-3786:
-
[~vladimirsitnikov] I tried to normalize the RexCall in its constructor, so
that we can finally compare 2 RexCall using member equals, and it works well.
But i also got 54 plan changes which are all operands order change [1], it this
as expected ?
[1]
https://github.com/apache/calcite/pull/2016/commits/99f55001923eb1618268cfb9d0d645a86eb2ceff
> Add Digest interface to enable efficient hashCode(equals) for RexNode and
> RelNode
> -
>
> Key: CALCITE-3786
> URL: https://issues.apache.org/jira/browse/CALCITE-3786
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.21.0
>Reporter: Vladimir Sitnikov
>Assignee: Danny Chen
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Current digests for RexNode, RelNode, RelType, and similar cases use String
> concatenation.
> It is easy to implement, however, it has drawbacks:
> 1) String objects cannot be reused. For instance, RexCall has operands,
> however, the digest is duplicated. It causes extra memory use and extra CPU
> for string copying
> 2) There's no way to have multiple #toString() methods. RelType might need
> multiple digests: "including field names", "excluding field names".
> A suggested resolution might be behind the lines of
> {code:java}
> class Digest { // immutable
> final int hashCode; // speedup hashCode and equals
> final Object[] contents; // The values are either other Digest objects or
> Strings
> String toString(); // e.g. for debugging purposes
> int compareTo(Digest); // e.g. for debugging purposes.
> }
> {code}
> Note how fields in Kotlin are aligned much better, and it makes it easier to
> read:
> {code:java}
> class Digest { // immutable
> val hashCode: Int // speedup hashCode and equals
> val contents: Array // The values are either other Digest objects or
> Strings
> fun toString(): String // e.g. for debugging purposes
> fun compareTo(other: Digest): Int // e.g. for debugging purposes.
> }
> {code}
> Then the digest for RexCall could be the bits relevant to RexCall itself +
> digests of the operands (which can be reused as is)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4055) RelFieldTrimmer loses hints
[
https://issues.apache.org/jira/browse/CALCITE-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133011#comment-17133011
]
Ruben Q L commented on CALCITE-4055:
[~danny0405] thanks for the info, I have tried to apply the same strategy in
the PR, please take a look when you have some time.
> RelFieldTrimmer loses hints
> ---
>
> Key: CALCITE-4055
> URL: https://issues.apache.org/jira/browse/CALCITE-4055
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The trimmed plan generated by RelFieldTrimmer can lose the hints that might
> be present in the input plan.
> The issue can be reproduced with the following test (to be added to
> RelFieldTrimmerTest):
> {code:java}
> @Test void testJoinWithHints() {
> final RelHint noHashJoinHint =
> RelHint.builder("NO_HASH_JOIN").inheritPath(0).build();
> final RelBuilder builder = RelBuilder.create(config().build());
> final RelNode original =
> builder.scan("EMP")
> .scan("DEPT")
> .join(JoinRelType.INNER,
> builder.equals(
> builder.field(2, 0, "DEPTNO"),
> builder.field(2, 1, "DEPTNO")))
> .hints(noHashJoinHint)
> .project(
> builder.field("ENAME"),
> builder.field("DNAME"))
> .build();
> final RelFieldTrimmer fieldTrimmer = new RelFieldTrimmer(null, builder);
> final RelNode trimmed = fieldTrimmer.trim(original);
> final String expected = ""
> + "LogicalProject(ENAME=[$1], DNAME=[$4])\n"
> + " LogicalJoin(condition=[=($2, $3)], joinType=[inner])\n"
> + "LogicalProject(EMPNO=[$0], ENAME=[$1], DEPTNO=[$7])\n"
> + " LogicalTableScan(table=[[scott, EMP]])\n"
> + "LogicalProject(DEPTNO=[$0], DNAME=[$1])\n"
> + " LogicalTableScan(table=[[scott, DEPT]])\n";
> assertThat(trimmed, hasTree(expected));
> assertTrue(original.getInput(0) instanceof Join);
> final Join originalJoin = (Join) original.getInput(0);
> assertTrue(originalJoin.getHints().contains(noHashJoinHint));
> assertTrue(trimmed.getInput(0) instanceof Join);
> final Join join = (Join) trimmed.getInput(0);
> assertTrue(join.getHints().contains(noHashJoinHint));
> }
> {code}
> which fails in the last line:
> {{assertTrue(join.getHints().contains(noHashJoinHint));}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3724) Presto dialect implementation
[ https://issues.apache.org/jira/browse/CALCITE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved CALCITE-3724. - Fix Version/s: 1.24.0 Resolution: Fixed Fixed in [f577b7e|https://github.com/apache/calcite/commit/f577b7e3d91191051cfdaade27e0a74f3603648a] ! > Presto dialect implementation > - > > Key: CALCITE-3724 > URL: https://issues.apache.org/jira/browse/CALCITE-3724 > Project: Calcite > Issue Type: Improvement >Reporter: Forward Xu >Assignee: Forward Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.24.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Implement PrestoSqlDialect -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4055) RelFieldTrimmer loses hints
[
https://issues.apache.org/jira/browse/CALCITE-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4055:
---
Summary: RelFieldTrimmer loses hints (was: RelFieldTrimmer loses join
hints)
> RelFieldTrimmer loses hints
> ---
>
> Key: CALCITE-4055
> URL: https://issues.apache.org/jira/browse/CALCITE-4055
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The trimmed plan generated by RelFieldTrimmer can lose the hints that might
> be present in the input plan.
> The issue can be reproduced with the following test (to be added to
> RelFieldTrimmerTest):
> {code:java}
> @Test void testJoinWithHints() {
> final RelHint noHashJoinHint =
> RelHint.builder("NO_HASH_JOIN").inheritPath(0).build();
> final RelBuilder builder = RelBuilder.create(config().build());
> final RelNode original =
> builder.scan("EMP")
> .scan("DEPT")
> .join(JoinRelType.INNER,
> builder.equals(
> builder.field(2, 0, "DEPTNO"),
> builder.field(2, 1, "DEPTNO")))
> .hints(noHashJoinHint)
> .project(
> builder.field("ENAME"),
> builder.field("DNAME"))
> .build();
> final RelFieldTrimmer fieldTrimmer = new RelFieldTrimmer(null, builder);
> final RelNode trimmed = fieldTrimmer.trim(original);
> final String expected = ""
> + "LogicalProject(ENAME=[$1], DNAME=[$4])\n"
> + " LogicalJoin(condition=[=($2, $3)], joinType=[inner])\n"
> + "LogicalProject(EMPNO=[$0], ENAME=[$1], DEPTNO=[$7])\n"
> + " LogicalTableScan(table=[[scott, EMP]])\n"
> + "LogicalProject(DEPTNO=[$0], DNAME=[$1])\n"
> + " LogicalTableScan(table=[[scott, DEPT]])\n";
> assertThat(trimmed, hasTree(expected));
> assertTrue(original.getInput(0) instanceof Join);
> final Join originalJoin = (Join) original.getInput(0);
> assertTrue(originalJoin.getHints().contains(noHashJoinHint));
> assertTrue(trimmed.getInput(0) instanceof Join);
> final Join join = (Join) trimmed.getInput(0);
> assertTrue(join.getHints().contains(noHashJoinHint));
> }
> {code}
> which fails in the last line:
> {{assertTrue(join.getHints().contains(noHashJoinHint));}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
